No Comments

An AI View of Scientific Creativity

Scott Fahlman,   October 9, 2010
Categories:  AI    


Can an AI system be creative?  A lot of people believe that the answer is no – obviously no.  After all, we are talking about a computer program.  It only does what its instructions tell it to do, and some human programmer wrote those instructions.  Furthermore, computer programs are deterministic: give a program the same input a thousand times, and it will give you the same answer a thousand times.  If you liked the system’s answer the first time, credit goes to the human programmer; if you didn’t, it’s not very creative to get stuck on that bad answer for all eternity.

That’s the popular view, but I disagree:  I believe that an AI system can exhibit what any fair observer would call creativity.  To explore the question properly, we have to take a closer look at what we mean by creativity, and then think about where it comes from.  That is the subject of this article.

Why scientific creativity?

In this discussion, I want to focus on scientific creativity, including also engineering, puzzle-solving, and creative planning of all kinds.  For now, I am avoiding artistic creativity because I think it’s harder to discuss that coherently – so let’s attack the easier (or better-defined) problem first.  In scientific (etc.) creativity, you generally are able to tell when you finally have a good solution to the problem you were working on: you have a theory that explains the phenomena at hand, with some predictive power for new observations; the bridge you designed stays up, and it proves to be less expensive or easier to construct than bridges built according to existing cookbook methods; the puzzle is solved; the plan you came up with accomplishes all of its goals with good efficiency.  The creative act here is generally in finding a good solution; the evaluation criteria are generally agreed upon.

In artistic creativity, you have this same problem of creating a solution, but it is coupled with the more complex problem of deciding whether the solution is any good.  We can usually tell whether the product is original in some way – until recently, nobody had ever dribbled paint all over a canvas, framed the result, and invited an audience to admire it – but is it art? Is it a good painting (or poem, novel, symphony, performance…)?  Is there some audience – preferably including more people than just the artist himself – who will find this work interesting or beautiful or provocative in some way?  There is a vast literature on artistic value and aesthetics, and (for now) I don’t intend to add to the clutter.

There is an even deeper problem when we think about computers making art that will be appreciated by humans: some excellent art is very intellectual and abstract, but a lot of what moves us is rooted in shared “visceral” human experience – things that a computer or robot would not have experienced in the same way.  For example, some of our emotional reaction we have to music is related to the human heartbeat, to other rhythms of our bodies and surroundings, and to sounds like crying and laughter and unexpected threatening noises. Our reactions to these things are (to some degree) hard-wired and shared across all human cultures.  Most good literature relates in some way to human fears, yearnings, and inter-personal relationships.  These are things that a computer might understand, as a blind person might understand colors, but they would not be a normal part of the experience of an AI system – even a long-lived robot that spends its whole “life” learning.  I would not absolutely rule out the possibility that an AI system will one day produce art that connects with a human audience with what feels, to us, like shared visceral experience, but getting there will be a much longer journey than coming up with AI systems that “creatively” solve scientific and engineering problems.

So, for now, we will focus only on the kind of creativity where there is some objective measure or general consensus on what is a good or bad solution – in which the cleverness is in how the solution was discovered.


Among those people who will admit that an AI system might show some degree of scientific creativity, there is a common belief that some degree of randomness is the secret ingredient.  If we put a random number generator in our code and use it to alter some of the system’s behaviors, then the system can do things that surprise its creator, avoiding the trap of always coming up with the same answer to a given problem.

I think that’s true, but it’s only a tiny part of the answer.  For one thing, in any really difficult problem, and any AI system complex enough to grapple with it, there will be plenty of randomness; we don’t have to add it explicitly.  Every problem (and every data set) will be a little bit different from the ones that we have seen before, and this inherent variation may cause the system to try different solution paths in different orders.

The AI system will also be a bit different each time.  A good problem-solving system will learn as it goes, encapsulating and generalizing lessons from its experience and using these to guide it toward some strategies and away from others.  If the system tries a particular path that fails to find a solution, or if it finds a solution that ultimately doesn’t work, there will be a memory of that effort.  If the system tries again on the same problem, it will be a different system because if this memory.  So an AI system with some learning built in will exhibit one of the key elements of creativity: Keep making new mistakes instead of repeating the same old mistakes.[1]

Also, while some random component may be necessary, random behavior by itself generally gets you nowhere.  Doing a random walk in the space of possible low-level actions or solutions is not going to solve any really hard problem, since good solutions will be extremely sparse in this space.[2]  A much larger role is played by the knowledge that we use to select strategies and to structure the space of possibilities that we want to explore, so that the random part of the search takes place in what the military would call a “target-rich environment”.  The right problem representation, a partial recipe, or a good plan for how to explore the space can often take the place of several billion years of unguided random search.

The French Horn theory of creativity

I think that a good metaphor for this creative process is playing a French horn.  Yes, there is a partially random excitation – flapping your lips to make the flatulent sound that we sometimes call “blowing raspberries”.[3]   No sound comes out if you don’t do this, but the process of making music has a lot more to do with four or five meters of beautifully crafted brass tubing, plus years spent learning how to work the valves, shape the left hand, and how to make exactly the right kind of “random” excitation that the score demands at any given moment.  In the realm of problem-solving, knowledge of the domain plays the role of the instrument, and accumulated experience is the essence of the player’s skill.

Some parents who want their kids to be creative teach them to run around behaving in a random and undisciplined manner.  To me, that makes as much sense as teaching kids to play the French horn by telling them to run around making raspberry sounds.  Yes, some spontaneity is required for creative problem solving, but most kids come equipped with this unless it is driven out of them.  Far more important is to teach them all the knowledge they will need about various fields, and the discipline to gain systematic experience in applying that knowledge.  (It helps if the kids think all of that learning and exploration is great fun – which it can be!)  It does take some courage and self-confidence to move away from what everyone else is doing and to try something new and perhaps heretical – to “think outside the box” – but that courage is much easier to muster if you have a solid understanding of the domain, what others are working on, and what possibilities they seem to be overlooking.

No Magic Needed

Here’s a hypothesis – actually only part of a hypothesis, but we will get to the other part later:  What we call “scientific creativity” is just good, effective problem solving that happens to lead to a surprising result. [4]Sometimes the surprise is that any result at all was found.  No magic is required – just a lot of knowledge, some good search strategies (that evolve as you gain experience with the domain), a willingness to look at some possibilities that others have missed, and perhaps just a dash of randomness or unpredictability in choosing new paths.

If this hypothesis is correct, there is no reason in principle why an AI problem-solving program could not exhibit this kind of creativity.  We just have to build in the right knowledge, representations, and strategies, plus the ability to learn more of these things based on experience, observation, and “being told”.  That word “just” is a stand-in for decades of research on this topic, not much of which is taking place at the moment.  (See my rant on this topic in an earlier article.)  However, I don’t see anything really fundamental that is missing from this picture.

The only way to prove that this hypothesis is correct is to do a lot more work in this area: either we will start to see AI systems with undeniable creativity, or we will get stuck and will have to admit that we’ve run into some fundamental barrier to further progress.  But we’re not stuck yet – just moving more slowly than most of us would like.

The idea of creative machines is troubling to some.  We like to think that scientific creativity is one of those special talents that we humans will never have to share with machines.  Some skeptics would even say that we don’t share this kind of problem-solving creativity with animals.  People who believe this obviously haven’t spent much time trying to keep squirrels out of their bird-feeder.  I think that the big difference is that humans are much better than animals at accumulating and passing on great heaps of accumulated knowledge and experience – language gives us an enormous advantage in doing this.  Animals without language must depend almost entirely on their own life-experience, and will probably never develop anything as complex as the bow and arrow, let alone integrated circuits and quantum physics.

One additional point:  Problem-solving is fractal, and therefore so is problem-solving creativity.[5]  When you tackle a big problem, you have to find the right overall structure for a solution.  That may be handed to you, or it may require some creativity to discover it.  But once you’ve got the overall structure, there are many smaller sub-problems that must be solved in order to fill in the details and execute (or test) the grand design.  Each of those sub-problems may require some degree of creativity as well – and then the sub-sub-problems and so on, until we get down to operations that that are well understood and completely routine.  Few of us will experience the exciting rush of creativity that goes with creating a major scientific theory or a world-changing invention, but every one of us must do some small-scale creative problem-solving every day.  And, at every level, I would claim, that is just effective problem solving that leads to a surprising answer – not magic.

But what about “flashes of inspiration”?

Perhaps the biggest problem with this “no magic” hypothesis is that it doesn’t match our subjective experience.  Time and again, scientists and other problem-solvers speak of having a “flash of inspiration” or an “Aha! experience” in which some key missing idea comes to them suddenly, often when they are thinking about something else.

There are many well-known examples.   In some cases, the “flash” is triggered by an observation:  Archimedes, settling into his bath and watching the water rise, suddenly understands how to measure the volume – and therefore the density – of a crown that may or may not be pure gold.  Eli Whitney, working on the cotton gin, observes a cat clawing at a chicken behind a wire fence, and suddenly understands that he should snag the cotton fibers and pull them away from the seeds, rather than trying to comb or roll the seeds out of a tangled mass of cotton.

In other cases, the “flash” seems to come from the inventor’s own mind, without any particular external stimulus.  One famous example: after years of trying to understand the structure of benzene, the chemist Friedrich August Kekulé hit upon a model with a ring of six carbons with alternating (but ever-shifting) single and double bonds.  That model worked, explaining all the odd properties of this compound that had puzzled chemists for decades.  Kekulé later wrote that the breakthrough came when, in a daydream, he saw the image of a snake eating its own tail – a common symbol for reincarnation or cyclic renewal in many cultures – and that suggested the dynamic, resonant ring of carbons.  Of course, it would have suggested no such thing if he had not already spent years wrestling with more conventional approaches to this problem.

Some of these famous examples may be apocryphal – explanations invented long after the fact – but they ring true to us because we all have experienced such “Aha!” moments in the small creative tasks we must perform in our daily lives.  It feels like magic, not mundane, grind-it-out problem-solving.  So what could be going on here?

Well, introspection is notoriously unreliable, so we could just dismiss these flashes as an illusion. But I think that there is something deeper going on here, and the flashes we experience provide a clue.

There are at least three different activities bundled into what we call “problem solving”.  First, there is information gathering: learning whatever is known about a problem, what approaches have already been tried, and perhaps running some experiments to learn more.  Second, there is choosing a representation or framework for attacking the problem.  Third, there is “grinding out” and testing the answer: working out all the details to see if the chosen framework can, indeed, provide a successful solution.

The first and third of these activities require conscious, step-by-step mental effort.  We are well aware when we’re doing this kind of work.  But the second step, choosing a representation and approach – a metaphor for the problem, if you will – is different.  It can be a conscious serial process: make a list of all the possibilities and try them out, starting with the ones that seem most likely to succeed.  But this can also be a sort of recognition process: having studied the problem to be solved, we have a mental sketch of its essential features and overall structure.  We then reach into our mental storehouse of schemes and metaphors, and something clicks – an approximate match, not a perfect one.  This is the candidate solution structure that we may or may not be able to massage into a complete answer.

This recognition process is very similar to other recognition tasks that we humans perform without apparent effort: vision, speech understanding, and so on.   Matching a set of observed features against a huge number of possible descriptions or schemas is not an easy task in terms of the computation required – far from it!  But it feels easy to us humans because our brains have powerful, massively parallel hardware to throw at the matching task.  I believe that the same (or similar) recognition hardware is used when we try to find a framework or metaphor that matches a problem.

Because so much is happening in parallel, we do not have a conscious awareness of all the mental effort that this requires.  We just see the candidate answers that pop out.  When one of those answers works, it’s a “flash of inspiration”.  Of course, we do this kind of matching all the time in our daily planning, but we don’t remember the matches that come easily, without a struggle.  The flashes we remember as creative are the ones that we have to struggle to find – often ones that others have struggled with and have failed to find.

In many cases our existing description of the problem will fail to match anything useful in our storehouse of metaphors.  The matcher may get stuck, perhaps because it has latched onto one obvious solution that doesn’t work, and it refuses to let go.  When that happens, one useful strategy is to think about something else for a while, and then come back to the problem.  Or, as noted above, some external stimuli – themselves being fed through the recognition machinery – may take on new meaning and catalyze a match.

So here is the revised theory:

  • What we call “scientific creativity” is just good, effective problem solving that happens to lead to a surprising result.
  • The part that seems creative or magical to us is the selection of the representation or approach. That is fundamentally a recognition process: matching the problem (as we understand it) against a vast store of stored metaphors and techniques. This is computationally demanding, but it happens in parallel, so it feels like a flash. We don’t feel the mental effort required to do the match.
  • These flashes of inspiration hardly ever occur until you’ve done a lot of work to investigate and understand the structure of the problem. That part is perceived as hard work. And, once the flash has occurred, it is of no value until you have done all the detailed “grind it out” work to fit your idea to the problem and verify that it works. That part, too, is perceived as hard work. As Thomas Edison put it, “Genius is 1% inspiration and 99% perspiration.”
  • For a problem that is both difficult and important, many people will try the standard methods and representations and will do all the necessary hard work. So what sets apart the “creative” person is their success in doing the recognition part, and in coming up with an answer that the others have missed.

If creativity were magic, then there’s not much any of us can do to become more creative – the flash occurs or it doesn’t.  But if the above theory is correct, then there are problem-solving strategies that we humans can adopt that might lead us to more creative results, more of the time.  That will be the topic of a future article, coming soon.

  1. This maxim — I’ve seen it worded in various ways — is often attributed to Esther Dyson. []
  2. …Well, a random walk might work if you can afford to invest the immense time and parallelism that we see in biological evolution by natural selection.  But even in natural selection, we see some very clever mechanisms to guide the search, for example by selecting and stabilizing various useful partial solutions and re-using them in various combinations, rather than wandering randomly in the lowest-level feature-space.   This preservation and shuffling of partial solutions seems to be the primary (non-recreational) role of sexual reproduction. []
  3. Some sources suggest that this term is derived from Cockney rhyming slang via “raspberry tarts”. []
  4. I’m certainly not the first person to suggest this.  This idea was “in the air” during my grad-school days at the MIT AI Lab, probably introduced by Marvin Minsky.  However, I don’t know if he would agree completely with the view of creativity I am presenting here. []
  5. I believe that I first heard this idea from Seymour Papert, but I don’t know if it was original with him. []

Leave a comment

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>