Where Good Ideas Come From

Reef, City, Web

Darwin visited Keeling Islands in 1836 and was surprised at the "infinite numbers" of organic beings. He too was taken by the sheer beauty of the reef world like the others. But his mind was already thinking about the variety of life concentrated in such a small area. What was even more surprising is the lack of flora and fauna on the island itself. #Darwin's Paradox #FFW

Many different life forms occupying a vast array of ecological niches, inhabiting waters that are otherwise remarkably nutrient-poor. Coral reefs make up about one-tenth of one percent of the earth's surface, and yet roughly a quarter of the known species of marine life make their homes there.

What is even more interesting is the persistence of organisms that build the coral reef. No matter how violent and powerful the waves were, the coral reef manages to overcome by breaking down and becoming stronger.

Max Kleiber discovered the law of Negative Quarter-Power Scaling. Plotting mass against metabolism on a logarithmic grid will result in a straight line - from rats and pigeons all the way up to bulls and hippos.

One lovely consequence of Kliebar's Law is that the number of heartbeats per lifetime tends to be stable from species to species. #FFW

Geoffrey West discovered that Kliebar's Law even applied to the energy and transportation growth of city living. But he also found that every datapoint involving Creativity and Innovation followed the law, but the quarter-power law was positive, not negative. As cities get bigger, they generate ideas at a faster clip.

The 10/10 rule: A decade to build the new platform, and a decade for it to find a mass audience. The rule applied consistently to the growth timeline of color televisions, HDTV, radios, VCRs, Cell phones, personal computers, GPS navigation devices. It even applied to Graphical User Interfaces.

The most elemental yardstick of measuring innovation, where technology is concerned, is the number of jobs that technology lets you do. The more number of jobs accomplished, the better the rate of innovation. With that metric, Youtube was many times more innovative than HDTV, though HDTV was a more complicated problem. Youtube took the 10/10 rule and made it 1/1.

Both the city and the Web have proven to be good innovation generators. A series of shared properties and patterns recur again and again in unusually fertile environments.

Watching the ideas spark on different scales - The ecological history of the reef, the sociology of urban life, the intellectual biography of a scientist - reveals patterns that single-scale observations easily miss or undervalue. The Long Zoom vantage point gives us new facts, not just metaphors to compare. It lets us see that openness and connectivity may be more valuable to innovation than purely competitive mechanisms.

We leverage more by connecting ideas than by protecting them. Good ideas may not want to be free, but they do want to connect, fuse, recombine.

The Adjacent Possible

Stephane Tarnier conceived the idea of an incubator while on a visit in a zoo, and drastically reduced infant mortality rates in Paris. But incubators remained inaccessible in the developing world because they kept breaking down. So Timothy Prestero, an MIT professor and Jonathan Rosen, a Boston doctor, created an incubator - The NeoNurture Device - from automobile parts, so that both a local supply of parts and a knowledge of repair were both available. #FFW

Good ideas are constrained by the parts and skills that surround them. Ideas are works of tinkering; they are built of whatever junk is available around us. We take the ideas we have stumbled upon and assemble them into a new shape.

Stuart Kauffman suggested the phrase The Adjacent Possible to capture both the limits and the creative potential of change and innovation. The Adjacent Possible is a collection of potential states in which a system can reinvent itself. It is always present in the background, expanding and contracting as the system evolves.

The Adjacent Possible is not an infinite space or a totally open playing field - the list of potential first-order reactions is bounded. It conveys that only certain changes can happen in the current state of a system, even though the set of all possible changes is infinite.

The boundaries of The Adjacent Possible grow as one explores those boundaries - each new combination ushers new combinations into the adjacent possible. It is like building a palace by opening one door at a time - each opened door leads to more doors to be opened.

One way to think about the path of evolution is as a continual exploration of The Adjacent Possible. Opposable thumbs in humans opened the possibility of creation and use of finely crafted tools and weapons. The semilunate carpal boon in Velociraptors opened up the possibility of wings and flight many million years later. The entire biosphere has been one persistent explosion of The Adjacent Possible. #FFW

There also seems to be a continuum between natural and man-made systems in The Adjacent Possible. From the first chemical reactions to carbon-based life forms to plastics - the same carbon atoms have been reassembled into increasingly complex forms by the relentless push against the adjacent possible.

The mystery of Darwin's Paradox ultimately revolves around the question of why a coral reef ecosystem should be so rich in variety while the surrounding waters of the ocean lack that same marvelous diversity. Similarly, big cities allow far more commercial exploration of the adjacent possible than towns or villages, allowing specialization in fields that would be unsustainable in smaller population centers.

Pattern of The Multiple - Different people separated by geography, language, and background, coming up with the same idea around the same time. Good ideas are built out of a collection of existing parts. Some parts are conceptual: ways of solving problems or new definitions of what constitutes a problem in the first place. Others are, literally, mechanical parts. This suggests that it was the availability of parts that caused these ideas to bubble up at the same time.

Ahead of time ideas usually end up as short-term failures because they are not part of The Adjacent Possible yet. The idea simply doesn't have the right spare parts. If Youtube were created in 1995, it would have failed because the ecosystem necessary to view Videos was not yet in place. #FFW

All of us have our own private versions of The Adjacent Possible. We are surrounded by potential ideas of new ways of doing things. The trick is to figure out ways to explore the edges of possibility around us. This can be a change in the physical environment, a special kind of social network, or a new way to seek and store information.

Innovative environments are better at helping their inhabitants explore the adjacent possible, because they expose a wide and diverse sample of spare parts and they encourage novel ways of recombining these parts.

Part of coming up with a good idea is discovering what those spare parts are, and ensuring that you're just not recycling the same old ingredients. The trick to having good ideas is not to sit around in glorious isolation and try to think big thoughts. The trick is to get more parts on the table.

Liquid Networks

A good idea is a network. A new idea is a network of cells exploring the adjacent possible of connections that they can make in your mind. A new idea is not a single thing; it is more like a swarm.

There are two preconditions to be satisfied for ideas:

• Network size: The network should have a huge number of connections.
• Network Plasticity: The network should be able to learn and adopt new configurations.

The creative brain behaves differently from a brain performing repetitive tasks. The neurons communicate in different ways, and the networks take distinct shapes. To make your mind innovative, you have to place it inside environments that share that same network signature: networks of ideas or people that mimic the neural networks of a mind exploring the boundaries of the adjacent possible.

The computer scientist Christopher Langton observed that innovative systems have a tendency to gravitate towards the edge of chaos: The fertile zone between too much order and too much anarchy. The liquid state represents the perfect medium to explore the adjacent possible. #FFW

In low density networks, ideas come and go. In dense networks, good ideas have a natural push to propagate. There is, as the economists call it, information spillover, with the dense network itself acting as the medium of storage.

The concept of Double Entry Accounting first became popular during the age of Renaissance in Italy, but the curious fact is that no one owned the double-entry accounting itself. It led to the rise of early merchant capitalism, but the idea was too powerful not to spill over into other nearby minds, or be owned by a select few.

A society organized around marketplaces distributes decision-making authority across a much larger network of individual minds. Cities and markets have more minds working for them to explore the adjacent possible, than the "authorities" can ever hope to match.

This networked innovation is not the global brain or the hive mind. This is not the wisdom of the crowd, but the wisdom of someone in the crowd. It's not that the network itself is smart; it's that the individuals get smarter because they're connected to the network.

Arthur Koestler, in his book The Act of Creation, and Thomas Kuhn, in his book The Structure of Scientific Revolution proposed that the act of innovation happens exclusively in the mind, without any relevance to the habitats that sustain or encourage innovation. But Kevin Dunbar, a psychologist at McGill University, approached the problem of understanding the act of creation in a different way - he studied innovation as it happened. #FFW

Dunbar's study showed that isolated eureka moments were rare. The most important ideas instead emerged during regular lab meetings around a conference table. Such meetings create environments where new combinations can occur, where information can spill over from one project to another. When you work alone, your ideas can get trapped in place, stuck in your own initial biases. Group conversations turn that private solid state into a liquid network.

While too much order in an office stifles innovation, open office plans are not great either because they introduce too much chaos. A more erudite approach of building the solid structures to be fluid and change as per needs is more beneficial. MIT's Building 20 and Microsoft's Building 99 are examples of this thought process applied in real-life settings. These buildings are designed to treat information spillover as a feature, not a flaw.

Slow Hunch

An FBI agent named Ken Williams sent a memo in July 2001, indicating that students enrolling in aviation colleges with strong ties to islamic radicalism may perform terrorist attacks in the future. The email was largely ignored, was lost in transmission and never made it to the FBI boss. #FFW

Dense, liquid networks allow information to flow smoothly along multiple unpredictable paths. These interconnections nurture great ideas because most great ideas come into the world half-baked, more hunch than revelation.

Liquid networks allow half-baked ideas to connect, and consequently, help them bake.

^^Hunches that don't connect are doomed to stay hunches.^^

Snap judgments are rare in the world of innovation. Hunches that turn to important innovations unfold over much longer time frames, so are very fragile and easily lost to the more pressing needs of the day-to-day issues.

The long incubation period that follows these hunches is also their strength because true insights require you to think about something that no one has thought before in quite the same way. The hunch usually stays in the back of the mind, not being engaged directly, but growing over time.

^^Sustaining a slow hunch is less a matter of perspiration than of cultivation.^^ You give the hunch enough nourishment to keep it growing, plant it in fertile soil, where its roots can make new connections. And then you give it time to bloom.

Darwin had the hunch of natural selection well before his claim of having hit upon the idea after reading Malthus on Population, as referenced from his notes. But hunches were like that; the gradual clarity that emerges is crystallized into a turnkey moment in time. It is constructed so probably because it is easier to convey, and it is easier to remember. #FFW

The evolution of the slow hunch is also visible in Darwin's early exploration of the principles of natural selection. Throughout his voyage on the Beagle, he meticulously recorded and published notes on the geology around the islands, instead of the zoology present in them. The hunch that natural selection played a hand in the incredible diversity on these islands was on a backburner in his mind.

Keeping a slow hunch alive is challenging - they pass in and out of our memory too quickly. So part of the secret of hunch cultivation is simple: ^^write everything down^^.

Darwin's detailed notes helped him reference notes in the past and combine them with the present knowledge. This process, which was called "commonplacing" at that time, was about transcribing everything of interest in a commonplace book.

The commonplace book, maintained by other stalwarts too, seems to provide enough structure to note thoughts, but prevents too much order so that serendipitous discoveries can lead to new insights.

Reading and writing are both necessary to keep the hunches alive. Every rereading will bring with it new kinds of revelations. Forgotten hunches will connect in novel ways to new thoughts. You need a system for capturing hunches, not necessarily categorizing them.

Tim Berners-Lee invented the World Wide Web from a slow hunch he gathered over two decades. It started when he read Enquire Within Upon Everything in his childhood years, but the hunch got stronger and was cultivated over many experiments done around work in CERN. #FFW

Cultivating hunches extends beyond the private dominion of memory and the commonplace book. They need a work environment that allows for experiments and information networks that let those hunches travel to other minds, where they could be augmented and polished.

Serendipity

Hunches need environments where new connections can be forged. The brain, for example, acts as a great medium by triggering memories and associations in a chaotic, semi-random fashion.

This kind of chaos is also visible in the Biosphere. Nature evolved sexual reproduction as a preferred strategy because that maximized the rate of innovation. Speed and simplicity were sacrificed for creativity.

The water flea Daphnia reproduce asexually in favorable environments, but switch to sexual reproduction when the conditions get tough. An All-Female population switches to producing males under difficult conditions. This is a strategy not only to produce sturdier eggs with sexual reproduction, but also to innovate and induce changes in its gene pool to adapt to the new challenges. #FFW #Heterogamy

When nature finds itself in need of new ideas, it strives to connect, not protect.

Serendipity, by definition, is through unlikely encounters, possibly across disciplines. But unless the environment is ready, these discoveries will slip by because they don't have an anchor. These environments could be one's mind, an organization, or the society itself.

How does one construct the right conditions for serendipity? One way is to go for a walk. Disengaging from tasks can put your mind in the right state to process and recognize these associations.

It's difficult to consume new ideas in the form of reading if it's done as a fringe activity around work. It may be better to block off a chunk of time and race through material quickly, so that we give a chance for thoughts to network and interconnect.

Technology enables us capture information in one single digital trove. They may be things we read, our notes, quotes from other books, or blog posts. When combined together, the chances of colliding with something different - something from the past that is forgotten - is increased.

Software that increases the chances of this collision has two characteristics: First, it should be able to link up two similar, but disconnected, ideas from different parts of the knowledge base. Second, because everything that went in was constructed by hand, what comes out has a very good chance of being relevant to us.

An irony of the serendipity debate: the thing that is being mourned has actually gone from a fringe experience to the mainstream of the culture.

The web, if anything, has increased the chances of serendipitous collisions, not decreased it. Filters have become necessary because there's too much noise, too much chaos.

The secret to organizational inspiration is to build information networks that allow hunches to persist and disperse and recombine.

Information networks tap into both individual as well as collective intelligences. The Individual hunch is bolstered by the group, by connecting it to other ideas or pushing it up to the top of the pack. These systems, that store ideas and make them public, create an architecture for organization serendipity: They give good ideas new ways to connect.

Error

Lee de Forest discovered the triode, the precursor to all logic gates, by error when he assumed that a spark gap transmitter was causing a flame to be amplified. Wilson Greatbatch grabbed the wrong resistor and ended building a device that could simulate a beating heart by triggering electrical impulses at the right instant in time. Louis Daguerre discovered that Mercury could produce the perfect image on a plate, thereby paving the way to modern photography. #FFW

Errors are not side-effects that happen on the journey to success. Errors are a pre-requisite to eliminate the incorrect assumptions and to guide the inventor on to the right hunches.

Being right keeps you in place. Being wrong forces you to explore. Being wrong on its own does not unlock new doors in the adjacent possible, but it does force us to look for them.

The trouble with error is that we have a natural tendency to dismiss it. Arno Penzias and Robert Wilson thought their telescope was faulty for more than a year, without realizing they were dealing with cosmic background radiation. #FFW

Conference tables tend to turn errors into insight, because others may conceive of novel ways of interpreting the same result. Being too close to the problem actually hinders hindsight.

Too much order destroys creativity. Noise, introduced in some form that cannot be distinguished as noise easily, tends to improve creativity. Good ideas tend to emerge in environments that contain a certain amount of noise and error.

The incredible diversity we observe on earth has been made possible by errors. Natural selection has a constant source of new possibilities to test just because some errors are introduced during mutations or transcription mistakes during replication.

Rare genetic disorders are actually mutation errors that have disastrous consequences. These errors are unfortunate side-effects of the natural process of evolution.

Studies suggest that each time parents pass their DNA to a child, that genetic inheritance comes with roughly 150 mutations. Evolution has built elaborate mechanisms to protect the sanctity of the genetic code, but has left a small window open for such mutations to occur. #FFW

Innovative environments thrive on useful mistakes but suffer if quality drops below the acceptable threshold. Being correct is like the phase-lock states of the human brain - all the neurons firing in perfect synchrony. But for innovation to happen, we need to have a small bit of noise firing random sequences intermittently.

Fail faster is a useful for the same mantra. Mistakes are not the goal, but they are inevitable steps on the path to innovation.

Exaptation

Exaptation - refers to the shift in the function of a trait during evolution. A feather adapted for warmth is exapted for flight.

Natural selection doesn't give good grades for effort. Nature has the same tendency to reuse old parts and create something new.

History is full of examples of innovations that are simply exaptations. Charles Babbage borrowed punch cards from the power loom for his Analytical Engine. The triode was supposed to amplify a electromagnetic signal, but proved to be a effective on-off switch. #FFW

The lobe-finned fish Sarcopterygii began exploring life at the water's edge 400 million years ago. It then had a small swim fan at the end of its fins, supported by narrow rays of bone. Eventually, the swim fan turned into an autopod, the basic architecture of all mammalian ankles and feet. The autopod itself exapted over time into fingers for grasping and wings of the Archeopteryx. In some cases, the autopod exapted backward as flippers of seals and sea lions. #FFW

Exaptations help us explore behind the doors opened by error and serendipity.

All decisive events in the history of scientific thought can be described in terms of mental cross-fertilization between different disciplines - Arthur Koestler - The Act of Creation

Cities create an environment ripe for exaptation because they create a liquid network where ideas get exchanged freely and frequently. This is one explanation of the superlinear scaling in urban creativity. Ray Oldenburg called these "third places" - a place between home and office, where pioneers from different disciplines would rub shoulders and exchange ideas, allowing random collisions to spark creativity.

The most creative individuals consistently had broad social networks that extended outside their organization and involved people from diverse fields of expertise - Martin Ruef, a study of relationship between diversity and innovation.

Mark Granovetter proposed the term Strength of weak ties for such networks. But the benefit is not just from the extended network. It is also from the diversity or variety in the backgrounds of people in the network. Information arriving from these weak times will be from different contexts, an idea-space as called by Richard Ogle. Idea Space is a complex of tools, beliefs, metaphors, and objects of study. These weak ties also help exaptation of ideas.

Innovators tend to have a lot of hobbies. The slow cycling of the mind through these tasks allows ideas to disseminate between different hobbies.

Chance favors the connected mind.

Platforms

Coral reefs are not just platforms that one stands on, but they are platforms that create a habitat for millions of other species. #FFW

Some organisms have a disproportionate impact on its ecosystem, and they are called Keystone Species. But there is a different term for species that create the habitat itself: Ecosystem Engineers. Beavers are such animals; they transform temperate forests into wetlands by felling poplars and willows to build dams. The Scleractinia Polyp is another example: it builds an entire reef as a side-effect of its life. #FFW

Biological platforms, like the natural reef, thrive on inventive collaborations of density, than on competition. The waste of one organism is the nutrition for another.

The reason why the shallows of the reef is thriving with organic life in contrast to the desert-like nature of the atoll, is because of the reef acting as a platform. Not only does it support life, but it acts as a platform that in turn generates a habitat.

Twitter is another platform that has spawned a whole habitat of applications around the 140 character message. #FFW

The Fourth Quadrant

Willis Carrier's story of inventing air-conditioning systems seems like an outlier to the whole story of innovation - a slow hunch, liquid networks, serendipity, and error. Is he an anomaly? #FFW

Capitalist economies have turned out to be the driving force across the world because they seem to have a better track record of improvement. But it's most important force is the market's relentless drive towards novelty and innovation.

The four quadrants of evaluation:

• Market/Individual: Private Corporations, Solo Entrepreneurs
• Market/Network: Marketplace
• Non-Market/Individual: Amateur Scientist, Hobbyist
• Non-Market/Network: Open-Source, Academic

It is in the nature of good ideas to stand on the shoulders of giants who came before them, which means that by some measure, every important innovation is fundamentally a network affair. To understand where these good ideas came from, one has to zoom out and take a time-lapse view.

Between 1400 - 1600 AD, most innovation clusters in the third quadrant: Non-market individuals. Information networks are slow and unreliable, and entrepreneurial economic conventions are poorly developed.

Between 1600 - 1800 AD, innovation shifts from the third quadrant to the fourth quadrant, as the information network grows stronger.

Between 1800 - present, the fourth quadrant has emerged as the overwhelming source of innovation, beating the assumption that capitalism and wealth-driven corporations were the places of good ideas.

You could develop small ideas in the seclusion of a locked room, cut-off from the rest of the world. But to make major incursions into the adjacent possible, you needed company.

Markets would have been the major contributors to innovation, if we had not erected walls around ideas and tried to protect them with the intent of benefiting them. We have purposefully created an inefficient system in the network of ideas, to gain from the imbalance in knowledge. Non-market fourth quadrant does not have this problem.

Open networks of academic researchers create emergent platforms where commercial development becomes possible.

The fourth quadrant has also been helped by the increased flow of information. The internet has effectively reduced the cost of information transmission to zero.

We tend to think Academics are steeped in their ivory towers and rarely corroborate with reality, but the fourth quadrant suggests that the free dissemination of ideas that happens in academic circle does result in good ideas. The lack of capitalist motivations and the presence of other kinds of rewards (tenure, professor roles) is enough to drive the dissemination of information.

It is not about whether you work in the private or public sector. It is about how you approach the work you are doing.

• Go for a walk.
• Cultivate hunches.
• Write everything down, but keep your folders messy.
• Embrace serendipity.
• Make generative mistakes.
• Take on multiple hobbies.
• Frequent coffeehouses and other liquid networks.
• Follow the links.
• Let others build on your ideas.
• Borrow, recycle, invent.
• Built a tangled bank.