How to Innovate πŸ’‘

Innovation comes from rapid experimentation and process rather than pure genius.

How to Innovate πŸ’‘

Issue No. 30

This is a project by Jeremy Brown. I'm a journeyman sharing insights on leading product & engineering teams, building products, and exploring technology.
I will also share occasional updates on my overall project as I build this newsletter and "The Retrospective" (a live show and podcast) in the open.

You can imagine that "Innovation" was a hot topic when I worked at Red Hat's Open Innovation Labs.

My big takeaway was that innovation comes from good processes rather than genius.

In this week's article, I want to lay the foundations for next week. Next week, I want to introduce an excellent framework for innovation that will help turn this "innovation algorithm" into something you can use with your team.

πŸ’¬ In this issue, I cover:

  • πŸ’‘ How To Innovate
    • πŸŒ„ How I See Innovation
    • πŸ”„ The Innovation Algorithm
    • πŸ€” Innovating Is Counterintuitive
    • πŸ§ͺ The Dyson Vacuum Was Based On Over 5000 Failed Experiments
    • πŸ” No Analogy is Perfect
    • πŸ“œ Every Rule Has Its Exceptions
    • πŸ—οΈ Health Foundations Enable Innovation
    • πŸ’» Applying this to Engineering
  • πŸ”¦ Highlight of the Week

πŸ’‘How To Innovate

We often think innovation is something only a genius like Newton or Einstein can do, making it seem impossible for the rest of us.

We often think of successful company founders like Steve Jobs, Elon Musk, and Jeff Bezos as super creative geniuses. The rare few who have "it"!

Is innovation reserved for the lucky few?

I don't think innovation works like that at all!

I think you can innovate if you follow a relatively simple algorithm.

How I See Innovation πŸŒ„

Picture yourself looking out across a landscape.

Photo by Waranont Joe on Unsplash

The hills in the landscape represent problems to be solved.

To solve a problem, we need to develop a solution to get to the top. In our analogy, this is the path to the top of the hill or mountain.

The problem/solution terrain we see is infinite.

And this landscape changes over time because of things happening around it, like the weather.

Photo by Juliana Malta on Unsplash

We can only really understand problems that are close to us.

We can't see over the horizon.

Photo by Andrew Hughes on Unsplash

However, considering the costs and benefits, we can determine which problems are worth solving. This helps us see which solutions will give us the best results. In our example, it's like choosing the best mountains to climb.

There is one temporary optimal solution to any solvable problem. In this analogy, that is the path to the mountain's summit.

There may be local peaks. We may have to backtrack. We make mistakes.

Suppose we have a good process for climbing the terrain and apply our process mechanically. In that case, we will eventually find the best solution.

Photo by Paula May on Unsplash

The Innovation Algorithm πŸ”„

We can approach this optimal solution heuristically (using intelligent guesses) and mechanically (following a step-by-step method).

Our intelligence can make this process faster and more accurate. Still, it is extremely rare that, through pure genius, we can devise a perfect solution.

The process is much more important than talent!

To solve the problem, we need a hill-climbing algorithm. This method starts with a first approximate solution. It keeps making small changes to make it better until it can't get any better.

Or, to put it another way:

  • From our starting point, identify the problems with the highest value if solved.
  • Think of a fast and cheap idea to solve the problem.
  • Run a small experiment.
  • Learn and reflect on failures and iterate incrementally on success.
  • Make a small change to your solution and try again.
I've "stolen like an artist" to bring these ideas to you.

The source of my inspiration here is Pieter Hintjens (RIP), who was a far better writer and thinker than I am. I'm a massive fan of his work.

"The Myth of Intelligent Design" introduces "The Theory of Heuristic Innovation", on which these ideas are based. It's an excellent place to get started with Pieter's writing, and I highly recommend his books.

Innovating Is Counterintuitive πŸ€”

The tricky thing about this algorithm is that though it is simple, it is counterintuitive.

  • We want to go in a straight line rather than iterate.
  • We don't want to show our work until it's perfect.
  • We imagine innovation as a solitary process rather than a collaborative one.
  • We want a 5-year plan, and the CFO wants to make a budget.

The reality is:

  • Innovation happens when people work together and share ideas rather than working alone or following a straight path.
  • Innovation is fluid (like a flowing river) not episodic (a series of lightbulb moments).
  • Great ideas are forged from many good ideas. First, we discovered how to use electricity, and then we invented the lightbulb to use it.
  • And you can't make a 5-year plan for your next innovation.

The Dyson Vacuum Was Based On Over 5000 Failed Experiments πŸ§ͺ

Here is a real-life example of how innovation actually happens, the story of how Sir James Dyson invented his vacuum cleaner taken from the New York Magazine - "James Dyson on 5,126 Vacuums That Didn't Workβ€” and the One That Finally Did":

Dyson's journey to the top of Vacuum Mountain began in 1978, when he decided there had to be a way to build a better device than the popular vacuums of the day. "I started Dyson with an idea: a bagless vacuum that didn't lose suction. It seemed so simple β€” bagged vacuums begin to lose suction as soon as they fill with dust," he told New York. "So, I invented a vacuum that didn't rely on bags, and cyclone technology meant the vacuum wouldn't lose suction." Sound familiar?

The idea came to him after seeing a local sawmill which used a 30-foot-high conical centrifuge that would spin dust out of the air. The same technology, Dyson reckoned, could be shrunk down and built into a vacuum cleaner, omitting the need for a bag and ensuring the device wouldn't lose suction and become less useful over time.

Easier said than done, of course. Dyson would spend the next 15 years perfecting his design, a process that resulted in 5,127 different prototypes. Those years were tough for the fledgling inventor and his family. "By 2,627, my wife and I were really counting our pennies," Dyson wrote in 2011. "By 3,727, my wife was giving art lessons for some extra cash."

"She was wonderful. But most other people thought I was mad," he told Inc. Still, Dyson knew he had something potentially great on his hands and pushed on. "It didn't happen overnight, but after years of testing, tweaking, fist-banging, and after more than 5,000 prototypes, it was there," he said. "Or nearly there. I still needed to manufacture it and go sell it."

No Analogy is Perfect πŸ”

Firstly, I know that no analogy is perfect. I like mountains and hills. I used to do a lot of fell running, and I still love hiking, so this picture resonated with me.

I hope you aren't too busy picking holes in the analogy to miss the points I'm trying to make about working small with rapid experimentation.

Every Rule Has Its Exceptions πŸ“œ

Secondly, every rule has its exceptions. Yes, there are those creative geniuses who "just" invent stuff. Still, more often than not, they collaborated with others on a problem, experimented and built their solutions on other ideas.

For example, the story of Isaac Newton formulating the theory of gravity after being hit on the head by an apple is likely more myth than fact. The legend suggests Newton was inspired to develop his theories of gravitation after observing an apple fall from a tree, prompting him to wonder why apples always fall straight to the ground.

Newton's biographer, William Stukeley, popularized the story in his 1752 memoirs. He wrote about a conversation with Newton, where Newton mentioned seeing an apple fall while contemplating. Still, there's no mention of him being hit on the head. This led Newton to ponder why objects fall toward the Earth.

While the apple tale is charming, Newton's ideas about gravity developed over time through extensive observation, experimentation, and mathematical reasoning rather than a moment of epiphany caused by a falling apple.

Health Foundations Enable Innovation πŸ—οΈ

A final caveat:

It is easier to follow this algorithm with healthy foundations, such as psychological safety and trust, healthy conflict, commitment, accountability, and a focus on collective outcomes.

Applying this to Engineering πŸ’»

Our main goal in making software should be to create the best processes to help us innovate and follow them step by step.

Easier said than done!

Thankfully, we don't have to start from zero; we can speed up our work using well-known good processes.

That's it for this week; next week, I will introduce you to an excellent framework for innovation that helps turn this "innovation algorithm" into something you can use with your team.

πŸ’¬ If you had some thoughts while reading this, I would love to hear them in the comments.

πŸ”¦ Highlight of the Week

One of the best CEOs I worked for, Jim Whitehurst at Red Hat, had this to say about innovation and planning.