Category: whitepaper

In lean startup, the point of building a prototype, running experiments with your intended audience, and measuring what happens is to LEARN.

• Did you identify the correct audience?
• Did you choose a problem to solve that is both real and significant – is it a problem worth solving?
• Does your solution work, at least in part, at a cost that audience is willing to pay (cost being defined not just as money, but potentially as time or effort)?

That is, are you on the right track?

If you are, great! It’s time to persevere, to make your next investment of resources in running your next round of experiments as you work your way towards  a fully fleshed out, marketable program, product, or service.

If you’re not, it’s time to revisit your assumptions about audience, problem, and solution, and use the data you gathered in the measure stage to see where you were off. Then you pivot your understanding of ONE of those elements, put together another prototype, and try again.

What if you learn you’re REALLY off? It’s OK to decide that you need to kill the project.

In fact, in the American Association of Veterinary State Boards case study related in Lean at 10: Culture Eats Methodology for Lunch, that’s exactly what the AAVSB did in their first iteration through the lean startup process. They learned that a product that had already been launched into the marketplace before starting to use the methodology was not salvageable, and they were able to make the evidence-based decision to kill it.

As I wrote about recently, remember that lean startup doesn’t guarantee success, it guarantees insight.

That insight proved valuable at the AAVSB, as they were then able to make a clear-eyed decision to terminate a failing program without casting blame on the board members who originated the idea or the staff members who had been charged with executing it. Staff wasn’t telling the board, “Your baby is ugly” (or vice versa), the data was saying, “You chose a problem that, while real for your target audience, was not significant for them.” That is, it wasn’t a problem worth solving. Terminating that program then freed up resources – money, volunteer attention, staff time – to work on something more promising.

That’s where the magic happens. You make a small investment and only make subsequent larger investments if the data supports them. You don’t waste significant resources going the wrong way.

Image credit: Joshua Brown on Pexels

What seems like the most difficult piece of the Build-Measure-Learn Cycle to you?

• Putting together a low-fi, low-cost, low-effort prototype?
• Figuring out what measurements will be meaningful?
• Making the persevere – pivot – kill decision?

In my experience, it’s usually choosing your metrics.

Why is that hard?

There are LOTS of options, and it can be difficult to know which ones are actually going to deliver the data you need to prove or disprove your hypothesis about audience, problem, and solution.

The key to designing a good measure is to focus on the assumptions behind what might make your idea a good product or service, not on the idea itself.

The hardest thing to avoid is choosing vanity metrics, things that make you feel good but don’t actually test your hypothesis. It’s tempting to choose measurements of volume – how many likes, followers, downloads, etc. But that’s usually not helpful.

Quoting Innovate the Lean Way:

Take conference attendance. Associations commonly measure the success of annual meetings by number of attendees. But is that really a good measure? What assumption does it test? Attendance numbers don’t tell you what those attendees were trying to accomplish by participating or whether they achieved their desired outcomes. They may have come to learn about a topic, to network, to do business, because they wanted to hear a particular speaker or meet a particular person, because they were intrigued by the host city, or for myriad other reasons. Measuring how many people attended doesn’t tell you anything about whether those goals were met.

Likewise, if you were to launch a mobile application for your association, it’s likely you would immediately begin reporting on downloads. !is is a clear vanity metric, as downloads don’t tell you whether the app is solving a problem your members think is worth solving. Think about how many apps you download because they’re free and you want to try them, but you never use them again.

A better way of testing the success of your mobile app would be to measure the number of members who use the app versus your website to perform particular functions or take advantage of particular services. Do they prefer one platform to the other? Do different cohorts of members use one versus the other? Can you isolate particular features that incline people to use one versus the other? Can you then make adjustments to move you closer toward your goals?

Some guidelines to consider as you think about what to measure to help you validate or invalidate your hypothesis. Good metrics tend to:

• Be a rate or a ratio
• Allow for comparison over time
• Be simple (if it’s too complicated, people can’t measure it, remember it, or use it)
• Be predictive
• Allow you to make changes based on what you learn

Some possible metrics to choose:

• Acquisition: How do potential customers / users discover our program, product, or service?
• Activation: How many potential customers / users take our calls to action?
• Retention: How many one-time customers / users become regular customers / users?
• Revenue: Are people willing to pay for our program, product, or service?
• Referral: Do our customers / users like the program, product, or service enough to tell others about it?

Image source: Lucas on Pexel

One of the common misperceptions of lean startup methodology is that it’s a guarantee that your idea for a solution to one or your audience’s problems will succeed.

It’s not.

• You could’ve chosen the wrong audience, like PayPal focusing on Palm Pilot users.
• You could’ve chosen the wrong problem, like Groupon trying to mobilize people for social causes.
• You could’ve chosen the wrong solution, like YouTube thinking people wanted a video dating site.

In all of those examples, the company was able to pivot successfully.

• Turns out EVERYONE wants to be able to send money electronically to their friends.
• People already have platforms to mobilize for social causes, but (at the time), they didn’t have a platform to mobilize for the cause of saving money.
• Video for dating? No thanks. Video for cute cats? Yes, please!

But what if you can’t?

In Lean at 10: Culture Eats Methodology for Lunch, the American Association of Veterinary State Boards’ first use of lean startup resulted in exactly that: the realization that the product in question was un-pivotable.

“We were used to hitting home runs for our members and affiliated customers,” said Chrissy Bagby, CAE, PMP,
Chief Strategy Officer at the AAVSB. “Then, when we had a launch fall flat, it didn’t feel great.”

The AAVSB ultimately decided to kill that particular product, using the evidence gained through the process to make a data-informed decision.

For many associations, that would be it. “We already tried lean startup, and it didn’t work. Our product was a failure, and we had to pull it.”

What allowed the AAVSB to continue using the methodology?

Their culture. 

The AAVSB team was willing to have difficult conversations, work together to create shared understanding, learn and institutionalize the PRACTICE of innovation (not just the concept), measure their effectiveness, and make decisions accordingly.

In other words, they realized the actual benefit of lean startup: insight.

To quote Guillermo Ortiz de Zárate, related to the National Council of Architectural Registration Boards’ story:

One of the axioms of lean startup is that people can’t be right all the time, so if you’re not pivoting, you might actually not be testing the right assumptions, or you might be choosing metrics that prove you right as opposed to those that will provide new insights. ‘Time to Insight’ has been one of our Metrics That Matter. How long did it take us to fail (learn) when using lean startup versus when using traditional product planning?

It’s that insight that will allow your association to keep iterating and moving towards solving a real and significant problem, a problem worth solving, for one or more of your audiences, at a price (in both money and time) they’re willing to pay.

Image credit: Eva Bronzini on Pexel

I recently participated in a virtual roundtable where the topic of conversation turned to non-dues revenue and the need to experiment with new ideas in a time of rapid change (it may be trite to point out that the pace of change keeps accelerating, but that doesn’t mean it’s not accurate).

One of the participants observed that membership organizations that are experiencing growth generally have a process in place to encourage innovation. And, in fact, his own organization was “leaning in” on innovation, but he was worried they wouldn’t be successful because they lacked a process to guide them.

Of course, I immediately thought of lean startup methodology.

But I also found myself thinking of Jamie Notter’s contributions to the recently-released Spark collaborative whitepaper, Lean at 10: Culture Eats Methodology for Lunch. Jamie astutely observed that, often, associations want to do lean startup, but the culture won’t allow us to.  One of the key cultural challenges he identified was “incomplete innovation.” He went on to point out:

“This is arguably the biggest cultural challenge for implementing lean startup. This pattern, which is very common among associations, is one where a culture values the concepts of innovation (e.g., creativity, future focus) more than the practices of innovation (e.g.,experimentation, prototyping, beta testing). In other words, we’re talking the talk around innovation, but we’re not walking the walk (which is why the innovation is incomplete).”

What are the practices you need to embed in your organization to be able to walk your talk on innovation?

• Testing new ideas
• Experimentation
• Risk taking

Does that sound scary?

It doesn’t have to be.

Learn more about this and the other culture patterns and practices that can impede your innovation efforts, and what you can do to overcome your culture roadblocks and innovate successfully by downloading the free whitepaper here.

Photo credit: Aedrian Salazar on Pexels