AI-driven Customer Discovery to find P/MF

Summary

The goal: achieve the same outcomes as face-to-face CustDev, but at a fraction of the wasted weeks (sometimes: months) it takes to get going.

For startups that haven’t broken-through into scaleup territory yet, the single most important task is to find Product/Market Fit (P/MF). Eric Ries’s Lean Startup gives a great starter, and Rob Fitzpatrick’s The Mom Test gives a fast crash-course in the essentials. The hardest next step for first-time founders is often: how do I find the people to interview? How do I move into Customer Development when I don’t have any customers to talk to? The Mom Test is a flywheel that does nothing if you can’t get it spinning…

Over Christmas we tried a new approach: AI-powered Customer Discovery.

Baseline: Traditional Approach

Google ‘how do I find customers to interview for lean startup’ or similar and you’ll find hundreds of blog posts, podcasts, etc. with concrete ideas, that tend to come down to the same thing:

1. Choose a simple customer profile / persona that you think is ‘probably’ interested
2. Find where those people hang out
3. Go talk to them (cold call / cold approach) – but get intros if you can (warm approach)
4. When you run out of responses, restart with a new (different) profile/persona and try again
5. For the few that respond: arrange a phone/zoom interview, and spend 20-45 mins talking to them

In reality this is time consuming and almost everyone hates cold-sales calls (both making and receiving). So in practice this often collapses down to the simpler:

1. Choose a job title and/or industry you want to sell to in future
2. Look them up on LinkedIn
3. If you have mutual contacts: ask for an intro; if not: send a LinkedIn message
4. Back and forth for days or weeks trying to find a time they’re free
5. Spend 1-2 hours each time in preparing to talk, talking, then writing up what you learned

Improvement: AI-driven replacement

How much of that can we automate? Will it backfire? How do we get 80% of the value for 10% of the effort? Is this too ambitious? How much will it cost to build such an AI?

Three rules:

1. Use AI (LLMs) for what they’re great at (don’t use them for things they suck at)
2. Spend almost zero up-front on automation (no point in using AI here if you lost months building it)
3. Giant wins not small steps (be ambitious – AI is wasted on incremental improvements)
4. Create an enjoyable experience (don’t give people an annoying ‘dumb chatbot’ which has them begging to speak to a human instead)

First target: use AI to improve ourselves as interviewers

Running “Customer Discovery” interviews seems trivial in theory but in reality requires some skill from the interviewer – and a lot of practice (The Mom Test does a great job of injecting you with the basic skills here). I find when I’ve gone a long period without doing Discovery that I’m a bit rusty at it when I join a new startup / try a new venture.

I also frequently work with first-time founders who haven’t run a Customer Discovery process in the past, and need some help coming up to speed with the practices and techniques.

So I built a custom AI to pretend to be a target customer, that we could interview ourselves, and use to practice our Mom Test technique. There’s a cut-down version of it here: https://chatgpt.com/g/g-674f969c4f7081918665ac4c4e66d498-practicing-the-mom-test

Technique

• Use a CustomGPT (requires a paid OpenAI account)
• Configure the CustomGPT with details of the target persona you *think* will like your product
• Configure the CustomGPT with knowledge of CustomerDiscovery, so it can coach you as you go along

I can build a CustomGPT in minutes, live, and make changes instantly (see here for an intro/step-by-step guide). The first iteration here took me about 30 minutes, most of which was defining the customer personas we thought were interested.

Outcomes

I was particularly pleased at how I got ChatGPT giving inline feedback on my interview technique – this example is verbatim (note the text in [square brackets])

OK, that worked great – gave us the opportunity to practice interviewing, and I could tweak the CustomGPT quickly and effortlessly to make it into a more difficult interviewee, or one with a specific persona (although that’s not really needed, turns out the generic example is good enough).

Second target: Can the AI run the interviews?

Let’s flip the script: if the LLM is good at pretending to be an interviewee … can it also be an interviewer? What happens if… ?

There are probably 50 startups out there offering this as a paid SaaS service, but … most such AI problems you can do yourself and save $$$ (and tweak it to your needs). Could we create a CustomGPT that interviews people, summarises the results, then saves them for us to review later? Total cost: $0 (free because I already pay flat fee $30/month for a ChatGPT+ account).

Technique

• Find a community of relevant people, forum, whatsapp group, etc. Make a post asking for people who have opinions on the topic to try talking to our AI and email us their conversation
• Build a new CustomGPT:
  • Re-use the prompts from previous CustomGPT that guided it in customer-discovery interviews (so it understands the theory)
  • Remove the [meta commentary]
  • Insert a specific product-idea it’s trying to explore
• [OPTIONAL]
  • Add a ‘post interview’ phase where it summarises what it’s learnt

Side note

Critical point to make here: CustomGPTs by design give zero information or feedback to the person/account that owns them. i.e. if I send a link to this CustomGPT to someone, and they fill it out, I get … nothing. No record of the conversation. No feedback, no results.

So it’s obviously not a ‘great’ solution – but what if we ask people to email us their chat transcript? ChatGPT/OpenAI has a simple, obvious “Share” link at the top right of each conversation (note: this is one of the ways that Claude/Anthropic falls tragically short as a competitor: after many months they still don’t let you share your conversations):

We tried it both with and without the ‘post-interview summary’. Mixed results there. I feel it has a lot of potential – especially for allowing the human interviewee to spot misunderstandings and correct them – but we weren’t getting great-enough results to keep it enabled in general.

Outcomes

In the space of 2 weeks we managed 30 customer-discovery interviews at a cost of zero face/phone/zoom time. Every moment we had spare could be spent refining the personas and looking for new ‘water cooler’ locations to find such people and engage with them, scout for ones who’d be interested to share their experiences and offer their input on our product.

As a bonus … we have a complete, faithful, transcript of every conversation.

And a negative outcome…

I discovered something unpleasant when doing this: while OpenAI technically allows any account (free or paid) to use a CustomGPT after its been created … they actively hurt the ‘free’ users who try to use it. OpenAI puts extremely low, artificial, usage limits on for free users such that approx 75% of the interviewees who didn’t have a paid ChatGPT account found themselves cut off half-way through the interview – and not once, but many many times, requiring multiple days to complete a 10-message text conversation.

If I’d known we were inflicting so much pain on our generous volunteers I’d have had serious second thoughts about doing it this way. In future I would actively avoid using OpenAI/ChatGPT for this process – I can build a clone of CustomGPTs in about an hour, from scratch, and then deploy it on GPT-4, Sonnet-3.5, Gemini, etc – we’ll be paying for API tokens at that point (instead of CustomGPT which has $0 run cost), but for short conversations like these the total cost will be around $0.01 / interview at most.

Overall

Despite the blip at the end this was an enormous win overall. I’m now excited to deploy a similar approach on other product ideas and see what happens. At some point I’ll probably make my own proprietary clone of CustomGPT to sidestep OpenAI’s dubious rate-limiting practices – but it’s not urgent, especially since several of my current interests (e.g. using AI to discover/remove Tech Debt for engineering teams) have a high correlation between ‘expected customers’ and ‘people who have access to a paid OpenAI account through their work’.