Good AI, quick AI: the PM’s guide to building and validating AI products

tl;dr: in this post, I’ll share how I’ve learned to work with data science and AI teams in an agile environment, and how you can actually ship AI products and not go into what seems to be an endless stream of ‘research’.

Out of all the jobs that product managers have, working with dev seems to be a mostly ‘solved issue’. We’ve been doing it for decades, we have Agile and Scrum, and Kanban (I won’t go into the discussion of how alive/dead all of these are).

Working with dev is so ‘simple’ today, that we made it into a junior role and titled it ‘technical PM’ (at least where I’m from, don’t be offended if you’re actually an awesome, super-senior technical PM).

Working with data science teams, however, still has its issues. Even though the tech world has been all about AI for at least the past 5 years, it still causes a lot of confusion for PMs, me included.

It’s getting better, though :)

The hard thing about AI

The hard thing about AI is that it’s, well, hard. It’s hard for multiple reasons:

• It’s more technical than ‘regular’ development, so we don’t always understand what’s going on as well
• It usually includes more ‘research’ and less ‘development’
• It often takes more time than developing a regular feature
• It’s less certain that we’ll actually succeed in building what we intend to
• AI tools and best practices are young, and still change way more often than dev tools and frameworks (look, we’ve managed to mostly stick with Angular/React for almost a decade now!)

These reasons are super un-agile-like. You’re adjusting your roadmap every few weeks, and then you go into a meeting with the AI team, where they’re sharing their plans for the next 3 years or so. They can break it down, of course — the first phase is not expected to take more than 3–4 months. It’s like building software with people building the light rail in Tel-Aviv! (for you non-Israelies, the Tel Aviv light rail has been planned, on and off, since about 1930. We’re still waiting).

The reason it usually takes so long to build AI features, is that in order to get the best results, you usually need to be very vertical-specific and collect the most relevant data. When I worked on a product for lawyers in the UK, we couldn’t really use any US data, otherwise the algorithm would start talking with a strange accent (and be less accurate).

After you have the best data, which is a super complicated task, you start adjusting the algorithm. You go and research the latest articles, try them out, benchmark the results, go and change a bunch of parameters and run in again and again and again. Well, the data science guys do that, while you, the PM, have long lost hope that the product will ever come into existence.

The fun thing about AI

There’s no way around it — this is usually the process for getting the best results. But wait — who said we have to start with the best results? In some cases, sure. I’d hate for Elon Musk to release his self-driving teslas with 2nd-tier algorithms, but in many cases, 2nd-grade is okay to get you started.

And the thing is that today, you have an out-of-the-box algorithm for just about anything. Actually, you have about 100. And they’re often pretty awesome. It might not be the SOTA, and will be less sexy for the pros, but hell, what an awesome way to find out if you users like the feature before you put about 50 man-months into it.

“Ha-ha!”, you say! “50-man-months, what an outrageous exaggeration!”. Well, no, that’s just about how much time we invested in *a first version* of a product in one of my previous companies. Also, please don’t shout “ha-ha” at a Medium post, you’d look ridiculous.

Building quick

Working on my current product, a knowledge-sharing app, our goal is to help companies organize their knowledge. While we’ve built some awesome manual organizing features, having tons of content means a lot of manual work, which is a bummer for the customers, and bad business for us.

Obviously, being a young startup, the last thing we want to do is go into 6–12 months of data collection, research and iterations before we even validated the need. Instead, our process is a lot quicker, much more similar to the classic agile dev process:

First, we talked to customers, defined the needs, the regular shabang. We then quickly researched the top services/libraries for what we needed. In our case, we were looking for keyword extraction from images and text, as well as synonyms detection to prevent similar keywords.

We defined our test set and required results (e.g., the sample images that represent what customers are using, and the keywords that we’d give them), and then went for a 1-week trial and error process, playing around with the different tools and parameters (and with AI, often parameters are where the magic is).

Within about a week, we got some really outstanding results, both for the image recognition, and for synonym detection, which could identify keywords such as ‘contracts’ and ‘agreements’, or ‘customer’ and ‘client’, and suggest them as similar. Really cool stuff, all done with out-of-the-box NLP libraries.

Working with AI people

There’s one thing I should get straight: this process does not replace data scientists. Actually, it’s really hard to get to even okay results without deep knowledge of how these algorithms actually work. In order to implement them, there’s an entire world of data extraction and pre-processing that is really different than adding buttons to a task list.

Moreover, doing the quick-and-dirty actually helps DS in their future research, since they will get a better sense of what the product should actually achieve, and also see what didn’t work in the process, saving them a lot of time later, reducing their research time from infinity to half-infinity (kidding guys, you’re the bomb!).

And most important: this MVP approach is meant to validate a solution and give quick value. If and when this experiment succeeds, very often you would need to go into a deeper phase of research to get really good results. All I’m saying is —get your feet wet first, don’t jump straight into the deep end.

Wrapping up

AI is super cool, and in some cases, it might add that little bit of magic to a product. It certainly did so for us (still in QA when writing this, but sign up to check it out for yourself), and while it’s definitely not a silver bullet to solve all of your problems, it is something worth exploring.

As PMs, we should not be worried about getting our hands dirty. We should work closely with the data science team, not in spite of, but because their work is relatively unfamiliar to us.

Otherwise, you’re running the risk of ending up in an ‘I’ll let we know when we’re ready’ situation, leading to confusion and uncertainty with the team, the customers, and with management.

When I first started working on AI products, it all seemed really complex and confusing, so I hope this helps someone make sense of things. If it does, clap away so others can also see this 👏👏👏.