Making or Breaking AI Products: The Overlooked AI Gap in Product Management ByHenrieke Max

Summer 2023, it’s quarterly planning time for my org at Delivery Hero. I massage my temples in an effort not to drift off mentally to the upcoming vacation, and use what brain juices I have left to absorb the discussion. ‘But are we doing something with AI?’, a product director enquires. I feel an inner chuckle. Even a tech-native company such as ours with a mature product org is not immune to AI-for-the-sake-of-AI conversations.

Years before my career in tech and data science began, I was working in the Policy team of the European headquarters of a global humanitarian aid organisation. In the simplest terms, my job was to advocate for greater adoption of cash delivery, rather than goods delivery, to people hit by conflict, war and natural disasters. The evidence that providing cash (rather than material forms of aid) was more effective was already robust then. Yet, screening and targeting eligible individuals or communities to receive cash was time-consuming, leading to delays in delivering aid in situations where time is of the essence. Much effort was invested to deliver cash faster, more securely and more cost-efficiently using technology such as blockchain. Leveraging algorithmic tools to determine eligibility was still in its infancy, however. Fast forward a decade, the non-profit organisation GiveDirectly has creatively combined different types of data, from mobile phone usage patterns to satellite imagery and socioeconomic household surveys, to more swiftly identify eligible cash clients [1,2,3]. Using intelligent targeting, they delivered 300 million USD to vulnerable clients in 11 countries in the COVID-19 pandemic, achieving a 30% reduction in administrative costs [4]. During a 2022 US hurricane, the same organisation served over 3.3 million USD to around 4.7k affected households within hours and days, rather than weeks and months [5]. Yet despite obvious benefits, the adoption of machine learning for cash aid targeting across the public and non-governmental sectors has been cautious and slow.

Transitioning from the non-profit to the corporate sector has shaped my view of the AI hype of recent years. The corporate sector is leading the way in investing in AI, with a 25.2 billion USD investment in generative AI in 2023 alone, this being but a quarter of the total AI private sector investment [6] Arguably, returns are lagging behind investment. This is sometimes attributed to the so-called ‘AI skills gap’, referring to the discrepancy between the current capabilities of a workforce and the advanced technical know-how required to extract value from AI. While that can be part of the equation, I argue that headway in regard to core technical skills has been made, but we continue to face an AI product management skill gap. Many companies become so focused on proving the technical feasibility of AI that they lose sight of the purpose and the intended audience, resulting in significant resource waste. Irrespective of sector, a dedication to what consumers, users, actually need, as well as a deep understanding of the market conditions in which users operate, makes AI-powered products successful. In the non-profit world, strong value alignment, the scarcity of resources and pressure of public accountability and limited access to technical expertise act as organic safeguards from something-with-AI. This does lead to slow adoption and underutilisation of AI, but the inclination to focus on societal impact improves the odds for developing usercentric solutions when resources allow. The corporate world, on the other hand, is at risk of malinvesting or overinvesting, losing sight of genuine user needs and a problem-driven approach. Companies need to pay more attention to this risk in systematic ways. My goal is to illustrate why, particularly in the for-profit world, the biggest AI gap lies in AI product management. We need conscious investment in data and machine learning product management capabilities. These have been largely neglected as corporates upskill or make strategic engineering and data hires. I’ll deep dive into this argument with three concrete steps on what filling the gap could look like:

1. Build awareness for the full spectrum of reasons AI-based products might fail

2. Double-down on upskilling for AI product management

3. Make Product the glue of your AI strategy

BUILD AWARENESS FOR THE FULL SPECTRUM OF REASONS AI-BASED PRODUCTS MIGHT FAIL

AI-powered products have more potential points of failure than products not relying on models. Especially in the tech industry, technical or business failures are more likely to be anticipated than those relating to human nature.

Business: Products leaning heavily on AI components, hereafter referred to as AI products, require high upfront investments. These could relate to data sourcing and curation, time to reach a good-enough starter model and procuring the tools needed to productionise and maintain models at scale. Such products risk failing if they cannot deliver incremental value from early on or even lack the ability to demonstrate satisfactory results in later stages, losing leadership faith and support. A recent example is the discontinuation of the IBM Watson-McDonald’s partnership on AI-enabled drive-thru orders in summer 2024. Reports of mishaps, such as adding uncalled for quantities, confusing distinct user orders or unsolicitedly adding unconventional extras had been amassing. Three years following the start of the work on automatic order taking (AOT), the quality of user experience had not reached adequate levels to reduce labour costs while keeping the user experience consistent. The automated voice-ordering was fully halted for the foreseeable future[7]

Technical: AI models can behave in unpredictable ways. They may perform excellently during testing, then fail to reproduce the same performance once live on the product. How users will interact with AI-generated outputs is highly uncertain, and can influence the model performance. A much-cited example is Microsoft’s chatbot Tay, which groups of Twitter users deliberately targeted with racist and misogynist comments, leading the bot to produce more than 95k tweets of the same nature [8,9]. Engineering teams must be prepared to monitor systems closely, implement pre-cautious content filtering, and promptly make adjustments, such as adding additional safeguards, once a model has gone live. Inability to respond quickly can become a deterrent to user adoption or erode organisational faith that the product can work.

Human: Understanding the user context, preferences and subjective benchmarks is critical. AI products may be launched or discarded prematurely when judged solely by standard model metrics or by whether they objectively solve a problem.

• First, the performance of a model is usually measured in mathematical terms, with wellestablished machine learning (ML) and deep learning (DL) success metrics. However, model performance is both subjective and relative. In many use cases, users determine how good is good enough, using their own experience as a benchmark. A mediocre model could be sufficient if users perceive it as a relative improvement to what they knew to date, a flawless model can fail if concerns around safety, transparency, social stigma, cultural norms or other personal or social factors impede adoption. One example of this is the story of Zume Pizza. With a hefty 450 million USD in venture funding, Zume aspired to automate pizza making and delivery using robotics [10]. Their core value proposition did not resonate with pizza consumers, as the end product was not tastier, more affordable, delivered faster or in any other way better than traditional pizza prepared by humans. Had Zume focused on automating pizza making for restaurant partners or other upstream users in the supply chain who may have benchmarked the cost and speed of human-made pizza against robot-made pizza as the primary success metric, their product may have been evaluated more favourably. Zume is also a prime example of how an abundance of funding can deter genuine problem-solving and result in overindexing for AI.
• Second, not every task that can be simplified, automated, or enhanced by AI should be, particularly if it replaces the labour of human users. Data scientists (DS) and machine learning engineers (MLE) may have an inclination to replace effort with a model wherever it is technically feasible (i.e. the data is available, of good enough quality, the deployment infrastructure permits it). And yet, technical feasibility does not trump user desirability. As humans, having a sense of mastery and agency gives us purpose. Convenience or ease can be a selling point in some contexts, in others it can deter adoption. Products that limit (human) users’ agency or autonomy can feel disempowering. Some AI products may attempt to facilitate users’ life to the extent that they challenge users’ expertise, reduce transparency, or cause them to lose touch with their ‘gut feeling’. Even if an AI product addresses an important problem, its acceptance hinges on whether users have sufficient trust and decide that the gains of the AI solution are greater than its trade-offs. This has been heavily misjudged by many, including FAANG [11] companies. A fitting illustration is facial emotion recognition (FER) technology. In 2017, Meta, then Facebook, patented a technology able to watch users through device cameras to identify their emotions based on their facial expressions as they browse social network content [12]. Applications are manyfold, and include leveraging users’ emotional states to determine which content and ads to display to them, or back-inferring the nature of existing content-based users’ emotions. While the business use cases are clear, users have spoken out against the usage of their, literally speaking, emotional data, by Meta and other tech giants. The dominant user sentiment was that this poses a violation of their rights as data subjects. There does not seem to be a user demand for this extent of personalisation, and Meta has since shelved not only its FER efforts, but even ceased features using facial recognition from photos and videos [13]

[14]A key message from Parasumaran and Riley’s 1997 paper on ‘Humans and Automation’ is transferable to AI product management today: AI products must support, not supplant users’ abilities. PMs need to foster an appropriate level of both trust and critical engagement in users, so they make use of the AI product, yet understand its limitations.

DOUBLE-DOWN ON UPSKILLING FOR AI PRODUCT MANAGEMENT

Software engineering (SWE), ML and DL are increasingly interwoven, yet distinct crafts. Traditional SWE applications without AI components come down to providing instructions in the shape of fixed, hard-coded rules of behaviour. AI-applications fundamentally differ in that they learn from the past and may continue the learning to improve or remain relevant to the evolving needs of the user. AI products are therefore a ‘living thing’, requiring not just continuous integration and continuous development (CI/CD), but continuous model (re-)training and monitoring. This distinction between the static versus ever-changing nature of a product is pivotal to the user experience. Traditional SWE has been the formative force of the Product profession, but it is crucial for AI product managers to be versed at additional skills to lead the creation, launch and management of ML and DL products.

Estimating return on investment (ROI) of AI investments: Measuring ROI is arguably a key responsibility of all product managers (PMs). However, an AI PM needs to be even more attuned to the costs of technologies, including ML infrastructure, cloud resources, data acquisition and annotation, and time investment of engineering teams. These can explode quickly when building AI products. Delivery life cycles are often longer, particularly in the first iteration, meaning the lags of returns may be greater. Moreover, the cost of maintaining a high-quality AI product post-launch could likewise be higher than for products without algorithmic components. AI PMs need the ability to manage expectations and communicate around probabilistic outcomes to non-technical audiences. They need to know techniques to slice the data science product lifecycle to deliver incremental value, fostering an iterative and experimental development style. That also entails grasping the principles of machine learning operations (MLOps) and short-cuts for prototyping, such as working with pre-trained models whenever feasible. When it comes to returns, PMs need to distil which return or key performance indicator improvement is expected, i.e. a user facing one such as improving user experience through personalisation, or an internal one, such as saving money or time. Determining how returns can be measured is a difficult, but essential skill. The variable and context dependent behaviour of AI models can call for knowledge of more advanced experimentation tactics than simple A/B testing. Quantifying returns may require more complex attribution methods than launching products or features not involving AI.

Mapping user problems and needs to AI capabilities: An AI PM needs to grasp the possibilities and boundaries of ML and DL to have initial hypotheses about whether a user problem is solvable by AI. If it is, they must know how to work with full-stack data and engineers teams to distil which methods might be most appropriate. That requires a conceptual understanding of the types of data that exist, foundational statistics, types of learning (supervised, unsupervised, reinforcement) and the most common algorithms used within them. It also involves the ability to judge what data would be required and support creatively sourcing and validating its quality. Not least, an AI PM ought to know which model performance monitoring is apt for the solution being built. Monitoring is not always straightforward or fully automatable, and it’s a popular stage of the ML/DL lifecycle to neglect or skip altogether. An AI PM understands why observing model or data drift is vital to uphold the relevance of a model, and therefore product, to the user.

Integration of ethical, data privacy and compliance considerations: When optimising for user experience, particularly in a competitive, profit-oriented context, ethical, data protection and regulatory aspects may fall off the radar. For instance, X (then Twitter) had to admit to racial bias after users reported that its automatic photo cropping feature systematically favours the inclusion of white faces over Black ones in photo preview mode [15]. The facial recognition platform Clearview AI scraped photos without user consent [16] , violating laws such as the EU’s GDPR and resulting in lawsuits and regulatory bans. Uber’s autonomous vehicle program faced scrutiny for its non-compliance with safety regulations and testing protocols [17]. The risks of damaging user trust and violating laws are greater when leveraging AI. The likelihood of regulatory scrutiny or user callouts of product issues related to data privacy, discrimination and bias is high. AI PMs must hold and maintain up-to-date expertise on these topics and be able to translate them into product requirements. This entails protecting and enabling agency over user data, guaranteeing the rights of user data subjects for data storage, historical traceability and deletion. PMs must develop a solid understanding of AI-specific guidelines and laws (GDPR, AI Act) and of approaches for auditing models for bias and ensuring systems for AI explainability are in place. Standard monitoring solutions focus solely on model performance, and are insufficient to observe model bias or transparency about model rationale. AI PMs need to understand the nuances of monitoring and advise on product components operating in the backend that users may not realise they need, but that protect them. For software and AI PMs alike, the main tenets of product management are market research, user needs, and strategy. Yet, PMs specialising in AI must learn these additional skills to guide on executing the sweet spot of what makes a viable business, what engineering can do, what is in the users’ interest, and what is ethical, particularly when these areas are not clearly aligned.

Make Product The Glue Of Your Ai Strategy

A better organisational awareness for what makes AI products less prone to failure and targeted investment in AI PM skill building can make single AI products successful. As a function owning checks and balances around the motivations, needs and expectations held around AI, Product also needs a strong voice in corporate AI strategies. Being conscious of business, tech and end users alike, AI product management is in a position to identify the macro needs for the pursuit of the AI roadmap. Concretely, it can help answer questions such as:

What type of infrastructure do we need mid-long term? This includes ensuring the tech stack matches the data storage and compute power needs to run the product at scale as its user base grows, planning for major tech migrations or code refactorings. It also includes ensuring the product’s AI ecosystem supports the desired deployment strategies to expose users to new models (such as standard A/B testing, canary or shadow deployment) and duplication of efforts amongst data science teams are minimised via collaborative tools, algo libraries, and feature stores.

What level of agency and customisation over our product do we require? Product is well-placed to chime in on decisions around investments in engineering capabilities vs. infrastructure-as-a-service/managed services for flexible scaling and general guidance on build-vs-buy principles, as these depend highly on the value proposition of the product and user requirements.

What measures, tools or systems need to be in place to comply with brand, user and legal expectations for relevance, transparency, data privacy, fairness and bias prevention? These must be appropriate to the specific risk level of exposing users to AI within the industry and use case. Product management must define overarching risk categories and risk mitigation strategies for each. For AI products, ensuring sensitive user data is safe, bias monitoring is being adapted to evolving needs, and intuitive explainability tools are accessible is key. Product strategising means thinking of these as core features of the product, planning for potentially crossproduct, scalable solutions and creating accountability mechanisms for ethical product development that are ingrained in the product life cycle.

How will the product remain relevant over time? This covers many aspects, from standards for model retraining, procuring fresh, clean and current data to ensuring the engineering teams and product functions are able to keep up to speed with AI developments and engage in continuous learning from academic and industry advances.

Conclusion

The AI gap is shifting from tech to product: The 2023 State of AI Product Management report cites ‘Data privacy concerns’ and ‘Lack of understanding about AI among stakeholders’ as the top two challenges faced by PMs in the context of working on AI products. ‘Technical limitations’, on the other hand, are cited as the second to last challenge. When participating Product professionals were asked whether lack of access to ML and DS talent was hindering their AI product progress, 68% had affirmed in 2021, but only 43% responded affirmatively in 2023. The same report shows that 71% of surveyed organisations did not have a dedicated AI PM role in 2023 [19] add guardrails to preempt harmful outputs

AI failures in the private sector, often in spite of massive investment, continue to be numerous. These are not limited to the average small to mid-sized enterprise.

Top tier tech companies have had to pause or scrap some of their signature AI undertakings. Particularly for the tech giants, a lack of data and engineering talent does not account for why some AI products fell through initially, or were even ultimately abandoned. It is time, I have argued, to shift focus from the AI gap in technical skills and capabilities to the AI product management gap. In the triangle of business, tech and user experience, the product profession is best placed to ensure user experience is not only considered, but comes first. In AI product management, user experience encompasses more than is visible to the users’ immediate perception. Model-fuelled products are moving targets, both influencing and influenced by users. The ‘human factor’ is much more far-reaching compared to products based on software engineering alone: be it understanding the individual psychological and socio-cultural user considerations influencing whether AI-products get adopted or fail, the ability to assess if AI is truly needed to solve a user problem and which manifestation of it is most suitable and economically viable, or having a grasp of how data protection, legal and ethical requirements translate into technical product requirements. Beyond the singular AI-supported feature or product, companies must evolve to productise their AI strategy as such. Scale of infrastructure, agency over customisation of algorithms, systems for fairness, data protection and explainability are all critical product components directly or indirectly affecting the users’ needs and interests.

Closing the AI product gap boils down to training AI PM talent that is equipped for the many facets of the job. The upskilling effort should be no less rigorous than that for engineering and data science. Engineering and statistics skills of sufficient depth and ROI measurement are non-negotiable AI PM assets. But the AI PM’s core expertise lies in building sensible intelligent user experiences. And although the public and non-profit sectors globally may not have nearly as much to show for when it comes to productionised AI in action, there are lessons to learn for the corporate sector as far as a focus on societal impact and accountability in the use of artificial intelligence is concerned. There are now countless frameworks, technical and methodological guidelines around the responsible, people-focused application of AI, such as by the World Economic Forum (PRISM) [20], advocacy organisations like the Algorithmic Justice League [21] and academic research outlets such as the Harvard Data Science Review [22]. Upskilling product functions for AI can leverage these resources, but will also need to involve more foundational teachings from psychology, social sciences, philosophy and legal studies.

For now, we are left with the irony that increasing tech know-how has enabled companies to do something-with-AI, commonly producing anythingwith-AI products. What they need now is rigorous AI product upskilling to create products that users actually need, and that can and should be AI-driven.

Author’s Note

Tara Dezhdar, Anela Hoti, and Alexia Vazoura offered ideas and constructive feedback on this article, drawing on their extensive experience in data science and Product. I thank them for generously sharing their expertise.