diff --git a/foundation-frontiers/posts/2026/02/25/why_great_models_still_fail.qmd b/foundation-frontiers/posts/2026/02/25/why_great_models_still_fail.qmd index 60439a47..8551e4c0 100644 --- a/foundation-frontiers/posts/2026/02/25/why_great_models_still_fail.qmd +++ b/foundation-frontiers/posts/2026/02/25/why_great_models_still_fail.qmd @@ -20,7 +20,7 @@ Success in practice goes far beyond code and algorithms. It comes down to solvin The importance of this skill emerged repeatedly in the “10 Key Questions to Data Science and AI Practitioners” interview series, run by the Data Science and AI Section of the Royal Statistical Society. The series gathers perspectives from practitioners at various career stages, from those starting their career to senior leaders. By posing the same ten questions, it uncovers motivations, challenges, and visions for the future while highlighting the breadth of career paths in the field. When asked what they considered the most undervalued skill, many participants highlighted the importance of something non-technical — the ability to understand the organisational context and the needs of users. -The importance of these skills for data science and AI practitioners is further evidenced by their emphasis in government and professional standards. The UK Government’s DDaT Capability Framework highlights that data science practitioners especially at higher levels are expected to “design and manage processes to gather and establish user needs”. Similarly, the Royal Statistical Society in The Alliance for Data Science Professionals Certification Guidance and Process: Advanced Data Science Professional states as a key skill the ability to be “engaging stakeholders, demonstrating the ability to clearly define a problem and agree on solutions” including being able to “Identify and elicit project requirements”. Together, these frameworks show that engaging directly with users and stakeholders is not optional—it is a core professional expectation for data science and AI practitioners. +The importance of these skills for data science and AI practitioners is further evidenced by their emphasis in government and professional standards. The UK Government’s [DDaT Capability Framework](https://ddat-capability-framework.service.gov.uk/role/data-scientist?utm_source=chatgpt.com) highlights that data science practitioners especially at higher levels are expected to “design and manage processes to gather and establish user needs”. Similarly, the Royal Statistical Society in [The Alliance for Data Science Professionals Certification Guidance and Process: Advanced Data Science Professional](https://rss.org.uk/RSS/media/File-library/Membership/Prof%20Dev/AdvDSP-Guidance-Notes-2024.pdf?utm_source=chatgpt.com) states as a key skill the ability to be “engaging stakeholders, demonstrating the ability to clearly define a problem and agree on solutions” including being able to “Identify and elicit project requirements”. Together, these frameworks show that engaging directly with users and stakeholders is not optional—it is a core professional expectation for data science and AI practitioners. ::: {.callout-note appearance="simple"} **The Case of the Vanishing Model** @@ -38,7 +38,7 @@ Then comes the handover presentation: The project stalls. Despite strong performance metrics, the model never makes it into production. The lesson is clear: even the most technically impressive solution will fail if it isn’t designed with real-world context in mind. The model simply “vanishes” and all that hard work goes to waste. This example is deliberately simplified. In some organisations, practitioners may work alongside business partners, product owners, or domain leads who help shape requirements and maintain alignment with broader goals. Yet this support does not remove the practitioner’s responsibility: technical success still depends on their own clear understanding of the business requirement and recognition their technical solution may be a small but an integral cog in a large machine. For the machine to work effectively all the parts must work together. A model is not just a mathematical construct; it is a product that must operate within the complex, resource-limited realities of an organisation. - ::: +::: ## Start with What We Are Trying to Achieve @@ -140,7 +140,9 @@ In the retention case, adoption failed because the model was delivered as a fini - Set up a lightweight KPI dashboard so everyone can see early wins in terms of contracts saved and revenue retained. - Create champions by involving stakeholders at every stage, so they owned and advocated for the solution. -Had the project taken an iterative, MVP-first approach, the practitioner would have avoided months of sunk effort and built momentum for adoption as trust grew over time. Adoption is not an afterthought—it is the decisive factor that turns technical excellence into sustained impact. +Had the project taken an iterative, MVP-first approach, the practitioner would have avoided months of sunk effort and built momentum for adoption as trust grew over time. Adoption is not an afterthought—it is the decisive factor that turns technical excellence into sustained impact. +::: + ## The Bottom Line @@ -152,7 +154,7 @@ Prove value under real-world conditions with well designed and measurable evalua In the end, the mark of a great model is not the elegance of its algorithm but its ability to have a positive impact. -*For a broader, strategic view of why organisations struggle to realise value from AI—and how leadership and structure can change the odds—check out [“Why 95% of AI Projects Fail.”](https://realworlddatascience.net/applied-insights/case-studies/posts/2026/01/12/why-95-percent-of-ai-projects-fail.html) +*For a broader, strategic view of why organisations struggle to realise value from AI—and how leadership and structure can change the odds—check out [“Why 95% of AI Projects Fail.”](https://realworlddatascience.net/applied-insights/case-studies/posts/2026/01/12/why-95-percent-of-ai-projects-fail.html)* **Sources:** [1]Allen, K. A., Charpentier, V., Hendrickson, M. A., Kessler, M., Gotlieb, R., Marmet, J., Hause, E., Praska, C., Lunos, S., & Pitt, M. B. (2023). Jargon Be Gone – Patient Preference in Doctor Communication. Journal of Patient Experience, 10, Article 23743735231158942. DOI: 10.1177/23743735231158942.