Why Do Some Artificial Intelligence Projects Fail?
85% of AI projects fail, 6 reasons why
Introduction;
1-Artificial intelligence systems do not solve the problem correctly
2-Artificial Intelligence and Innovation Gap
3-Artificial intelligence systems do not perform well enough and are often not useful
4-Artificial intelligence systems do not produce enough VALUE
5-We overlook the easy
6-Ethics, prejudice, social harm
Conclusion
It will proceed as follows.
Introduction
Despite eye-catching headlines and exciting technical developments, the track record of AI projects has been less than stellar. According to research firms like Gartner and HBR, an estimated 85% of AI projects fail before or after going live.
This is more than double the failure rate of software projects.
It is common knowledge that AI is more difficult to deploy than normal software. The main reason for this is that AI itself produces uncertain results and its capabilities in the hands of users are not fully understood and surprising :).
Unintended consequences that occur after the software goes live can lead to the failure of projects and shake the confidence of users in AI systems. This also makes it difficult to predict the costs and return on investment (ROI) of projects.
Here are some examples of AI projects that are at risk of failure, based on my knowledge from sources spanning years of academic and industry research:
- Recommender systems
- Driverless cars
- Computerized vision
- Health care diagnostics
- Autonomous robots
- Financial risk assessment
1. AI systems do not solve the problem correctly
“Data scientists come up with things people don’t want. Designers come up with ideas that can’t be real” John Zimmerman , CMU
Data scientists and technical experts often work separately , rarely interacting with the rest of the business, thus producing projects that are unlikely to deliver added value and change.
Designers are in the best position to make a technology useful and usable because they approach problems from the users’ perspective. However, they are generally not invited to participate until the end of the projects where functional decisions are made , and they lack the capacities and knowledge of what the artificial intelligence capabilities to work on can do .
Artificial intelligence research mostly focuses on mechanisms (how something technically works) rather than capabilities (identifying the benefits that artificial intelligence will provide to people and turning them into a product that will deliver these benefits to users).
I would like to make a distinction here; people who receive engineering education may differ in the sector because they are mostly trained by knowing and learning the training and disciplines of seeing the problem, solving the problem, diagramming the solution phase.
Let’s understand the job description;
Data Scientists work in the field of improving and inventing mechanisms .
Designers and Product Managers work in the field of transforming a capability into a desired product.
Procter & Gamble temporarily placed Data Scientists within business units. Accenture Song explored AI innovation through data science and design topics. Conducted successful AI ideation sessions with teams of Researchers, Data Scientists, Designers and domain experts .
Since that study, more research has been done and the result:
“ Design participants had the most success when they constantly collaborated with data scientists to help envision what they would do and embraced a data-centric culture …”
Many studies recommend prioritizing AI use cases based on impact first , risk second , and data third: “Do we have the data to realize this use case?” Do we have permission to use it? Is it clean enough to be useful? If we cannot pass this step, we do not start. “We find another use case.”
2. Artificial Intelligence and Innovation Gap
Technical advances are often followed by design innovations. New enabling technology leads people to envision many new forms of technology being incorporated into different aspects of life.
Engineers create new technologies that enable new capabilities. Designers do not invent new technologies. Instead, they create new versions of known technologies (Louridas, 1999)
However, in the last 60 years of artificial intelligence research, there has not been a rich design innovation like other technologies. I think the reasons are that artificial intelligence is invisible (it comes from the difficulty of understanding), it proactively serves results, and its use for design is difficult to detect .
Generative AI has brought AI to the center of the interface. How? It has accustomed the society, which was previously accustomed to artificial intelligence figures that used Siri to predict and market in incoming notifications or to customer services — IN THE BACKGROUND — to the ChatGPT page and its use.
Today ‘s real AI revolution is about UX (user experiences) .
People interact directly with AI models and try to optimize the results in some way. However, other types of AI (we have hundreds of AI models running in the background of our devices, in products, and in our environment :)) are rarely discussed and often go unnoticed.
Researchers call this the ‘AI-innovation gap’ and it is often attributed to:
2 identified vulnerabilities in Data Science
Lack of effective ideas
Most data scientists’ time is spent analyzing, building models, and extracting valuable insights from data. Relatively less time is spent on ideation, i.e. discovering the right problems to solve with data. Instead of immediately focusing on a single idea, what are the hundreds of alternative ideas that might be better to ask in the first place?
Lack of effective mental models and their communication
The responsibility for explaining data science concepts to collaborators and colleagues often falls on AI developers. In practice, communication gaps arise, especially in the transfer of technical knowledge and its value . Because managers or designers cannot produce effective comments on a subject they do not fully understand.
According to Barr Moses, ‘ The sad truth is that even the creators of AI aren’t exactly sure how it works .’
Solutions to the issue were suggested:
1. Sensemaking framework ( What Did My Artificial Intelligence Learn? How Do Data Scientists Make Sense of Model Behavior? )
2. Developers’ Mental Thoughts ( How AI Developers Overcome Communication Challenges in a Multidisciplinary Team: A Case Study )
And 2 identified vulnerabilities in the Design
Designers fail to recognize obvious places where machine learning can improve user experience
For example: Starbucks’ gift card model
The Starbucks app can automatically direct you to the payments screen when it detects that you are in a store, but users still have to go to the payments section again and again to use the gift card. This is certainly not difficult to solve, but it is an overlooked place where artificial intelligence can offer great convenience.
Designers have the knowledge and imagination of artificial intelligence that has taken over us, driven by media hype and criticism.
Instead of less risky, shippable AI products, designers tend to come up with science fiction-inspired capabilities that AI can’t realistically do. The perception of end users is also influenced by the media, they think that artificial intelligence will replace them, behave like humans, and work deterministically. This results in two mutually adaptable agents to be designed for both the probabilistic AI system and the user with uncertain AI expectations and mental models .
3. Artificial intelligence systems do not perform well enough and are often not useful
There are various outputs that artificial intelligence systems have difficulty satisfying and, as a result, the systems may not be useful. (I think we will overcome this with development time.)
If we take a closer look, these actually apply when the AI is not up to the task and (PLEASE) delegating the task to a human would be a more promising choice:
- Health diagnostics for rare* diseases.
Healthcare clinical decision systems provide predictions for common scenarios. But doctors say it doesn’t really work in unusual scenarios where AI performs poorly.
(I say especially rare because I am definitely a supporter of its use in health.)
- Automated content moderation for semantic nuances
AI content moderation systems can struggle to accurately identify subtle language (satire, sarcasm, and cultural references). (Especially our beautiful Turkish) Systems can remove content that doesn’t actually violate guidelines, when a person needs to be flagged for approval. - Autonomous vehicles that respond to environmental changes
Vehicles are trained to stop at stop signs. However, when a security guard crossing the street randomly holds a stop sign in the middle of the street, the vehicle stops and does not know how to proceed. Systems tackle new situations, such as unforeseen weather obstructions that change before they can react, or mapping negative areas such as trenches in the ground. Or when the traffic police asks him to stop, he has to stop, but our rebellious child can continue his movement.
I’m sure they will all improve, but additionally remember to instead apply AI where AI is and can be wonderfully superior;
(1) Mundane , repetitive data entry tasks where AI reduces human error . Taking notes, searching for information, and sorting photo albums by person, time, and location.
(2) Sort large data sets at scale and provide insights, models, and recommendations. Examples include meeting recaps, tagging moments in time by keywords or images, and finding an alien spy bubble by looking at world data.
(3) Fast, real-time tasks that humans cannot do mentally or physically. Autonomous robots navigate warehouses or maneuver into remote parts of the world to capture data and examine infrastructure, taking the guesswork out of soldiers putting themselves in harm’s way.
(4) Human-AI collaboration. Co-pilots, agents, AI partners, whatever term you like. This concept has its origins in controlling robots. The Sheridan -Verplank Scale describes levels of human-machine interaction, from no autonomy to fully autonomous machines. Research shows that intermediate levels don’t work for robots. There is a high cognitive load for a person moving from controlling the machine to taking over the task completely. Designing effective handoffs, even across computer interfaces, will prepare people to anticipate and then participate in tasks .
4. Artificial intelligence systems do not produce enough VALUE
The cost of creation and maintenance is high, the return is low.
Alexa was launched with the aim of getting people to order more products from Amazon. Instead, people used Alexa to play music. Amazon is now losing money when people ask Alexa to play music instead of shopping.
Hundreds of AI tools have been developed to catch COVID . Research teams around the world rushed to help.Software has been developed that allows hospitals to diagnose or triage patients more quickly . But unfortunately, almost none of them worked; some were harmful . Many problems were linked to poor data. This has led to reduced adoption and investment issues, which is an example of high AI investment with low service value.
Similarly, IBM Watson for Oncology has achieved technical success in developing artificial intelligence that can process and recommend cancer treatments. However, it failed to provide sufficient return on investment due to high development costs, integration difficulties, and limited effectiveness in improving treatment outcomes beyond current medical knowledge. It remained suboptimal for financial returns.
Ideas for Creating Artificial Intelligence
Our inflated expectations will be dashed when AI does not perform tasks as promised. However, it may not yet be possible to accomplish these tasks with existing data.
More on Designing Successful AI Products and Services .
5. We overlook the easy way
Simple AI opportunities that align with user and business goals exist, but companies aren’t building them.
- Instagram TikTok, and Twitter ask taggers to post to get views and attract them to their profiles, but does not learn which tags taggers frequently use, forcing people to type the same tags over and over again. It doesn’t recommend a new one, it doesn’t save it like it does an e-mail signature.
It’s not hard to create, but it’s an AI opportunity that goes unnoticed or undone. - Self-parking cars only serve a small group of users who are afraid of parking in tight spaces. It is very difficult to create artificial intelligence so that a car can park itself. But it starts with a simple guess that can happen in any car: Is this space big enough? Can I park in a space or not? What is my realistic percentage of success compared to my past parking experiences? I don’t want to waste time, but we mostly have a mentality to make artificial intelligence abilities difficult.
- But even simple AI can make headlines.
Accident detection, AI can save lives and call for help.
Handwashing detection keeps you safe and hygienic during COVID.
Autocorrect : Never send an embarrassing text again :)
6. Ethics, prejudice, and social harm
There are many articles on this subject, please check them out first :)
AI ethics is broad but includes three main themes:
bias,
privacy, and
transparency.
(1) Bias occurs when available training data does not accurately represent the population the AI aims to serve . For example, AI can help recruiters screen resumes. But systems are learning to prioritize resumes that exhibit patterns found in resumes of past successful male candidates or educational backgrounds, downgrading underrepresented groups. AI can also evaluate the feasibility of diagnosing a type of disease that affects both men and women. However, if the company only has data for women, the models will produce results that are biased for men. And it will be misleading.
(2) Privacy requires that AI models are secure and do not leak personal data .
Unauthorized access, misuse or violations can lead to identity theft, discrimination, etc. may cause. Stanford HAI offers three recommendations to reduce data privacy risks . Ann Cavoukian’s Privacy by Design principles also provide a mathematical definition of data privacy.
(3) Transparency suggests that users should understand how AI models work to evaluate their strengths, limitations, and functionality.
The field of explainable artificial intelligence (XAI) has emerged as AI increasingly powers high-consequence decisions, from healthcare to criminal justice . ‘Transparency’ is now widely used as a design pattern for equalizing user expectations.
Transparency also allows humans to understand, trust and effectively manage their intelligent partners (our AI) where decisions are limited by the machine’s inability to explain thoughts and actions in critical situations.
Two Distinctions
(1) Transparent systems create models that can explain their logic in ways that humans can understand.
(2) Interpretable systems create patterns that, while not transparent, are nevertheless understandable and predictable; It comes down to recognizing that deep learning techniques, in particular, can be extremely complex and difficult to explain.
HCI and AI research has produced many user-centered algorithm visualizations, interfaces, and toolkits to help with AI literacy. TensorFlow playground , Wekinator , AI Explainability 360 Toolkit , Facets of Google .
There are two types of users, not every piece of content is suitable for everyone.
(1) User-centered AI advocates systems designed to be accessible to people without AI technical knowledge or expertise. Products that embody this perspective are: Google Teachable Machine , Apple’s CreateML , Microsoft’s Lobe.ai , Generative AI tools.
What I found from specific research: How non-experts build models , collaborative machine learning with families .
(2) Expert-centered AI develops AI systems for domain experts with deep knowledge. According to GitHub Co-pilot Salesforce Einstein, systems are often complex and customizable. It is designed for experts who are familiar with specific workflows and classifications. Novices and non-experts get lost figuring out how these systems work.
Conclusion
Data Science and Design emerged from different disciplines. But as AI moves closer to controlling user experiences, we are increasingly connecting and meeting in AI products.
Industry (for-profit) is often not good at thinking critically and doing useful things that lead to unintended consequences.
Academies (non-profit organizations) act as knowledgeable critics who are good at critical thinking, and I have cited many academic sources in this article.
Design AI for what it can do.
Situations include situations
where (1) the risk of incorrect results or risk mitigation is low,
(2) machines can minimize human error,
(3) the AI is expected to be a trainee, not an expert.
So COOPERATION is needed for artificial intelligence. :)
See you in the next articles!
Please feel free to write me about it, I will always be a message away.
You can also follow me on my Linkedin and Twitter accounts!🐞
