The Real Cost of Bad Data: Why Your AI Project's Biggest Risk Has Nothing to Do with AI

I want to tell you about the most expensive conversation I’ve ever had with a client, in terms of what it revealed.

We were four weeks into an AI project — a document classification pipeline that was supposed to route incoming contracts to the right team based on content. The LLM was solid. The orchestration worked. The UI was clean. But the accuracy was stuck at 61%, no matter what we did to the prompt or the model.

So I did what you do when a model is underperforming: I pulled a random sample of the training data and looked at it.

Fourteen of the fifty records I reviewed had mislabeled categories. The labels had been assigned manually, years earlier, by different people using different interpretations of the same category definitions — which had also changed twice since then but nobody had retroactively updated the historical data. We weren’t building AI on top of a dataset. We were building AI on top of a disagreement that had been crystallized into rows in a database.

The model wasn’t the problem. The model was faithfully learning the noise.

What Bad Data Actually Costs

There’s a widely cited rule in data circles: 80% of data work is cleaning, and 20% is analysis. Anyone who’s worked in analytics or AI for more than a year knows this is roughly right, and also that it dramatically undersells how bad the cleaning problem is.

But “cleaning takes 80% of your time” doesn’t capture the business cost of bad data that you don’t clean before you build on top of it. Here’s what that actually looks like:

Your AI project costs 3x what it should and delivers half what it promised. When you’re iterating on prompts and models trying to improve performance that’s actually bottlenecked by data quality, you’re spending engineering time in the completely wrong place. The model isn’t the variable. The data is. And every hour you spend fine-tuning the prompting strategy is an hour you could have spent fixing the labels, standardizing the formats, deduplicating the records.

Decisions made on AI output compound the original data errors. This is the part that costs real money. An AI system trained on bad CRM data produces customer segmentation you don’t trust. So you either don’t use it (wasted investment) or you use it and make targeting decisions that are systematically skewed by the noise in the underlying records. Those decisions touch customers, touch revenue, touch relationships — and the errors propagate outward from the AI system into the business in ways that are very hard to trace back.

Technical debt accrues faster than you can pay it down. Bad data siloed in one system is a problem. Bad data that gets ingested by an AI pipeline and distributed across embedded vectors, fine-tuning datasets, evaluation benchmarks, and downstream exports is a problem that has multiplied into every system that depends on those outputs. Cleaning it retroactively means cleaning it everywhere. That math usually doesn’t pencil.

The Three Categories of Bad Data That Kill AI Projects

Not all data problems are created equal. In the AI context, I’ve seen three categories show up consistently.

Structural inconsistency. The same thing is represented differently across records, systems, or time periods. “New York,” “NY,” “New York, NY,” “NYC” for the same field. Date formats that change depending on which team imported the records. Category labels assigned under different schema versions that were never reconciled. Product SKUs that changed structure after a rebrand. Structural inconsistency makes it impossible for a model to learn reliable patterns — the signal you want to learn keeps changing its form.

Label disagreement. Any data that was created by humans assigning categories, scores, or classifications carries implicit disagreement between those humans. “High priority” means different things to different people. “Resolved” in a support system might mean actually resolved or might mean the ticket was closed to hit a metric. When you train AI on this data, you train it to reproduce the disagreement — at scale, consistently, in ways that look like predictions.

Temporal drift. Data that was accurate two years ago may not reflect how your business works today. Products that no longer exist. Customer segments based on a pricing model you changed. Contact records with titles from people who left. Training AI on historical data that doesn’t represent current reality produces models calibrated to the past — which is almost always not what you want.

Why Migration Is the Forcing Function

Here’s the insight that comes up in almost every data migration project I work on: the migration itself reveals data quality problems that have been quietly invisible in the old system.

When everything lives in one system, bad data is tolerated. The system’s UI presents it in ways that make it manageable. The team has workarounds. Nobody has fully mapped the shape of the mess because they don’t have to — they can navigate it by feel.

Then you start migrating to a new system. You have to define a schema. You have to map every field. You have to decide what to do with records that don’t conform. And suddenly the problems that were invisible become concrete and enumerated: 12,000 records with missing required fields. 4,700 duplicate contacts with conflicting data. 900 company records with no industry classification because that field was added in 2022 and nobody backfilled the old records.

This is uncomfortable. Clients sometimes experience it as the migration “finding problems” — as if we created the issues by looking for them. The problems were there before. The migration just made them impossible to ignore.

The right response to this is gratitude, genuinely. You are better off knowing the shape of your data quality problem before you build AI on top of it than after. The after version is much more expensive.

What a Data Quality Plan Actually Looks Like

There is no magic. There is only systematic effort applied in the right order.

Audit before you build. Before any AI project begins — and ideally before any data migration begins — run a structured data quality audit. Profile every field: completeness rate, uniqueness, value distribution, format consistency. This doesn’t have to take weeks. A solid SQL profiling script against your primary tables will surface the worst 80% of problems in a day. The point is to see the shape of the problem before you commit to an architecture that assumes clean data.

Fix at the source, not in the pipeline. The temptation is to clean data in the ETL pipeline — add transformations that normalize inconsistencies, fill nulls with defaults, reconcile format differences. This is better than nothing but it propagates the problem. The source data is still wrong. When someone queries the source directly, they get the wrong answer. When another system ingests from that source, they get the wrong answer. Fixing it at the source — even if that means a remediation project on legacy data — is the only fix that actually holds.

Define and enforce data contracts. For any new data that flows into a system that AI will use, define a schema contract: required fields, value types, allowable values for categorization fields, validation rules. Validate at ingest. Reject or quarantine records that don’t conform rather than letting them slip through dirty. This stops the problem from continuing to accumulate while you fix the existing backlog.

Build evaluation into the pipeline. Once you have AI running on real data, instrument it. Sample outputs regularly. Compare AI decisions to ground truth on a subset of cases. Track accuracy over time. When accuracy starts drifting — and it will, because data drifts — you want to catch that signal early, before it’s been compounding for months.

The Conversation I Wish I’d Had Earlier

Back to the contract classification project. Once I showed the client the label disagreement problem, we spent three days doing something that felt backward: getting a small group of subject matter experts to re-label 2,000 records using a clearly defined rubric — not their judgment, a rubric.

Then we retrained on the clean subset. Accuracy went from 61% to 89% without touching the model. Same LLM. Same prompts. Completely different data.

The three days on labels were worth more than the four weeks we’d spent on prompt engineering. That doesn’t feel right when you’re in the middle of an exciting AI project and someone suggests you need to spend a week looking at database records. But it’s the truth of how AI actually works in practice.

Your AI is only as smart as the data it learned from. The most important investment you can make in an AI project isn’t the model. It’s the foundation you build on.

If you’re planning an AI project and you’re not sure what shape your data is in — or if you’ve hit a performance ceiling and want to rule out data quality as the culprit — let’s take a look before you go further.