Section 5

What's Still Broken

The Challenges That Remain

2025 Reality

Why These Challenges Spiked in 2025

Systems crossed a threshold. They stopped being "text generators" and started being "workflow executors."
Agents ran longer. Used tools. Operated across multiple context windows. Interacted with real environments.
Manual testing hit a wall. Teams reached a breaking point—"flying blind" after changes, unable to distinguish regressions from noise.
These aren't random problems. They are the predictable cost of systems getting more capable and more connected.

Hallucinations

Not just fake facts anymore. In 2025, hallucinations showed up as false claims about what the system did inside a workflow.
The dangerous pattern: Agent says "your flight has been booked"—but the reservation never appeared in the database.
This is hallucinating the world state. The model confidently reports completing actions it never attempted.
High-stakes domains exposed the risk. Legal, medical, and financial tools showed hallucination remains a major practical risk.

Inconsistent Reasoning

Inconsistency always existed. But 2025 made it visible because agent behavior is multi-step and path-dependent.
Two runs, different outcomes. Same prompt can take different tool sequences and reach completely different results.
Non-determinism is the default. Think in success rates across multiple trials—a task can pass once, fail the next.
The key reframe: It's not enough to ask "did it work once." You need to ask "how often does it work."

Over-Autonomy

2025 put "agency risk" on the map. Agents could take real actions—especially via browser and computer use.
The pattern we saw: "Delete the test file" becomes "I've cleaned up all test files and reorganized your directory."
The tradeoff became clear: Confirmation steps reduce autonomy but block high-risk operations. That's a feature, not a bug.
Human approval gates became a design pattern. Not a nice-to-have—an explicit requirement for production systems.

Tool Grounding

Tool grounding became measurable. Systems used more tools more often—failures became obvious and quantifiable.
Wrong tool selection: Models pick semantically similar tools that are functionally wrong.
Malformed calls: Arguments in wrong formats, required fields missing, JSON that almost parses but doesn't.
Phantom tools: Agents calling tools that don't exist—hallucinating capabilities based on what "should" be available.

Context Drift

Long context grew fast. But 2025 proved longer context doesn't guarantee better use of information.
"Lost in the Middle" is real. Performance degrades when relevant information sits in the middle of long inputs. Best recall: beginning or end.
Context pollution compounds. Without structure, more context means more distraction. Compaction and structured notes help.
The 2025 lesson: Long context increased capacity, but without active management it increased noise.

Actionable: Structure your context. Put critical instructions at start and end. Compact aggressively over long horizons.

Retrieval

Retrieval got better, failures got subtler. The problem shifted from retrieval accuracy to end-to-end grounding.
Semantic similarity ≠ relevance. Query "Q4 customer churn" and get docs about satisfaction, Q3 churn, Q4 revenue. All close. None useful.
The new failure mode: Models sometimes fail to leverage retrieved passages—especially when irrelevant context is present.
The ideal behavior is binary: Answer correctly OR say "I don't know" when info is missing.

Multi-Agent

Multi-agent systems scaled in 2025. But coordination errors emerged as a new class of failures.
Error cascades: Agent A makes a small mistake. Agent B builds on it. By Agent D, the error is unrecognizable but catastrophic.
Coordination breakdowns: Agents assume what others will do. Assumptions conflict. Deadlocks and race conditions emerge.
The tradeoff: More agents = more parallelism, but also more contradiction risk, duplication risk, and propagation risk.

Actionable: Design coordination protocols explicitly. Don't assume agents will self-organize correctly.

Debugging

This is where all challenges compound. Debugging became painful because failures come from trajectories, not single outputs.
Without evals, debugging is reactive. Teams wait for complaints, reproduce manually, fix, and hope nothing else regressed.
Agent observability is immature. We need standardized metrics, traces, and logs—most teams are still building custom solutions.
Different roles need different views. Developers, testers, and ops all need visibility into the same system, but with different lenses.

If you can't trace what the system did, you don't control it. In 2025, teams stopped pretending otherwise.

These challenges are predictable. More capability = more failure modes. Design for them upfront.
Hallucinations are now operational. Verify actions in target systems. Don't trust self-reports.
Think in success rates, not single runs. Product reliability depends on consistency across trials.
Build human gates before irreversible actions. Confirmation steps are a feature.
Invest in observability early. If you can't trace trajectories, you can't debug agents.