JSON幻觉率从30%降到0.1%

Title: How Schema-Constrained Decoding Saved My SaaS from LLM Hallucinations (and $2K in Refunds)

Product: TalkFlow AI

Revenue: $10,230 MRR

Last month, I stared at my Stripe dashboard in horror. 12 refund requests in 48 hours. All the same bug—my AI chatbot was spitting out JSON that looked right but was subtly, catastrophically wrong. Empty fields where phone numbers should be. The string "null" instead of actual null values. One customer sent me a screenshot of their CRM filled with "status": "I'll check that for you!" because the LLM decided to get chatty mid-JSON.

I was bleeding trust and about $2,100 in annual contracts. That week fundamentally broke something in how I think about Function Calling. Here's the unfiltered, messy journey of how I went from naive JSON parsing to implementing schema-constrained decoding, and why this might save your indie SaaS from the same nightmare.

The "Just Add Function Calling" Trap (Month 1-3)

When OpenAI launched Function Calling in June 2023, I shipped it in a weekend. Felt like a genius. My product, TalkFlow AI, lets non-technical teams build voice agents that trigger backend actions—booking appointments, updating CRMs, sending invoices. The pitch was simple: speak naturally, get structured output.

Under the hood? GPT-4 with function_call parameters and a basic try/catch for JSON validation. That's it.

At $3K MRR, this worked beautifully. My 47 customers were happy. Then I onboarded a real estate agency with complex scheduling logic—multiple agents, timezone offsets, property IDs that looked like PROP-2024-XJ9-001. Suddenly, GPT-4 started generating:

{
 "agent_id": "Sarah (the one in Austin office)",
 "time_slot": "tomorrow morning",
 "property_ref": "that blue house on Oak Street"
}

I wish I was joking. That's an actual response from my production logs on March 12th, 2024, 3:47 PM UTC. My validation layer tried to coerce "tomorrow morning" into an ISO datetime and silently failed, creating bookings for January 1st, 1970. Unix epoch zero.

I discovered this when the agency owner called me at 8 PM. Furious. His agents were getting calendar notifications for 54 years ago. Try explaining that one.

The Pivot: Schema-Constrained Decoding (Not Just Validation)

I did what every indie hacker does first: frantically Googled. Found Pieter Levels tweeting about how he "just uses better prompts."

Respectfully, Pieter, that doesn't work when you're dealing with 47 fields across 12 function schemas. Prompt engineering is a band-aid. The wound is architectural.

Here's what I eventually understood, after way too many late nights: LLMs are autoregressive samplers, not structured data generators. Each token is predicted based on probability distributions, not logical constraints. Even with perfect prompts and response_format: { type: "json_object" }, the model can still generate:

1. Type violations: Integer fields getting floats, strings getting arrays

2. Missing required fields: The model decides phone_number is optional because the user didn't mention it

3. Hallucinated fields: Adding "customer_mood": "angry" when my schema only allows status: "escalated"

4. Invalid enum values: "payment_method": "credit_card" when my system expects "cc", "ach", or "wire"

The solution isn't better validation. It's schema-constrained decoding. Also called constrained sampling or grammar-based generation. Fancy terms for a simple idea: modify the token selection process itself so the LLM literally cannot generate tokens that violate your schema.

Actually, wait—I should clarify that this is different from what OpenAI's structured outputs do now. Theirs is great. But when I built this in March 2024, it wasn't available yet. I was on my own.

Implementation: How I Built It (Without a PhD)

I'm a bootstrapper, not an ML researcher. But here's the approachable version that took me from zero to production in 3 weeks. Well... 3 very long weeks.

Week 1: Understanding the Token-Level Problem

When GPT generates "phone": ", the next token is sampled from ~50,000 possibilities. Without constraints, it might pick "555-", "null", or "I'll ask them". With schema constraints, we mask all tokens except those that could start a valid phone number string.

This happens during generation. Not after. That's the whole ballgame.

I used the outlines library by Rémi Louf. Shoutout to open-source—this saved me months, probably. It compiles your JSON Schema into a finite-state machine that guides token selection. Here's my stripped-down implementation:

from outlines import models, generate
from pydantic import BaseModel, Field
from typing import Optional

class BookingRequest(BaseModel):
 agent_id: str = Field(pattern=r'^[A-Z]{3}-\d{4}$')
 time_slot: str = Field(pattern=r'^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z$')
 property_ref: Optional[str] = Field(pattern=r'^PROP-\d{4}-[A-Z0-9]{4}-\d{3}$')
 priority: int = Field(ge=1, le=5)

model = models.openai("gpt-4")
generator = generate.json(model, BookingRequest)
result = generator("Book Sarah for tomorrow at 2pm, property PROP-2024-XJ9-001")
# Guaranteed valid or throws a clear error

The key insight: generate.json() doesn't just validate output. It constrains the model's token vocabulary at each step. If the next required token must be a digit, the model can only choose from tokens 0-9. No more "tomorrow morning" because those tokens are literally inaccessible. The probability gets set to zero.

Week 2: Productionizing with Fallbacks

Constrained decoding is heavier. 15-20% slower in my tests. I couldn't just swap it in everywhere. Here's my tiered approach:

• **Critical functions** (payments, calendar writes, CRM updates): Full schema-constrained generation with `outlines`
• **Semi-structured data** (notes, summaries): Standard function calling with strict Pydantic validation post-hoc
• **Free text** (chat responses): No constraints, but never used for structured data

My cost per 1,000 API calls went from $12.40 to $14.80. But my refund rate dropped from 4.2% to 0.1%. The math works: I was losing $2,100/month in refunds to save $240/month in compute.

Bootstrapper lesson I learned the hard way: optimize for customer trust, not server costs. I think a lot of us get this backwards.

Week 3: The Edge Cases That Still Haunt Me

Even with constrained decoding, I hit three problems that almost made me quit:

1. Regex isn't enough for cross-field validation: My schema required end_time > start_time. That's semantic, not syntactic. I added a second validation pass using Pydantic's @validator decorators that runs immediately after generation. Catches 93% of remaining issues. Not perfect. But close.

2. The "silent null" problem: When a user says "I don't have a property reference," the constrained model would sometimes generate "property_ref": "" instead of omitting the field. This passed schema checks but broke my database. Solution: explicit min_length=1 on optional string fields and nullable=False as default. Took me 4 hours to figure that one out.

3. Streaming breaks constraints: My product streams responses for UX smoothness. But token-by-token streaming means the first few tokens might be valid while later ones aren't. I had to buffer the entire function call response before displaying it. Added 800ms latency. Users noticed the pause. I got two support tickets about it. Worth it for correctness, but still annoying.

The Results (With Real Numbers)

After 6 weeks of running schema-constrained decoding:

• **Function call success rate**: 78% → 99.3%
• **Customer-reported bugs**: 23/month → 2/month
• **Refund rate**: 4.2% → 0.1% (saved ~$2,060/month)
• **MRR impact**: Lost 3 customers during the buggy period, gained 11 after. Net: +$1,840 MRR
• **Time spent on LLM debugging**: 15 hours/week → 2 hours/week

I shipped this as a "Reliability Update" to all customers on April 3rd, 2024. Three of them emailed me within 24 hours saying they noticed the improvement. One enterprise customer upgraded from $199/month to $499/month specifically because "the system finally does what we expect."

That felt good. Really good.

What I'd Do Differently

If I could go back to my $3K MRR self, here's what I'd change:

1. Start with constrained decoding from day one: I wasted 4 months building validation layers that were fundamentally flawed. The cost difference is negligible at small scale. The engineering debt of retrofitting constraints? Brutal.

2. Use JSON Schema as the single source of truth: I had three different representations of my data shape—OpenAI function definitions, Pydantic models, and database schemas. Now I generate all three from a single JSON Schema file using datamodel-code-generator. Life is simpler.

3. Test with adversarial inputs weekly: I now have a script that runs 200 intentionally confusing prompts. Stuff like "Book me when the sun is high but not too high, for the house with the red door." Ensures zero schema violations. Catches regressions before customers do.

4. Build a "constraint explainer" for debugging: When the model can't generate valid output, it often produces nothing or errors. I added logging that shows which constraint failed and what the model was trying to generate. This turned 2-hour debugging sessions into 5-minute fixes. Game changer.

The Bigger Picture for Indie Hackers

Here's what I keep thinking about: we're in a gold rush of AI wrappers. But the ones that survive won't be the ones with the best prompts. They'll be the ones that solve the boring reliability problems.

Schema-constrained decoding isn't sexy. It doesn't make for good Twitter threads. But it's the difference between a product that works 78% of the time and one that works 99.3% of the time.

When I see indie hackers like Danny Postma and Marc Lou raising prices on their AI products, I suspect they've quietly solved these same reliability issues. Customers pay a premium for trustworthy AI. Not just AI.

I'm now at $10,230 MRR with 84 customers and a 1.8% monthly churn. My CAC is $67—mostly content marketing and a few targeted ads on X. I'm not Pieter Levels. Not even close. But I'm building something sustainable. And it's because I stopped treating LLMs as magic and started treating them as probabilistic systems that need guardrails.

Anyway. That's my story. What a ride.

What's your experience with Function Calling reliability? Have you tried constrained decoding, or are you still in the "better prompts" phase? Drop your horror stories below—I'll share the worst one in my next update. Seriously, I want to hear them. Misery loves company.

Product: TalkFlow AI - Voice agents with structured actions

Revenue: $10,230 MRR | Customers: 84 | Churn: 1.8%

Previous milestone: $7K MRR after pivoting from chatbot templates to voice-first

#buildinpublic #aiengineering #functioncalling #indiehackers #bootstrap