When a SaaS product crosses 50,000 active users, the support math breaks. Ticket volume outpaces headcount. Response times slip. Agent burnout climbs. You can hire faster — or you can change the math.

We chose to change the math. Here’s exactly how we built the AI triage and resolution system that now handles 78% of support tickets without a human touching them — and how we got the false positive rate low enough that the team actually trusts it.

“The goal was never to replace support agents. It was to make sure they only see tickets that actually need them — and that those tickets get their full attention.”

The Problem at 50k Users

At 50k users, the incoming ticket mix looks roughly like this (based on our audit at intake):

  • ~35% are questions already answered in documentation
  • ~20% are known bugs with existing workarounds or fixes in progress
  • ~18% are billing and account management requests with deterministic resolution paths
  • ~12% are feature requests that need logging but not resolution
  • ~15% are genuinely novel issues requiring human investigation

That means 85% of tickets have a known, scriptable resolution path. Yet every ticket was hitting the same queue and the same agents. The waste wasn’t the 15% — it was the 85% consuming agent bandwidth at the same rate.

The Triage Architecture

Our AI support system has three core functions: classify, respond, and escalate. Each is distinct, testable, and has its own accuracy target.

Classify

Every incoming ticket runs through a multi-label classification model before any human sees it. Classification outputs: issue type (from a controlled taxonomy of ~40 categories), severity (P1/P2/P3/P4), user tier (free/paid/enterprise), and sentiment score.

We use a fine-tuned model here rather than a general-purpose LLM — fine-tuning on 6 months of historical tickets gives us 94% classification accuracy across our taxonomy, which is meaningfully better than prompt-engineering a general model and dramatically faster per ticket.

Classification doesn’t just route — it enriches. By the time a ticket reaches a human, they can see the predicted issue type, recommended KB articles, similar past tickets, and the user’s account history. Agents resolve tickets 40% faster just from this enrichment, even on tickets where the AI doesn’t attempt resolution.

Respond

For the 85% of tickets with known resolution paths, the system generates and sends a response automatically. Response generation is tiered:

  • Deterministic responses for billing, account management, and known-bug categories: templated responses with variable substitution (account details, fix ETAs, workaround steps) that never touch an LLM. Fast, cheap, reliably accurate.
  • AI-generated responses for documentation questions and feature requests: GPT-4o pulling from the current KB and generating a contextual response. Every AI-generated response includes a “Was this helpful?” prompt that feeds back into quality tracking.

Responses are sent immediately for P3/P4 tickets. For P1/P2, they’re drafted and queued for 5-minute agent review before sending — we never auto-send on high-severity tickets.

Escalate

The escalation logic is the most important piece and the easiest to get wrong. False negatives (escalating tickets that the AI could have resolved) cost agent time. False positives (not escalating tickets that needed humans) cost customer trust.

Our escalation triggers: classification confidence below 0.82, any mention of data loss or security concern (keyword + sentiment model), user tier = enterprise + severity P1 or P2, second reply from the same user within 24 hours (indicating the first response didn’t resolve), and any ticket where the generated response confidence is below threshold.

The result: 22% of tickets escalate to human agents. Of those, fewer than 3% are later reclassified as “could have been auto-resolved” — our false positive rate on escalation.

Getting to Near-Zero False Positives

The 3% false positive rate didn’t happen on day one. We started at 11% and drove it down over 8 weeks through a specific process:

Every Monday, an agent reviews the previous week’s escalated tickets and flags any that the AI should have handled. Those cases go into a retraining dataset. We retrain the classification model monthly and update the escalation trigger thresholds based on the false positive analysis.

The agents who did this review initially complained it was extra work. Within four weeks, they stopped complaining — because the queue they were seeing was getting cleaner, and the tickets landing on their desk were the ones worth their time.

Human feedback on AI decisions isn’t a burden. It’s the mechanism that makes the AI get better. Build the feedback loop into the process from day one.

What the Numbers Look Like at 6 Months

  • 78% auto-resolution rate — up from 0% at baseline
  • 4.2 minute median first response time — down from 6.8 hours
  • CSAT score: 4.6/5 — up from 4.1 pre-deployment
  • Agent ticket volume down 72% — same headcount handling meaningfully more total volume
  • Escalation false positive rate: 2.8% — agents rarely see tickets they didn’t need to

The CSAT improvement is worth highlighting. Skeptics of AI support automation often predict it will hurt satisfaction scores. In this deployment, it improved them — because 78% of users got an accurate, immediate response at any hour, and the 22% who needed humans got agents who weren’t burned out from answering the same documentation question for the eighth time that day.

What Not to Do

Three failure modes we’ve seen in other deployments:

Don’t auto-respond to everything from day one. Start with drafting responses for agent review, not auto-sending. Build confidence in the classification and response quality over 2–3 weeks before removing the human review gate on any category.

Don’t ignore the feedback loop. A system without weekly human review of escalation decisions will drift. Set up the review process as a standing ritual before you go live.

Don’t optimize for auto-resolution rate in isolation. It’s easy to game this metric by setting aggressive thresholds and sending confident-sounding responses to tickets that deserved escalation. Measure CSAT and re-open rates alongside auto-resolution rate — they’ll tell you if you’re actually solving the problem or just closing tickets.