How to Build a Customer Support Chatbot with LLM APIs

A customer support chatbot is the classic LLM product: high traffic, predictable question types, and a clear success metric — resolve the ticket without a human. LLM APIs make it buildable in a week, but the difference between a chatbot that helps and one that infuriates is in the details: grounding, escalation, honesty, and cost.

This guide walks through the architecture of a production support bot: retrieval over your knowledge base, answer generation with source links, escalation rules, and the cost math per conversation.

The architecture in one picture

1. The user message goes through classification: billing question, product question, bug report, or not-supported.
2. Retrieval finds the relevant knowledge base articles for the question.
3. The LLM answers using only the retrieved articles, with citations.
4. A fallback path detects low confidence or out-of-scope requests and hands off to a human with full context.

Grounding: the knowledge base is the product

The bot is only as good as the articles it can retrieve. Write support articles in question-and-answer form, keep them current, and structure them so retrieval can find the right chunk. If an article is outdated, the bot will confidently repeat it — grounding removes hallucination but not staleness, so article hygiene is a feature, not a chore.

Answer generation: honesty rules

• Constrain the model to answer only from retrieved content; if the answer is not in the articles, it must say so.
• Attach citations — show the article title and a link with every answer so users can verify and read more.
• Add an 'I do not know' affordance: a bot that admits limits and offers a human beats a bot that fabricates a fix.

Escalation: design the handoff, not just the chatbot

The handoff is where most bots fail. Escalate on explicit signals (the user says 'agent' or 'human'), on low retrieval confidence, on repeated failed answers, and on sensitive topics like refunds or account access. Hand the human the full transcript, the retrieved articles, and the bot's draft answer so they start mid-conversation instead of from scratch.

Cost control: know the price per conversation

• Most conversations are short: a question, one retrieval, one answer. Small models with a tight system prompt handle the majority cheaply.
• Cache the system prompt and common prefixes; reuse retrieval results across identical questions.
• Put a per-conversation token ceiling and a per-conversation turn cap in place — runaway loops are a real cost leak.
• Measure cost per resolved conversation and compare it against the cost of a human ticket; the bot wins only if the numbers say so.

Metrics that matter

Resolution rate — the share of conversations that end without a human — is the headline number, but track containment cost, escalation rate, and net promoter movement. Most important: hold a weekly review of conversations that escalated, because each one is a script for a better retrieval chunk or a better answer, and the bot improves fastest when you mine its failures.

FAQ