The AI Chatbot Build Checklist: 17 Things to Get Right Before Launch

Most AI chatbots fail for boring reasons. Not because the model was too weak, but because nobody wired up an escalation path, nobody capped the token spend, and nobody wrote a single evaluation before shipping. We have now taken more than a dozen customer-facing chatbots to production — support bots, onboarding assistants, internal knowledge bots — and the failures we have seen (including two of our own early ones) almost always trace back to a skipped item on this list. This is the exact pre-launch checklist we run at Fajarix. Steal it.

Grounding and Knowledge (Items 1–4)

1. Ground every factual answer in retrieved content

If the bot answers questions about your product, pricing, or policies from the model's pretraining, you will ship confident nonsense. Every factual claim should come from a retrieval step — your docs, your help centre, your database — injected into the prompt with the model instructed to answer only from that context. When we retrofitted retrieval onto a client's bare-LLM support bot, hallucinated policy answers dropped from roughly 1 in 12 conversations to fewer than 1 in 200.

2. Give the bot an explicit way to say it does not know

Your system prompt must contain a literal instruction for what to do when the retrieved context does not answer the question — and your evals must test it. Bots without a sanctioned escape hatch invent one. The phrase we use is some variant of: if the provided context does not contain the answer, say so and offer to connect the user with a human.

3. Version and date-stamp your knowledge base

Stale content is the slow killer. Pricing changes, feature launches, policy updates — if there is no owner and no re-indexing pipeline, your bot will confidently quote last quarter's prices. We set a hard rule: every document in the retrieval index has an owner and a review date, and the ingestion pipeline re-runs on every content change, not on a cron schedule.

4. Test with real user phrasing, not your phrasing

Your team asks questions the way your docs are written. Users do not. Pull 100 real queries from your support inbox or search logs and run them through the bot before launch. On one project, the bot scored well on our internal test set and failed 40% of real queries — because customers called the product's core feature by a completely different name than the docs did. We added synonym expansion to retrieval and rewrote three doc titles. That is a two-day fix before launch and a reputation problem after it.

Escalation and Safety (Items 5–8)

5. Build human escalation before you build anything clever

The single most important feature of a customer-facing bot is a working handoff to a human — with full conversation context attached. Not a dead-end email address. The ticket or live-chat handoff should include the transcript, the retrieved documents, and the bot's confidence signals, so the human does not restart from zero. We build this in week one of every chatbot project, before prompt tuning even starts.

6. Define hard escalation triggers, not just soft ones

Some situations must never be handled by the bot regardless of how confident it is: legal threats, medical or safety issues, mentions of self-harm, requests to cancel or delete an account, repeated user frustration. These are keyword-and-classifier triggers that route to a human immediately. Decide the list with your legal and support leads, write it down, and test each trigger explicitly.

7. Add guardrails on both input and output

Input guardrails catch prompt injection and off-topic abuse (people will absolutely try to use your support bot as free ChatGPT). Output guardrails catch the model leaking system prompts, generating competitor comparisons you never approved, or making commitments — refunds, discounts, legal promises — it has no authority to make. A cheap classifier pass on every response costs fractions of a cent and has saved our clients from screenshots on social media more than once.

8. Scope the bot's authority in writing

Can the bot issue refunds? Up to what amount? Can it change account settings? Reset passwords? Every action the bot can take needs an explicit allowlist with limits, agreed with the business owner, enforced in code — not in the prompt. Prompts are suggestions; API-level permission checks are guarantees.

Evaluation and Quality (Items 9–12)

9. Write evals before you tune prompts

You cannot improve what you cannot measure. Before any serious prompt iteration, build an evaluation set: 50–150 real questions with graded reference answers, split across easy, hard, and must-escalate cases. Run it on every prompt or model change. Without this, prompt tuning is vibes, and every fix silently breaks something else.

10. Use an LLM judge, but calibrate it against humans

Automated grading with a strong model is how you make evals affordable, but validate the judge first: have a human grade 50 outputs, have the judge grade the same 50, and measure agreement. We aim for at least 85% agreement before trusting the judge for regression testing. Below that, tighten the grading rubric.

11. Test the multi-turn failure modes

Single-question evals miss the ways real conversations degrade: users changing topic mid-thread, correcting the bot, pasting in huge blocks of text, or asking follow-ups that depend on three turns of context. Script at least 20 multi-turn scenarios. The most common production bug we see is context loss on turn four or five — the bot forgets a constraint the user stated earlier and gives an answer that contradicts it.

12. Run a red-team hour with your own team

Before launch, put five colleagues in a room for an hour with one instruction: break it. Extract the system prompt, get it to badmouth a competitor, make it promise a refund, make it swear. Log everything and fix the top findings. This costs one afternoon and finds things no eval set will.

Operations and Economics (Items 13–17)

13. Cap costs per conversation and per day

Set a hard token budget per conversation (we typically start around 30–50K tokens including retrieval context) and a daily spend ceiling with alerting at 50% and 80%. One runaway retry loop or one user scripting requests against your endpoint can turn a $300/month bot into a $3,000 surprise. We have seen it happen; the cap turns an incident into a log line.

14. Add rate limiting per user and per IP

Related but distinct from cost caps: without per-user rate limits, your bot becomes a free LLM API for anyone who finds the endpoint. Limit conversations per user per hour and messages per minute, and require some form of session identity even for anonymous visitors.

15. Log every conversation with structured metadata

Full transcripts plus: retrieved documents, model and prompt version, latency, token counts, escalation events, and user feedback. This is your debugging tool, your eval-set mining source, and — depending on your sector — a compliance requirement. Store it from day one; you cannot retroactively log the launch week where everything interesting happened.

16. Instrument the metrics that map to business value

Deflection rate alone is a vanity metric — a bot that frustrates users into giving up 'deflects' brilliantly. Track resolution rate (did the user's issue actually get solved), escalation rate, user satisfaction on a per-conversation thumbs rating, and — the one most teams skip — the reopen rate on conversations the bot marked resolved. A healthy support bot in our experience lands at 40–65% genuine resolution, not the 90% some vendors promise.

17. Plan the model swap before you need it

Models get deprecated, prices change, and a better model will ship within months of your launch. Keep the model name, prompts, and generation parameters in config, not scattered through code, and keep your eval suite ready so you can qualify a new model in a day. Teams that hardcode everything spend weeks on what should be an afternoon.

How to Actually Run This Checklist

A checklist nobody schedules is a blog post, so here is the cadence we use. Items 1–8 are build-phase requirements — they go into the project plan as tickets with owners, and the escalation path (item 5) is the first thing built, not the last. Items 9–12 run as a formal gate two weeks before launch: we block the release date until the eval suite passes agreed thresholds and the red-team findings are triaged. Items 13–17 are launch-week configuration, verified in production, not staging — cost caps and rate limits have a way of existing in config files and not in the actual deployed environment.

Then launch gradually. We never flip a chatbot to 100% of traffic on day one. The standard rollout: internal staff for a week, then 5% of real users, then 25%, then full — with resolution rate, escalation rate, and cost per conversation reviewed at each step. At the 5% stage on a recent deployment, we caught a retrieval bug that returned outdated pricing pages for one product line. It affected 3% of conversations. At full traffic that would have been a customer-trust incident; at 5% it was a Tuesday fix nobody outside the team ever heard about. Staged rollouts convert catastrophes into anecdotes, and they cost you nothing but two weeks of patience.

The Honest Summary

Notice what is not on this list: choosing the perfect model, fine-tuning, or exotic agent frameworks. In our experience those decisions move the outcome far less than grounding, escalation, evals, and cost controls. A mid-tier model with all 17 items done beats a frontier model with five of them skipped — every single time. Budget roughly 30% of the project for the bot's core intelligence and 70% for everything on this list, and your launch will be uneventful in the best possible way.

If you are scoping a chatbot and want a partner who treats this checklist as the floor rather than the ceiling, our AI automation team designs, ships, and operates production conversational systems end to end.

Ready to put these insights into practice? The team at Fajarix builds exactly these solutions. Book a free consultation to discuss your project.