Problem
FAQ chatbots usually fail in one of two ways: they make up answers (a fee, a limit or a policy that doesn't exist), or they go stale as the FAQ changes. Both are data-quality problems that look like model problems. In financial services, a confidently wrong answer about fees or fraud steps is a compliance and trust risk.
Approach
I built the full pipeline in n8n as two workflows, on a self-hosted Docker stack (n8n, Qdrant, PostgreSQL):
1. Ingestion. A web form accepts Markdown, PDF, Word, CSV or JSON files. Each file is split into overlapping chunks, embedded with OpenAI text-embedding-3-small, and stored in Qdrant with source, category and ingestedAt metadata. Re-uploading a file deletes its old chunks first, so the knowledge base never holds two versions of the same answer.
2. Chatbot. A Claude Opus 5.5 agent with conversation memory searches the knowledge base as a tool, answers in plain language and ends with Source: <file>. When retrieval finds nothing relevant, the agent returns a no-answer signal. The workflow then logs the question to a Postgres faq_gaps table and sends a safe fallback that points to human support, instead of a guess.
Built-in assurance
- Evaluation suite. 20 test cases (16 in-scope with the facts each answer must contain, 4 out-of-scope that must be declined). A Python script measures answer accuracy, refusal accuracy, citation rate and latency, writes a report, and fails with a non-zero exit code when quality gates aren't met, ready for CI.
- Guardrails. Retrieved text is treated as data, not instructions (prompt-injection resistance), and the bot never asks for PINs, passwords or one-time codes.
- Gap analytics. A SQL view ranks the most frequently asked unanswered questions, so the FAQ improves from real demand.
- Embeddable. The same bot runs as a hosted chat page or as a widget on any website.
Result
A reusable, documented RAG template that any team can run with docker compose up and point at its own FAQ, with the measurement built in from day one. The repository includes the importable workflows, a sample knowledge base for a fictional digital wallet, the evaluation harness and full setup documentation.
What this demonstrates
- Designing grounded AI systems where data quality is the foundation: idempotent ingestion, traceable metadata, gap feedback loops
- Treating LLM output like any other system under test, with explicit quality gates
- Shipping AI workflows that are secure, reproducible and handed over with documentation