← All posts

· Feedbot team

RAG Chatbot: How to Train ChatGPT on Your Own Data

What a RAG chatbot is, how retrieval-augmented generation works step by step, and whether to build one with LangChain or buy a hosted tool.

A focused young man studies on a laptop amidst stacks of books in a cozy library setting
Photo: Tima Miroshnichenko / Pexels

TL;DR. A RAG chatbot answers questions from your own content. For each question it searches your documents, picks the few passages that match best, and asks a language model to answer using only those passages. That is what most people mean by “training ChatGPT on your own data”: the model isn’t retrained, it’s handed the right pages at the moment it answers.

  • RAG has five moving parts: chunking, embeddings, retrieval, grounding and a fallback for when nothing relevant is found.
  • Most bad answers come from retrieval, not from the model. Fix what gets retrieved before you touch the prompt.
  • Build it yourself with LangChain or LlamaIndex and a vector database if the bot is part of your product or your data is unusual. Buy a hosted tool if you need a support bot on your site this week.
  • Test it with real questions, including ones your docs can’t answer. A good RAG chatbot says “I don’t know” to those.

We make Feedbot, a hosted chatbot that works this way. This guide is mostly about the technique, and it applies whichever route you take.

What is a RAG chatbot?

Retrieval-augmented generation was named in a 2020 paper by Lewis and colleagues at Facebook AI Research. The idea is simple. A language model knows a lot in general but nothing about your refund policy, your API limits or the hours of your shop. Instead of trying to teach the model those facts, you look them up and paste them into the prompt every time.

A RAG chatbot is a chat interface built on that loop. It’s the standard way to build a support bot, a docs assistant or an internal “ask the wiki” tool, because:

  • Facts stay current. Change a page, re-index it, and the next answer uses the new version. No retraining.
  • Answers can be checked. You know which passages the model saw, so you can show sources and debug wrong answers.
  • The model can admit gaps. If retrieval finds nothing relevant, the bot can say so instead of guessing.

How retrieval-augmented generation works, step by step

Here is the whole pipeline. The first three steps run when you add content. The rest run on every question.

  1. Collect. Gather the content: website pages, help center articles, PDFs, Markdown docs, FAQ lists. Strip navigation, cookie banners and footers so only real content is left.
  2. Chunk. Split each document into passages small enough to fit several into a prompt, usually a few hundred to a couple of thousand characters. Split on headings and paragraphs, not in the middle of a sentence, and keep the heading with each chunk so “Refunds” still means something when the passage is read alone.
  3. Embed and index. Run each chunk through an embedding model. It returns a vector, a long list of numbers where texts with similar meaning end up close together. Store the vectors, the chunk text and metadata (source URL, title, date) in a vector index.
  4. Embed the question. When a user asks “can I get my money back after 30 days?”, embed that question with the same model.
  5. Retrieve. Find the chunks whose vectors are nearest to the question’s vector. Many systems also run a keyword search in parallel, because embeddings are weak on exact strings like error codes, SKUs and product names, and merge the two result lists. Keep the top few results, often between 3 and 10.
  6. Ground. Build a prompt that contains instructions (“answer only from the passages below; if they don’t contain the answer, say you don’t know”), the retrieved passages and the conversation so far. Send it to the language model.
  7. Answer and cite. The model writes the reply. Good systems attach the source of each passage so the user, or you, can check it.
  8. Fall back. If no passage scored above a relevance threshold, or the model reports that the passages don’t answer the question, the bot says it doesn’t know and offers another route: a human, a contact form, a link to search.

That’s it. Everything else in RAG engineering is about doing one of those eight steps better.

Chunking: the step people underrate

Chunk size is a trade-off. Small chunks are precise but lose context: a passage that says “this is not available on the Basic plan” is useless if the chunk doesn’t say what “this” is. Large chunks keep context but dilute the match and eat prompt space.

Practical rules that hold up:

  • Split by structure first (headings, list items, table rows), then by length.
  • Prefix every chunk with its page title and nearest heading.
  • Keep tables together, or turn each row into a sentence. A table split across two chunks answers nothing.
  • Use one page per topic in your source content. A page that covers shipping, returns and warranty in one block of text produces muddy chunks.

Embeddings find “money back” when the page says “refund.” Keyword search finds “ERR_402” when the user pastes the exact error. Neither is enough alone for a support bot, which is why hybrid retrieval (semantic plus keyword, results merged) has become the default in most serious setups. Some teams add a reranker, a second model that rescores the top 20 or so candidates against the question before the best few go into the prompt.

Grounding, citations and “I don’t know”

Grounding is the instruction that turns a chatty model into a careful one. The prompt should say three things plainly: use only the provided passages, say when they don’t contain the answer, and never invent prices, dates, or policies. Then test that it obeys.

Citations make answers verifiable. Even if you don’t show them to users, log which chunks were used for each answer. When someone reports a wrong answer, the log tells you in seconds whether the problem was missing content, bad retrieval or the model ignoring good passages.

The “I don’t know” path matters as much as the answers. Every question the bot couldn’t answer is a gap in your content, and a list of those gaps, reviewed weekly, is one of the most useful outputs of a support bot.

How to train ChatGPT on your own data

There are five common ways, and only one of them is training in the strict sense.

Approach What it does Good for Limits
Paste content into a chat You give the model the text in the conversation One-off questions about a document Doesn’t scale, nothing persists, not customer-facing
Custom GPT with knowledge files OpenAI lets you upload files that a GPT searches when answering Internal helpers for people who already use ChatGPT Lives inside ChatGPT, not on your website; limited control over retrieval
Your own RAG pipeline You run the eight steps above with an LLM API Products where the bot is a feature, unusual data, strict control You build and maintain chunking, indexing, evaluation and the UI
Hosted RAG chatbot A service crawls your site and files and runs the pipeline for you Support and sales bots on a website Less control over internals; monthly fee
Fine-tuning Trains the model on example inputs and outputs Teaching a format, tone or classification task Poor at storing facts that change; needs retraining for each update

For a bot that answers customers from your docs, fine-tuning is almost always the wrong tool. Use retrieval. If you also want a specific tone, put it in the system instructions first and only consider fine-tuning if that fails.

Build or buy a RAG chatbot?

What a DIY stack looks like

A typical self-built RAG chatbot in 2026 has these pieces:

Layer Common choices
Orchestration LangChain or LlamaIndex, or plain code calling an API
Embedding model An embeddings API from your LLM provider, or an open-source model you host
Vector store pgvector in your existing Postgres, Qdrant, Pinecone, Weaviate or Chroma
Ingestion A crawler, PDF and DOCX parsers, HTML cleanup, a scheduler for re-indexing
LLM Any chat model API
Evaluation A test set of questions plus a framework such as Ragas
Front end A chat widget, auth if users are logged in, conversation storage, a way to hand off to a person

The first demo takes an afternoon. The tutorials ranking for “rag chatbot” will get you there. The work that follows is what takes months: parsing messy PDFs, re-crawling changed pages, deduplicating near-identical content, handling questions in other languages, building the handoff and the inbox, watching costs, and keeping quality from drifting when someone edits the docs.

Build it if the chatbot is part of the product you sell, your data lives in systems a hosted tool can’t reach, you need control over every step (for example, per-user document permissions), or you have engineers who want to own it.

What a hosted tool gives you

A hosted RAG chatbot runs the same pipeline behind a dashboard. You give it a URL or files, it chunks, embeds and indexes them, and you paste a script tag on your site. You also get the parts that aren’t RAG but that a customer-facing bot needs: a widget, conversation history, human handoff and analytics.

Buy it if the goal is a support or sales bot on your website, your content is public pages and documents, and you would rather spend the engineering time on your product.

As an example of what “hosted” means in practice, here is how Feedbot handles it, as described in our knowledge base docs: sources can be website pages, sitemaps, PDF, DOCX, Markdown, TXT or CSV files, plain text and FAQ. Each source is split into passages of about 2,400 characters, each prefixed with its nearest heading. For each reply it runs a semantic search and a keyword search, merges them, and gives the model the 5 most relevant passages, plus a business profile and your custom instructions, which are included in every reply. When it doesn’t know, it says so and the question shows up as a gap. Website import and file upload start on the $5 Starter plan; the free plan uses text and FAQ you type in. More on the knowledge base chatbot page.

Other hosted options include Chatbase, Intercom Fin and Tidio’s Lyro, each priced differently. Our AI chatbot pricing breakdown compares what they cost at real volumes.

How to evaluate a RAG chatbot

Don’t judge a RAG chatbot by chatting with it for ten minutes. Build a small test set and run it every time you change content, chunking or the prompt.

  1. Collect 30 to 50 real questions from your inbox, search logs or sales calls. Include easy ones, ones phrased oddly, ones in other languages and at least 10 that your docs can’t answer.
  2. Write the expected answer (or “should say it doesn’t know”) and the source page for each.
  3. Check retrieval separately. For each question, did the right passage appear in the retrieved set? This is often called context recall. If it didn’t, no prompt will save the answer.
  4. Check the answer. Is it correct, is every claim supported by the retrieved passages (faithfulness), and does it actually address the question?
  5. Check the refusals. For the unanswerable questions, did the bot decline and offer a next step, or did it make something up?

Ragas and similar tools automate some of these scores with an LLM as the grader. They’re useful for spotting regressions, but read a sample of answers yourself too. Automated graders miss tone problems and confidently wrong answers that sound plausible.

Common RAG failure modes and how to fix them

Symptom Likely cause Fix
Says “I don’t know” when the answer is on your site Page never indexed, blocked by robots.txt, or content rendered by JavaScript the crawler didn’t run Check the source list; re-import; add the fact as text or FAQ
Answers with an old price or policy Index is stale Re-crawl after every content change; delete outdated pages
Mixes up two products or plans Chunks lost their headings, or one page covers several products Prefix chunks with titles; split pages by topic
Can’t find exact codes, SKUs or names Embedding-only retrieval Add keyword search (hybrid retrieval)
Right passage retrieved, wrong answer Too many passages, conflicting content, weak instructions Remove duplicates, tighten the grounding instruction, test a stronger model
Invents details not in your docs No fallback rule, or the model fills gaps with general knowledge Explicit “only from passages” instruction; test with unanswerable questions
Long, hedging answers Prompt asks for completeness over brevity Set a length and format in instructions
Good in English, worse in other languages Docs and questions in different languages Use a multilingual embedding model; check answers per language

Notice that most rows are content and retrieval problems. In our experience the fastest way to improve a RAG chatbot is to read 20 bad answers, look at what was retrieved for each, and fix the content.

What a RAG chatbot can’t do on its own

Retrieval gives a bot knowledge, not abilities. A plain RAG chatbot can tell a customer your refund policy. It can’t check whether their order qualifies, issue the refund or know who they are. For that you need identity (the bot knowing which logged-in user it’s talking to), actions (calling your API), and a human handoff for everything else. If you’re planning a bot that does those things, our pages on the AI customer service agent and customer service automation cover how the pieces fit together, and ChatGPT for customer service covers where a general chatbot stops being useful.

Sources

Questions

Can't find an answer? Ask the bot in the corner, or write to us.

What does RAG mean in a chatbot?

RAG stands for retrieval-augmented generation. Before the language model answers, the system searches your own documents for the passages most related to the question and adds them to the prompt, so the answer is based on your content instead of the model's general training data.

Is RAG the same as fine-tuning?

No. Fine-tuning changes the model's weights by training it on examples, which is good for teaching a style or format. RAG leaves the model alone and gives it fresh text at question time, which is better for facts that change, like prices, policies and docs.

Can I train ChatGPT on my own data?

Not in the sense of retraining it. You can upload files to a custom GPT, call a model through the API with your own retrieval step, or use a hosted RAG chatbot that crawls your site. All three are retrieval, not training.

What is a vector database in RAG?

A vector database stores each chunk of your content as an embedding, a list of numbers that represents its meaning, and finds the chunks whose embeddings are closest to the question's embedding. Popular options include pgvector for Postgres, Pinecone, Qdrant, Weaviate and Chroma.

Is a RAG chatbot agentic AI?

Not by itself. A basic RAG chatbot retrieves text and answers. It becomes agentic when it can also decide to take actions, such as calling an API, searching again with a different query or creating a ticket.

Why does my RAG chatbot still give wrong answers?

Usually because retrieval failed: the right passage wasn't in the index, was split badly, or ranked below less relevant text. Check which passages were retrieved for a bad answer before you change the prompt or the model.

Put one agent on your site today

Start on the free plan. Upgrade when conversations grow.

Start freeTalk to sales