Key takeaways
- RAG finds the passages in your documents that answer a question and has the model answer from them, with citations. The model is not retrained.
- For facts that change, use RAG. Fine-tuning changes how a model behaves, not what it knows; long context works for a few short, stable documents.
- When an answer is wrong, check the documents and the search before the model: duplicates, old versions, scanned PDFs and passages cut in the wrong place.
- Citations and an honest “I don’t know” make mistakes visible, and turn every unanswered question into a content to-do.
- Permissions must be checked before the search, so nobody can surface a document they couldn’t open themselves.
Teams that ask to “train ChatGPT on our documents” often want something else: an assistant that answers from their own policies, manuals or reports, gets the answer right, shows where it came from, and stays current when a document changes. Retraining a model is the wrong fix for that; RAG is the one that fits. Here it is without the jargon, compared with the alternatives, with the reasons it still goes wrong and a checklist for your documents.
What is RAG in AI, in plain English?
RAG, short for retrieval-augmented generation, is a way of making an AI model answer from a set of documents you choose, by finding the relevant passages first and handing them to the model with the question. The name comes from a 2020 research paper by Patrick Lewis and colleagues; the idea has since become a common way to build an assistant on a company’s own content.
Think of a library and a librarian. The model is a well-read librarian who has never seen your files. RAG gives the librarian a catalogue of your library. When a question arrives, the librarian looks it up in the catalogue, pulls the three or four pages that matter, reads them, and answers from them, with the page numbers.
In practice it runs in two stages:
- Prepare the library, once and then on every change. The system reads your documents, splits them into short passages, and stores them in a search index that can find passages by meaning, not only by exact words.
- Answer each question. It searches the index for the passages closest to the question, sends those passages and the question to the model, and instructs it to answer only from them and to say which passage each part came from.
The model itself is never changed. That is the point: your documents remain the source of truth, and when they change, the answers change with them.
Can ChatGPT learn your company documents?
Not by reading them once: a general chat model does not learn from your files, and retraining one on them is rarely the right way to get accurate answers. There are three real options, and they solve different problems.
- RAG looks up your documents at the moment of each question.
- Fine-tuning retrains a model on examples so that it behaves differently: a fixed format, a tone, a classification task. Our advice: fine-tuning changes how a model behaves, not what it knows, so it is the wrong tool for facts that change. OpenAI’s guide lists uses such as classification, a set output format and fixing instruction-following. OpenAI is also winding down its fine-tuning platform, which is no longer open to new users.
- Long context means pasting the documents straight into the conversation. Current models can read a lot of text at once, so for a small, stable set of documents this can be enough.
These options combine. A support assistant can use RAG for the facts and a carefully written set of instructions for the tone. A team with a single 30-page handbook may never need RAG at all, until the handbook becomes forty documents owned by six departments.
Why do RAG answers still go wrong?
RAG answers go wrong for three reasons: the documents are wrong, the search picked the wrong passages, or the model used the right passages badly. The model gets the blame, so check the documents and the search first: those are the cheapest places to fix.
Citations and an honest “I don’t know” are the two features that make the rest of these failures visible. The NFTR Assistant we built answers from its knowledge base, cites its sources and reports unanswered questions to admins, so the team can see which content is missing and add it. It sits inside the NITI Aayog Frontier Tech Repository portal, which we built. Every unanswered question becomes a to-do item for the content team, rather than a guess shown to a user.
Have documents in mind?
Tell us what your people ask and where the answers live, and we’ll talk through whether RAG, long context or something simpler fits.
What does RAG look like on a company policy library?
A worked example shows the difference best. Take a 400-person company with an HR and operations policy library, and follow the same four staff questions through three set-ups: a general chat model, the handbook pasted into a long-context model, and a RAG assistant with citations.
Who can see what in a RAG system?
A RAG assistant can show a user any passage in its index, so it must check what each user is allowed to see before it searches, not after it answers. If the board minutes and the salary bands are in the index and the search ignores permissions, an intern’s question can surface them.
- Carry the permissions over. Each passage keeps the access rules of the document it came from, and the search only looks in passages the person asking may open.
- Separate indexes for separate audiences. A public website assistant and a staff assistant should not share one index, even where much of the content overlaps.
- Keep what should never be searched out of the index. Personal data, contracts under negotiation and legal advice are easier to protect by leaving them out than by filtering them later.
- Log the questions and the passages used, and decide who can read those logs: questions people ask often contain sensitive details themselves.
How these rules are set up for a given client depends on where their documents live and who needs access.
Are your documents ready for RAG?
Your documents are ready for RAG when each answer exists in one current, readable place, with an owner who keeps it up to date. Check your library against the tests below before you build anything: the score tells you whether to start building or start tidying.
A low score is not a reason to give up. It is a list of the first month’s work, and the tidying pays off for the people who read the documents, too.
What does a RAG system cost to run?
A RAG system’s running cost comes mainly from model usage, which grows with the number of questions, the amount of text sent with each one and the model you choose, plus the search index and the time people spend keeping the documents current. Per question, it calls the model once per answer, where an AI agent calls it at every step of a loop, so plan for a lower cost per question than an agent.
Measure before you commit: run the assistant on a few hundred real questions, record the cost per answer, and multiply by the volume you expect. For what building one involves, see AI chatbot development, and for the wider build and running costs, the AI app development cost guide.
When is RAG not the right tool?
RAG is the wrong tool when the job is to do something rather than to answer something. If the assistant needs to file a ticket, update a record or chase a supplier, it needs to act through tools, which makes it an agent, with all the guardrails that brings: see what an AI agent is. RAG is also overkill for a handful of short documents that rarely change, where long context does the job, and it cannot fix a task that needs a different skill, such as reading handwriting or calculating a figure, which belongs in code. For the models and integrations behind all three, see our page on AI and LLMs.