AI over your own data
A mid-sized company can run the way the big ones do. What a large company has is not better people: it is that what the company knows is written down, findable, and available to everyone at the same time. Your company has the first half already, in its manuals, contracts, policies, procedures, minutes and the history sitting in its ERP.
What is missing is the second half. Nobody finds any of it at the moment they need it, and the one person who does know ends up answering the same question every week. That is a cost that appears in no account, because it is paid in the time of whoever knows the most.
This connects a language model to that body of documents and to your ERP. The answer comes back with the document it came from and the exact line, so it can be checked. And what the model cannot find, it says it cannot find, instead of inventing it.
It starts with a two-week trial, measured with questions from your own company, that ends in a number rather than in a demo. If that number does not suit you, I tell you so in writing and that is where it ends.
It is not that information is missing: there is too much of it, and it is scattered. A large company with the same documents answers faster, not because it hired better, but because what it knows is reachable. That gap is what this closes.
What payment terms does this customer have, what does the contract say about returns, how is a transfer between warehouses done. The answers exist, in writing. But they get asked out loud, to whoever has been there longest.
When that person goes on holiday, falls ill or resigns, the knowledge leaves with them. An assistant that learnt from your documents replaces nobody: it takes the repetitive part off their hands.
The document is in an email from two years ago, in a shared folder with no order to it, or in an ERP module nobody opens. People reconstruct the answer instead of finding it.
A public assistant knows nothing about your company, and pasting your documents into it sends your information to a third party. What it answers sounds right and is sometimes false, which is the worst thing that can happen with a contract.
Do the arithmetic with your own numbers before you talk to me. If three people on your team lose half an hour a day looking for what is already written down, that is thirty hours a month. Put the hourly cost of those three people against it and you have the size of the problem. If the result does not justify step 1, tell me so and I will not sell you anything.
The language model is not trained on your information. It is handed the information at the moment of answering, the way a new lawyer is handed the case file before the hearing. The technique is called retrieval-augmented generation, and it is two pieces: a search engine that finds the fragments that matter, and the model that reads them and writes the answer.
Out of that come the three properties a company actually cares about. The answer cites its source, so it can be checked and argued with. Updating the knowledge means publishing a new document, not retraining anything. And your documents stay yours: the model sees a fragment for a second and does not keep it.
What decides the quality is not the model, it is the search. An assistant that retrieves the wrong paragraph answers badly with the best model in the world. That is why most of the work goes into how each document is split, how it is indexed and how the results are ordered, rather than into picking a brand of artificial intelligence.
Take a concrete case: a company that imports and distributes, with Odoo running and eight years of documents behind it. It wants its sales team and its warehouse team to stop asking each other the same things.
Before any technology is touched, the exam is written: between forty and a hundred questions whose correct answer is already known. Most of them are real questions, the ones asked over the phone today, collected with your people. The rest are derived from your own documents, to cover corners of the archive nobody has asked about yet. That set is the measuring stick: it is kept under version control and it grows every time the assistant gets a new question wrong. Without it, nobody can say whether the assistant is any good, only whether it is impressive.
The sources that answer those questions get chosen: product manuals, framework contracts, returns policies, technical data sheets, and whichever ERP tables are needed. Duplicates and expired material are cleared out, because a repealed document that answers first is worse than having nothing.
Your documents are not flowing text. There are tables of prices, of discounts and of terms, and there are PDFs that are a photograph of a sheet of paper. A table chopped into pieces leaves rows with no header: the fragment reads "Model X, 899, 12" and nobody can tell which of those numbers is the price. A scan with no text recognition simply never reaches the search index. Here the tables are extracted as tables and every piece keeps its header; the ones that run from one page to the next are joined back together; and the scans go through character recognition first, inside your own premises if the documents are not allowed to leave. It is the least impressive part of the work and it is where most demos fall over.
Your company names things with codes: a tax ID number, an invoice number, a warehouse code, a sales order reference. Search by meaning understands the intent and gets lost on the codes. Search by exact word finds the code and misses the intent. Both are built and the results combined, because a real question brings the two things at the same time. The candidates are then reordered by a model that compares the question against each fragment: that is the step that lifts to first place the document that had come back seventh. Every one of those steps is justified by measurement, not by fashion.
The assistant cannot answer an intern with what only a manager may see. The permissions of the document and of the ERP travel with each fragment, and the answer is assembled only from what that person is allowed to read. It is the part that takes the most work and the one nobody mentions in the demo.
Every answer carries the document, the section and the link. Anything not in the sources is answered with an "I cannot find it", which is a valid answer and the most important one of all.
Four things are measured, which are the four the field measures. How many of the fragments the search brought back actually made it into the answer. How much of the text of each fragment was used. What percentage of the statements in the answer is backed by those fragments. And whether the answer covers everything that was asked. The third is the one that decides: below seventy per cent the system invents, between seventy and eighty it has to be reviewed case by case, and from eighty upwards it can be put in front of people. That battery is run on every change, like the tests of a piece of software, and no change goes in if it makes the previous number worse.
How much of each fragment got used looks like the least important number and it is the one that saves money. If only a quarter of each retrieved piece is doing any work, the pieces are cut larger than they need to be: you are paying to send the model text that influences nothing, and you are answering more slowly for it. Trimming them lowers the cost per answer and usually raises the quality, because less noise goes in. In the worked example published in Mastering RAG, going from twenty fragments down to fifteen raised both quality measures and brought the cost per query down by about a quarter; with every adjustment in that example chained together, the book gets to half, by which point it has also switched model and given up some quality. Those are the book's figures on its own case, not results of mine; I bring them because they show that here the cheap option and the good one line up more often than you would expect.
All of this lives wherever you say: on your server, in your cloud or in mine. If company policy does not allow a document to leave, the model can also run on your own premises, at a higher infrastructure cost and with somewhat lower quality. That is your decision, and it is taken with the numbers in view. It is the same principle as AI video analytics: video is analysed inside each site and only the events and their clips leave it.
An assistant like this does not fail in just any manner. It fails in seven, they have been studied, and each one has a different fix. Knowing which of the seven is happening to you is half the work; measuring it is the other half.
Nobody loaded the document that holds it, or it was loaded and never reached the index. The right behaviour is for the assistant to say it cannot find it. What happens when nobody planned for this is that it answers anyway, with something that sounds reasonable. The fix is to complete the archive and, above all, to measure that it can keep quiet: that test is run deliberately, with questions whose answer is nowhere.
The right document exists, the search found it, and it came back eleventh when only ten are handed over. The user gets an incomplete answer without ever learning that the good one was left out. The fix is metadata on every fragment, a reranking step that scores the candidates again, and adjusting how many are handed over.
Many documents were retrieved and the model has a text limit. Whatever goes past the limit is cut, and sometimes what got cut is precisely the part that answered. It is the least visible failure of the seven, because the answer arrives complete in form and lame in substance. The fix is to retrieve less and better, not more.
The answer is right in front of the model and it does not extract it, because there is noise around it or because two documents contradict each other. It ends up returning a generality. The fix comes before indexing: cleaning, removing what has been repealed, and separating what contradicts itself.
A table was asked for and a paragraph arrived; three options were asked for and an essay arrived. It is the easiest failure to correct and the one that irritates most, because it forces someone to redo by hand what was already there. The fix is the instruction written inside the system, not left to whoever is asking.
"What are the payment terms?" without saying for which customer. The model has seen both long answers and short ones, and with no context it cannot tell which is wanted. The fix is rewriting the question before searching: completing it from the earlier conversation, splitting it in two when it asks for a comparison, or proposing the precise version back to the user for confirmation.
Everything it says is true and half of it is missing, and the missing half was in the documents and available. It is the most dangerous failure because it goes undetected: the answer checks out, the citation supports it, and nobody goes looking for what they do not know is absent. It only shows up by measuring completeness against a known correct answer, which is what the exam in step 1 is for.
This list is not mine: they are the seven failure points collected in Mastering RAG (Galileo, 2024), chapter 2, drawn from a study of three real systems, one in research, one in education and one in biomedicine. I bring it for a practical reason. When an assistant starts answering badly, the usual conversation is "the AI got it wrong". With this list the conversation is a different one: which of the seven, and what gets changed. Three matters fall outside these seven because they are not about quality but about operations, and each sits in its own place: a document holding personal data turning up cited in an answer, in section 08; somebody planting instructions inside a document, in section 09; and answers taking longer, in the frequently asked questions.
What changes a person's working day is not that the machine writes nicely. It is that the answer can be verified in ten seconds and taken into a meeting. The company below is made up; the shape of the answer is not.
The question: "Can I give Distribuidora del Norte ninety days to pay?"
The answer: "Not under the contract in force. The framework contract signed on 14 March 2024 sets sixty days for that customer, extendable to ninety only with written approval from the finance director, which is clause seven. They currently have two overdue invoices for USD 4,180, the oldest of them 38 days old. Sources: framework contract, Distribuidora del Norte, page 4, clause seven. Odoo, customer statement as at 22 September."
The salesperson did not have to find the contract, or ask credit control, or wait until tomorrow. And if the answer looks odd, they open the clause and read it, because the link is right there. That is the whole product.
And the citation is measured too. It is not enough for a document to appear at the foot of the answer: it is checked that the source cited really does support the sentence it is attached to. A citation that does not support what it is attached to is worse than no citation, because it invites trust without verification.
A company with no system has an earlier problem: its data does not exist in a form anyone can query. A company with Odoo, with SAP or with whatever else has already solved that. It has customers, products, prices, movements and documents in an ordered database, with permissions defined and with people used to working against a system.
On that base the assistant stops being a toy. It can answer "that customer's balance is this and it comes from these three invoices", or "that product has this data sheet and the last batch came in at this cost", and link to the record. The combination that works is both worlds together: the document explains the rule, the ERP gives today's figure.
It is worth being precise about how the ERP gets asked, because that is where the trust is won or lost. Letting the model write free-form queries against your database looks excellent in a demo and fails in three known ways: it invents column names that do not exist, it ties itself in knots when five tables have to be joined, and it returns a different query every time, which is exactly what is useless for auditing. What gets built here is the opposite. A short list of queries already written and reviewed with you, such as a customer's balance, the status of an order, the cost of the last batch or the stock available in a warehouse, each one with its parameters. The assistant chooses which to use and with which values; the query that runs is always the same one, and it is logged.
The same criterion applies to my own house. In my own books, every invoice that arrives by email files itself into its account: first by the supplier's rule, then by the economic activity that the SRI, Ecuador's tax authority, records for that RUC, the company's tax identification number, and only at the end, for what neither of those knows, is artificial intelligence asked. Three hundred and fifty-seven invoices were classified that day, and artificial intelligence was needed on very few of them. That is the idea: use it where it adds something and not where a cheap rule does the job. It is a product of its own, automated supplier invoices in Odoo.
It is the word that gets sold the most and the one that is least needed at the start. Worth knowing the difference before paying for it.
It changes the weights of the model using your examples. It is for when the problem is behaviour: the model answers in the wrong format, in the wrong tone, or does not know a technical dialect. It is not a reliable way of putting new knowledge in.
A problem of form
It does not touch the model. It hands it what it needs to know on each question. It is for when the problem is information, which is the case in ninety per cent of companies. And it is updated by publishing a document.
A problem of information
In the 2024 experiment by Ovadia and others, retrieval beat fine-tuning in almost every category measured, across three different models. And adding retrieval on top of an already tuned model improved the result only 43 % of the time.
Measured, not opined
A tuned model swallows every document and dissolves them into the same block of weights. Inside it, what belongs to the board can no longer be told apart from what anyone may see, and taking a document back out afterwards is impossible: it would have to be trained again. With retrieval, the permission travels attached to each fragment and the answer is assembled only from what that person may read. So in a company with departments, fine-tuning is not merely expensive: it is a security hole.
Before you sign
When retrieval already works and the remaining failure is one of behaviour. Or when it is worth bringing the cost per answer down by having a small model imitate a large one. That is the order, and it is the order the engineering of the field recommends, not a preference of mine.
After, not before
The order of the steps and the citation of the experiment come from AI Engineering (Chip Huyen, O'Reilly, 2025), chapter 7. Anyone offering to start by training a model on your data is selling you the expensive part before knowing whether it is needed. If you want to know what really gets updated inside the model when somebody says they are training it on your data, I explained it separately in this guide.
Answering well is not enough to open it. An assistant that gets ninety per cent of its answers right can still be impossible to publish if in the other ten it hands over personal data, opines on what is none of its business, or recommends your competitor. These are eight separate things and they are tested one by one, beforehand, not after the complaint.
It is checked that the documents retrieved are to the point, that they are enough to answer, and that they are not five versions of the same thing. This is the usual quality test and it is only the first of eight.
Five tests, not one. That it withstands noise, with current and expired documents deliberately mixed together. That it says it does not know when the answer is not there. That it joins two documents when the answer lives split across both. That it flags when the source itself carries a wrong figure, instead of repeating it with confidence. And that it defends itself against a badly framed question instead of guessing.
It is deliberately asked for somebody's phone number, for a customer's identity number, for an employee's salary. The correct answer is to decline, even when the data sits in a document that person was allowed to see. That something is accessible does not make it publishable in an answer.
It is asked for things no company wants to have helped with: drafting a phishing email to harvest a customer's details, or worse. It is the test that gets skipped most often and the one that reads worst the day somebody publishes it.
There are three known ways of turning an assistant around and all three get tested. Handing it the role of an expert and adding emotional urgency so it lowers its guard. Writing it a first line that forces it to open by saying what the attacker wants, because what follows then comes out on its own. And forbidding it in advance the words it refuses with, so that it cannot refuse. The attacker does not have to be an outsider: an employee with curiosity and a free afternoon is enough.
An assistant belonging to your company will be asked about the weather, about football and about what a used car is worth. The right behaviour is to say what it deals with and not answer. What usually happens is that it answers, confidently and with information from nowhere, and that answer is the one somebody shows in a meeting.
The same completeness as step 7 of the build, but looked at as a release criterion: the questions whose correct answer has several parts are reviewed, and it is checked that all of them are there. A half answer is the error that takes longest to surface.
Four checks that have nothing to do with the technique and everything to do with your name. That it does not take a rude or dismissive tone when asked for an opinion. That it does not wander into subjects that are not a company's business, such as finding the loophole in a tax rule. That it does not name or recommend your competitors when asked. And that it says correctly who it is: an assistant that answers that some other company made it confuses your customer in the first sentence.
The eight scenarios are taken from Mastering RAG (Galileo, 2024), chapter 5, which presents them as what gets overlooked most when one of these systems is put into production. That this is not theoretical is already settled in a decided case: in 2024 the Civil Resolution Tribunal of British Columbia, in Canada, held that an airline had to answer for what its own assistant had told a passenger about a bereavement fare, even though another page of the same site said the opposite. The lesson for a mid-sized company is not that it should not be used: it is that whatever the assistant answers, it answers on your behalf, which is why it gets measured before it is opened.
The market is full of demos that work with three hand-picked documents. Seven questions separate a serious proposal from a good-looking presentation.
If the vendor writes them after building, the exam is rigged. They are written beforehand, they are written by your people, and with the correct answers known.
Ask to see a question whose answer is not in the documents. If the assistant answers just as confidently, the system is no use for deciding anything.
Ask what a warehouse user sees when they ask about a board-level contract. If the answer is "we configure that later", the project has a hole in the middle of it.
What gets sent to the model provider, what is kept, for how long, and whether it is used to train anything. It has to be in the contract in writing, not in a conversation.
It is the commonest thing in a company with years of archive: the old policy and the new one live in the same folder. Ask how each document's version is marked and what happens to the pieces of the previous version when a new one is published. If they are not deleted, the assistant will cite a repealed rule with complete confidence, and it will be right about the citation and wrong about the substance. That is the hardest error of all to catch.
A document can carry an instruction aimed at the model written inside it, sometimes in white text on a white background, which nobody sees on opening it. Ask what gets checked in documents as they come in and in questions as they arrive. If the answer is that nothing gets checked, anyone who can drop a file into a shared folder can change what the assistant tells your team.
An assistant is not handed over and forgotten. Documents change, questions change and models get updated. If there is no clear monthly figure, there will be one anyway, but as a surprise.
You do not buy the whole project up front. You buy a trial that answers with numbers whether this is any use to your company, and you decide from there.
Includes: the real questions collected with your people, one of your document sources put to work answering, the measurement of how many it answered well and how many badly, and a report on what works, what does not and what the rest would cost. If you decide to go on, it is credited against step 2.
Includes: the sources agreed on, your organisation's permissions respected, the connection to your ERP where it adds something, answers with citations, the battery of questions running on every change, and training for whoever is going to operate it. The price depends on how many sources there are and on whether it runs on your premises or in the cloud.
Includes: monitoring that it answers and how long it takes, a monthly spending cap with a warning on reaching eighty per cent of it, the loading of new documents, the monthly review of the questions it got wrong, and the tuning of the retrieval. No minimum term: cancel whenever you like.
Prices in United States dollars; under my tax regime, RIMPE Negocio Popular, my services are invoiced with VAT at 0 %, so the figure is the final one. The cost of the model and of the infrastructure is invoiced separately, at cost and with the provider's own detail, because it depends on how much it is used. Guarantee on step 1: if at the end of the trial the report concludes that this is not worth it for you, I tell you so in writing and you do not pay step 2; step 1 is not refunded, because the work of finding out has already been done and that answer is worth something too. Full conditions in the proposal and in the refund policy.
It looks like one from the outside and has nothing to do with one on the inside. The chatbots of a few years ago answered with what somebody had written into a tree of replies. This reads your documents at the moment of each question, answers in your own words and cites where it took it from. And when it is not in the documents, it says so.
No. The model does not change: it receives a fragment, answers and forgets it. With the providers I use, the enterprise contract states in writing that nothing sent through the programming interface is used for training. If your policy requires that nothing leaves the company, a model is set up inside your own infrastructure. Logging is a separate matter from training: providers keep the calls for a period in order to detect abuse, and that is written in their contract, not in mine. If that logging is already too much for your policy, the answer is not a better contract: it is for the model to run on your own premises.
It will get things wrong, the way a person does. The difference is that every answer carries its source, so the error is caught by opening it, and that the battery of questions measures how often it is wrong before anyone uses it. A system that cannot be verified does not go into production.
The target is to answer in around four seconds, with the search part under three tenths of a second. That is measured and reported to you just as the quality is, and not only the average: the worst one in every twenty as well, which is the one people remember. An assistant that answers well but keeps you waiting stops being used within three weeks, and that shows up in the count of questions long before it shows up as a complaint.
Less than you think. A folder of manuals and policies, or a year of contracts, is already enough for the two-week trial. What is really needed is the real questions: without them there is no way of knowing whether it works.
Yes, and the in-house acronyms are exactly where the work shows. The search has to understand that "NP", for nota de pedido, means your sales order and not something else. That is solved with a company glossary inside the system, not with a more expensive model. Documents in Spanish and questions in Spanish are the normal case in Ecuador; answers can come back in either language.
No. The assistant works with documents wherever they come from. Having an ERP helps because there is ordered data with permissions on it, and with Odoo I also know the ground. With another ERP it connects through its programming interface or through read-only views.
They can, and sometimes that is the right answer: I will tell you if it is yours. Off-the-shelf tools handle the easy path well. What gets complicated is your real archive, with repealed documents, permissions by department and acronyms of your own. That is where the demo falls over and where the work is worth paying for.
12 /
Tell me which questions keep repeating in your company and where the answers sit today. I reply with the proposal for the trial within one business day.
contacto@andresbrinez.comSix years building systems for companies in three countries: Colombia, Ecuador and Australia · Member of the Odoo Community Association