Data engineering
What separates a mid-sized company from a large one is rarely the software. It is that in the large one the data arrives on its own, always by the same route, and that route is written down. In the mid-sized one a person moves it, every month, by hand, and the day that person is away nobody can repeat what they did.
Here you find out where every figure your company decides with comes from, how many hands it passes through, at which point it gets copied by hand, and what breaks when somebody changes a field or installs a module. With that in hand it gets fixed in the order that pays, not in the order that hurts.
And when automating is the right move, you get the load that stands on its own: scheduled, with retries, with quality checks, with an alert when it fails, and with the documentation for somebody else to run it without calling me.
Data x-ray: USD 1,200, two weeks, about 6 hours of your people's time. Pipeline that stands on its own: from USD 3,400, five to six weeks.
Almost nobody goes looking for data engineering. People go looking for why the sales report does not agree with the ledger, and why the answer depends on who runs the report.
Sales says one thing, accounting another and the warehouse a third. All three are usually right: they measure different things, with different cut-offs, and nobody wrote down which measures what. The meeting goes on that.
The ERP, a shared spreadsheet, the logistics provider's system and an inbox. To put them together somebody exports, pastes and adjusts, and in that pasting the trail of what changed is lost.
The process lived in their head and in a folder of files numbered by hand. Nobody could repeat it, and the following month the report came out late and different.
Somebody changes a field, installs a module, or a provider changes their file without warning. The load keeps running and the number keeps appearing. Only now it is wrong, and that gets discovered weeks later.
Launch offer: the first three to commission the data x-ray get three months of the monthly watch at no cost, the same one shared with the dashboards. It is not a discount: the price does not move. There is one signal that almost always means this work is worth it for you: if the question "which of the two numbers is the right one?" gets asked in your company more than once a month, you are already paying the cost, only in the hours of your best people and in decisions postponed.
Two steps. The first is a finding and is bought on its own: you can take it away and act on it with your own team or with another supplier.
Where every figure used for deciding comes from, how many hands it passes through, at which point it is copied by hand, what breaks when somebody changes a field, and in what order it is worth fixing. It is the sibling of the architecture map, one level lower down: at the level of the data.
Each figure's route drawn out, from the table where it is born to the report where somebody reads it. It is what lets you answer in ten seconds what breaks if a given field is touched. There is a full one published, for an invented company.
One to three sources into the agreed destination, with the load scheduled at whatever hour suits your operation and retries when the source does not answer.
Expected row counts, duplicates and nulls in mandatory fields. A load that arrives incomplete stops and says so, instead of writing half of it and leaving behind a number that is believable and false.
With the name of the load, the time, what was expected and what arrived. That the alert reaches you is part of the written guarantee further down, not a promise to keep an eye out.
How to launch it by hand, how to reprocess a day, what to check when something fails and what it connects to. Written so your own people can run it without calling anyone, which is the only proof that it was not left tied to me.
What you get runs on Python, on PostgreSQL, on scheduled tasks and on continuous integration runners. It is what is already in production in my own house, it is what I can keep standing without anyone's help, and it is what your company can operate without paying for a new licence or hiring a profile it does not have today.
This is not a stance against the orchestration platforms that are in fashion. It is that a mid-sized company putting up its first automated load on a platform nobody in the house can operate has swapped one problem for another: the data used to depend on a person, and now it depends on a tool only the supplier understands. If your team already runs one and wants to keep it, I say so on the first call and either work on yours or point you at someone who knows it better than I do.
The method does come from two books that have been read and are cited. From Fundamentals of Data Engineering, by Joe Reis and Matt Housley, the data lifecycle that structures the whole job: generation, storage, ingestion, transformation and serving, each stage with its own questions. And from Designing Data-Intensive Applications, by Martin Kleppmann and Chris Riccomini, the part that gives this line its promise: what breaks when a schema changes, and how to write a load that survives a source changing without warning.
Behind it there is also work of my own already done and running in production: bulk loads of documents left in draft with their reversal script written before starting, an Ecuadorian localisation for Odoo 19 with five hundred and eighty automated tests running on every change, and the upgrade radar, which is an analyser that reads modules and dependencies. None of that comes from a chapter: it comes from production.
A data project gets expensive through whatever nobody bounded at the start. This is bounded before quoting.
Your company contracts, pays for and signs the infrastructure. I do not put myself between you and your cloud provider, because the day we stop working together that turns into a rope.
In your name, always
Moving a company from one system to another is a different project, with a different risk and a different price. This line reads what your systems already hold; it does not replace them.
A different job
Stream processing doubles the cost of running it and almost no mid-sized company genuinely needs it: with an hourly load the decisions come out the same. If your case is one of the ones that truly demands it, it is quoted separately and with the caveats stated.
Quoted separately
I do not take on shifts or twenty-four-hour cover. Loads are scheduled at whatever hour suits your operation and the alerts are logged, but whoever answers at three in the morning has to be a team, not one person.
Honesty about capacity
What counts as a sale, what is excluded and what date it cuts at is defined by your company. I write it down, make it reproducible and point out where two departments contradict each other; the definition is yours, which is why it gets signed.
The definition is yours
The same importer and distributor that appears elsewhere on this site: Odoo running, eight years of history. No data comes from a real client.
The complaint it starts with: the monthly sales report closes on one number and the income statement on another, and the difference changes every month. Nobody is lying; nobody can explain it.
What the x-ray finds. The sales report comes out of a spreadsheet somebody exports on the second of the month, cut at the last day of the month and without deducting credit notes issued after the close. Accounting cuts on the same day but does deduct them. Returns reported by the logistics operator come in through a file that is pasted in by hand once a month, and in the months that file arrives late they land in the following month. Three routes, three different cut-offs, and a manual copy in the middle.
The order proposed. First the definition gets written and a single cut-off agreed, because that costs no money and removes half the gap. Then the manual copy of the logistics operator's file is taken out, because that is the one that breaks the month in silence. And only at the end does the rest get automated, once it is clear what is being automated.
The pipeline that ends up in place. Three sources: the ERP, the logistics operator's file and the list-price spreadsheet that still lives outside the system. Daily load at six in the morning, retries if the source does not answer, checks on expected rows and duplicates, and an email alert when something stops, carrying what was expected and what arrived.
And the test that closes it. At handover the month's sales are recalculated from source in front of you, and they have to give the same number as the report. If they do not, it is corrected before the second half is invoiced. With that, the figure stops depending on who runs the report, which is all that was being bought.
A guarantee that cannot be checked is a statement of goodwill. These three can be checked: either the number repeats or it does not; either the alert arrived or it did not.
If you cancel within fourteen days of payment and before receiving the first deliverable, I return the deposit in full.
At handover, any figure in scope is recalculated from source and gives the same number. If it does not, I fix it at no cost before invoicing the second half.
With the watch subscribed: if a load fails and the alert does not reach you, that month of subscription is not charged and the fix comes at no cost. That is what separates a watch from a promise to keep an eye out.
For this to apply: read-only access to the source systems is available on the agreed date, and whoever knows each source attends the validation session. Without those two things there is nothing to compare against, and the guarantee would mean nothing to either of us.
Any pipeline works on the day it is handed over. Serious work is told apart from the rest by what happens the first day something fails, and that can be asked before signing.
Ask to see the alert, not to have it described. A load with no alert is not automatic: it is a load somebody has to go and look at, and nobody looks after the third week.
That is what actually happens when something fails and gets retried. If running it twice duplicates records or duplicates an order, the pipeline is badly written. The right answer is that resending changes nothing.
Expected rows, duplicates and nulls in mandatory fields, at the very least. If the only check is that the process finished without an error, an empty file will pass as a successful load.
Ask which licences are needed, which profile has to be hired to operate it, and whether the documentation lets your people launch and reprocess unaided. A pipeline only the supplier understands is a monthly fee dressed up as a project.
It will happen: a field renamed, a new column, a file with a different separator. Ask whether that stops the load and raises an alert, or whether it comes in half done. The second produces numbers that are believable and false, which are the worst kind.
The lineage document describes your data and your operation, not a tool's catalogue. If what you are handed only makes sense inside the product being sold to you, you were handed sales material.
The finding is bought on its own. Plenty of companies stop there for the first year, fix what the document points at with their own team, and come back later for the automation.
Where every figure comes from, how many hands it passes through, where it is copied by hand, what breaks when somebody changes a field, and in what order it is worth fixing. Document and lineage diagram.
Up to three sources into the agreed destination, with the load scheduled, retries, quality checks, an alert when it fails and the documentation for somebody else to run it. More sources are quoted.
The alerts watched, whatever failed reprocessed, and the adjustment when a source system changes without warning. The same subscription covers the monthly reconciliation of your decision dashboards: one subscription for both. Paid a year ahead, USD 1,900.
An hour per system for read-only access, an hour of interview per source with whoever knows that system, and two hours of validation session with whoever uses the figures. That is about six hours at step 1 and about ten at step 2, spread over six weeks.
I say this before invoicing rather than after: the part that delays this work most is not the work, it is read-only access to each system. If your company cannot get those within two weeks, this is not yet the right purchase for you, and I would rather tell you on the call.
Prices in United States dollars; under my tax regime in Ecuador, RIMPE Negocio Popular, my services are invoiced with VAT at 0 %, so the figure is the final one. Half on starting and half at the validation session. Guarantee on reproducible figures: full refund of the deposit within fourteen days and before the first deliverable; any figure in scope is recalculated from source in front of you and gives the same number, and if it does not it is fixed at no cost before the second half is invoiced; with the watch subscribed, if a load fails and the alert does not arrive, that month is not charged. I take one engagement at a time: on the call I tell you the date of the next opening. Full conditions in the proposal and in the refund policy.
No, and the order matters. A dashboard shows the figure; this makes the figure the same wherever it comes from. If your data already comes out of the source clean, start with the one-decision dashboard, which is cheaper. If every report gives a different number, starting with the dashboard is painting over damp.
No. Whatever your systems hold is read, through their database, through their API or through the files they already produce. With Odoo I know the ground and the work moves faster.
Up to three at step 2. A source is a system with its own way of handing over data: your ERP, a logistics operator's file, the bank's export. The fourth and beyond are quoted, because each one brings its own way of breaking.
Yes. The pipeline runs on your infrastructure, or on whatever you contract in your own name, and the credentials are yours. If you would rather I hosted something, I put in writing beforehand where each piece of data ends up.
It happens often, and it shows in the first week: the data is wrong because somebody records it late or twice, not because the system fails. In that case I say so, and the work that suits you is process mapping, not this.
Yes, and it is a legitimate purchase. The document says what to fix and in what order; your team can carry it out, or you can ask anyone you like for a quote with that document in hand.
It breaks when the world changes: a new module, a field somebody renames, a provider who changes their file. That is why the monthly watch exists, and why its guarantee is that the alert arrives rather than that nothing ever fails.
10 /
Tell me which figure does not agree between two reports and which systems it lives in today. I reply with a proposal within one business day.
contacto@andresbrinez.comIf you would rather first see how a clean figure gets read, that is the page on data you decide with.
Six years building systems for companies in three countries: Colombia, Ecuador and Australia · Member of the Odoo Community Association