Data platform
Data Engineering and Migration Services
Almost every business past a certain size has the same problem: the data exists, but it is in six places, and no two of them agree.
The CRM says one revenue figure, the accounting system says another, and the spreadsheet the finance team actually uses says a third. Nobody is lying. Each system is counting something slightly different, and no one has written down which definition is correct.
Data engineering is the work of bringing those sources into one place and, more importantly, resolving the disagreements so the answer is the same regardless of who asks.
Getting the data out is the easy part
Extraction is largely a solved problem. Most systems have an API, an export or a database connection, and connecting to them is a known quantity.
The difficulty starts immediately after. Records that represent the same thing but do not match on any field. Dates stored in three formats and two time zones. Free-text fields where a dropdown should have been. Deleted records that were never really deleted. Historical rows written under rules that changed two years ago and were never backfilled.
Budgeting for extraction and not for reconciliation is the most common way these projects run long.
Reconciliation, and why it needs a person
When two systems disagree, something has to decide which is right, and that something has to be a human who understands the business.
Typical decisions: the CRM owns the customer relationship, the accounting system owns the money, so revenue comes from accounting and segmentation comes from the CRM. A sale is dated by invoice, not by order. A cancelled order is excluded from revenue and retained for conversion analysis. Two records with the same email are the same customer even if the names differ.
None of that is technical. All of it has to be agreed and written down, because it becomes the definition every future report inherits.
Migration without a gap in the history
Migrations fail in a specific way: the new platform works, and the historical data did not come with it, so every comparison to last year is broken.
The approach that avoids it is unglamorous. Move history first and reconcile it against the old system before anything switches over. Run both in parallel long enough to prove the numbers match. Only then cut over. Keep the old system readable for a defined period afterwards.
The reconciliation step is the one under pressure to skip, and the one that determines whether anyone trusts the new platform.
Pipelines built to be operated
A pipeline that works on the day it is built and fails silently three weeks later is worse than no pipeline, because reports keep rendering against stale data and nobody notices.
So everything ships with failure alerting that reaches a person, idempotent runs so a retry cannot double-count, incremental refresh where the source supports it, and freshness checks that make a stale table visible on the dashboard that reads it.
Data quality is a running concern, not a phase
Teams often treat data quality as a cleanup done once before launch. It is better understood as a continuous property that degrades without attention.
Source systems change without warning. A field that was mandatory becomes optional. A new team starts using a status code in a different way. An integration breaks and fails silently, so a table stops updating while every report that reads it keeps rendering.
The practical answer is a small set of checks that run with every load and are visible to the people reading the numbers: row counts within expected bounds, required fields populated, referential integrity holding, and freshness within the expected window. When one fails, someone is told, and the dashboard reading that table shows it is stale rather than quietly serving old numbers.
That last detail matters more than it sounds. A silently stale dashboard is more damaging than one that is visibly broken, because people keep making decisions from it.
The first thing to do
Write down, in one page, which system is authoritative for each thing your business measures. Revenue, customer, date of sale, what counts as active.
Most businesses cannot complete that page without a conversation, and the gaps are exactly where the reporting disagreements come from. Bringing an attempt at it to the first conversation makes the scoping much faster, and occasionally it turns out to be most of the fix.
What the engagement includes
- A source inventory. Every system holding relevant data, how it can be read, what it is authoritative for, and where it disagrees with the others.
- Agreed reconciliation rules. The entity, date and inclusion decisions written down and signed off, since these become the definitions every future report inherits.
- Pipelines built to be operated. Idempotent runs, incremental refresh, failure alerting that reaches a person, and freshness checks visible to the people reading the numbers.
- Migration with history intact. History moved and reconciled against the source before cutover, with a parallel period to prove the numbers match.
- Documentation for handover. What each pipeline does, what it assumes, how to extend it and what to do when it fails.
How the work runs
- Inventory and profile. What exists, what condition it is in, and where the sources disagree. Profiling before promising.
- Agree the rules. Reconciliation decisions made by people who know the business, written down and signed off.
- Build incrementally. One source at a time, reconciled at each step rather than all at once at the end.
- Prove in parallel. Run old and new together until the numbers match. No cutover before that.
- Instrument and hand over. Alerting, freshness checks and documentation your team can operate from.
Proof
The layered design these pipelines land into is described under medallion architecture.
Related
Talk about your situation
The first conversation is short and mostly questions. Get in touch and tell me what you are trying to fix. Or see how this is priced.
Frequently Asked Questions (FAQs)
What does a data engineer do?
Builds the systems that move data from where it is created to where it is analysed, and resolves the disagreements between sources along the way. In practice the reconciliation work, deciding what the data means when two systems conflict, takes more time than the pipeline code.
How long does a data migration take?
It depends on the number of sources and the amount of disagreement between them far more than on data volume. The extraction is usually quick. Reconciling history and proving the new numbers match the old is the part that determines the timeline, and skipping it is what makes migrations fail.
What is the difference between ETL and ELT?
ETL transforms data before loading it into the destination. ELT loads it raw and transforms it afterwards using the destination's compute. ELT suits a lakehouse, because storing raw data is cheap and keeping it untransformed preserves the audit trail. That is the same reasoning behind a bronze layer.
Can you work with our existing warehouse?
Yes. The architecture matters more than the platform, and the layered approach works on most modern warehouses and lakehouses. Recommending a platform change only makes sense when the current one genuinely cannot do the job, which is less often than it is suggested.
How do you handle personal data?
It is classified before it is indexed, and pseudonymised or hashed at the conformance layer wherever the analysis does not need the identity. Downstream users then work with a stable key rather than a name, which keeps most analysis possible without spreading personal data across every table.
What happens when a source system changes?
That is what the bronze layer and the quality checks exist for. Because raw data is preserved untouched, a change in the source can be identified and reprocessed rather than guessed at. The checks that run with every load catch the change quickly, instead of it surfacing weeks later as a number nobody can explain.