Databricks Consulting Services

The usual starting position: sales data in a CRM, finance data in an accounting package, web data in an analytics tool, operations data in spreadsheets, and no single place where anyone can ask a question that crosses two of them.

Databricks solves that by putting the data in one governed place and letting analytics, reporting and AI all work from the same tables. The platform is capable. What decides whether a project pays back is the architecture laid on top of it, and specifically whether anyone drew a clear line between raw data, cleaned data and the numbers the business actually reports.

That line is the medallion pattern, and it is where I start every engagement.

What a Databricks engagement actually involves

Very little of it is Databricks-specific configuration. Most of the work is deciding what your data means.

Which system is authoritative when the CRM and the accounting package disagree about a customer. What counts as one customer when the same person appears four times with different spellings. Which date defines a sale: the order, the invoice or the payment. Whether a cancelled order counts.

Those decisions have to be made once and encoded, or every report built afterwards produces a different number and the platform gets blamed for it. Making them explicit is the consulting; the implementation follows from it.

The medallion pattern, and why the layers are not optional

Three layers, each with one job. The discipline is that transformations only ever move forward, so any number can be traced back to the record it came from.

Bronze holds raw ingested data exactly as it arrived, with nothing corrected. It is deliberately ugly. Its purpose is that when a number looks wrong six months from now, the original is still there to check against.

Silver holds cleaned, conformed, de-duplicated data. One row per real entity, types corrected, business rules applied, keys resolved. This is where the meaning decisions above get encoded.

Gold holds the aggregates the business reads: the tables behind dashboards, reports and models, shaped for the question rather than for storage.

Skipping silver is the common shortcut and the expensive one. Reports get built straight off raw data, each one re-implements the cleaning slightly differently, and within a year no two dashboards agree. Rebuilding at that point costs more than doing it properly first.

Governance from day one, not after the audit

Unity Catalog is the reason to make governance an early decision rather than a later retrofit. Permissions at catalog, schema, table and column level, lineage that shows where a number came from, and an audit trail that answers who accessed what.

Setting that up while there are five tables is straightforward. Setting it up once there are two hundred, and every team has already built something against ungoverned data, is a project of its own. Governance also becomes the foundation for anything AI-related later, because a retrieval system can inherit those permissions instead of needing its own.

Keeping the bill sensible

Databricks bills for compute, and the common ways to overspend are predictable: clusters left running, jobs on all-purpose compute instead of job compute, full reloads where an incremental refresh would do, and nobody watching until the invoice arrives.

Every build I do ships with job compute for scheduled work, incremental refresh where the source supports it, sensible auto-termination, and a scheduled cost review. The same discipline as the spend guardrails on the AI work, applied to a different bill.

Starting small enough to prove it

The failure pattern in platform projects is a twelve-month programme that delivers nothing visible until month nine, by which point the sponsor has changed and the budget is under review.

A better shape: pick the two or three sources that answer one genuinely contested business question, take those through bronze, silver and gold, and put the answer in front of the people who argue about it. That is weeks rather than quarters, it proves the definitions hold, and it produces something the business can see.

Everything after that is repetition of a pattern that has already been shown to work, which is a much easier thing to fund than a platform in the abstract.

How to begin without a platform programme

Pick one business question that two teams currently answer differently. Bring the two or three systems that feed it.

Taking that single question through bronze, silver and gold is a matter of weeks, proves whether the definitions hold, and produces an answer the business can see. It is also a far easier thing to fund than a platform in the abstract, and everything afterwards is repetition of a proven pattern.

What the engagement includes

A source and meaning audit. Every system that holds relevant data, what it is authoritative for, and the explicit decisions about entities, dates and definitions that everything downstream depends on.
Medallion architecture. Bronze, silver and gold layers with forward-only transformations, so every reported number traces back to the record it came from.
Unity Catalog governance. Permissions, lineage and audit set up while the estate is small, and structured so AI workloads can inherit them later.
Scheduled jobs and refresh. Job compute, incremental refresh where possible, sensible auto-termination and failure alerting.
Handover documentation. How the pipeline runs, what each table means, how to extend it, written for your team rather than for me.

How the work runs

Inventory the sources. Every system, what it holds, how it can be read, and which one wins when two disagree.
Agree the definitions. Entity, date and inclusion rules written down and signed off before any table is built.
Build bronze and silver. Raw ingestion preserved untouched, then cleaned and conformed with the agreed rules encoded.
Shape gold to the questions. Aggregates built for what the business actually asks, not for how the data happens to be stored.
Govern, schedule and hand over. Unity Catalog, scheduled refresh, cost controls and documentation your team can run from.

Proof

A worked example, from scattered sources through to governed gold tables and a natural-language query layer, is in the awards manufacturer case study.

Talk about your situation

The first conversation is short and mostly questions. Get in touch and tell me what you are trying to fix. Or see how this is priced.

Frequently Asked Questions (FAQs)

What does a Databricks consultant do?

Mostly, decides what your data means and encodes those decisions so every report agrees. That covers auditing the source systems, agreeing entity and date definitions, designing the medallion layers, setting up Unity Catalog governance, building the scheduled pipelines and documenting it for handover. Platform configuration is a small part of the work.

What is medallion architecture?

A three-layer data design. Bronze holds raw data exactly as ingested. Silver holds cleaned, de-duplicated, conformed data with business rules applied. Gold holds the aggregates the business reads. Transformations only move forward, so any reported number can be traced back to its source record.

Do we need Databricks or would a warehouse do?

A conventional warehouse is often enough if your work is structured reporting on structured data. Databricks earns its place when you have a mix of structured and unstructured material, when the same data has to serve both analytics and AI, or when governance across a lot of scattered sources is the actual problem. Buying it for reporting alone is usually overspecified.

How do you control Databricks costs?

Job compute for scheduled work rather than all-purpose clusters, incremental refresh rather than full reloads, aggressive auto-termination, and a scheduled cost review so overspend is caught in days rather than at the invoice. Most overspend comes from clusters nobody turned off.

Can you work with our existing data team?

Yes, and it is the better arrangement. The architecture and definitions are the part worth bringing someone in for; the ongoing pipeline work is usually better owned in-house. The handover documentation is written on that assumption.