An active WIP build of an end-to-end Databricks pipeline on the Bronze / Silver / Gold medallion pattern, using Delta Live Tables and Unity Catalog — a reference project to make my "I know Databricks" claim mean something more than "I've run a notebook."

Why Databricks specifically

Databricks keeps showing up in enterprise data job descriptions, and not in a generic "we use it sometimes" way — in the load-bearing way, as the platform. The reason is that it does three things in one place that historically required stitching together three separate stacks:

1. Spark for distributed compute, with all the maturity and ecosystem that comes with the JVM Spark world (Delta tables, Photon, structured streaming, MLlib).
2. Delta Lake as the storage layer, which gives you ACID transactions, time travel, and schema evolution on top of cheap object storage. This is the non-negotiable foundation for the lakehouse pattern.
3. Delta Live Tables (DLT) for declarative pipelines, which is the abstraction layer that turns "a pile of notebooks chained together by jobs" into "a versioned, monitored, expectation-checked DAG you can read."

On top of that, Unity Catalog for governance and the native Power BI connector at the egress make Databricks one of the rare platforms where the ingest, transform, governance, and BI consumption layers live behind a single identity boundary. That coherence is the actual product, not any individual feature.

I picked Databricks over the alternatives I'd considered:

• Microsoft Fabric. Coherent in the same way, but Microsoft-locked. If your enterprise is committed to Azure, Fabric is plausibly the better choice. For a portable skill set, Databricks runs natively on AWS, Azure, and GCP, and the architectural patterns translate cleanly across clouds.
• Snowflake + dbt + Airflow + Tableau. This is the modular-tool stack, and it's what most companies actually run today. It's also what most companies are getting tired of running, because the integration tax (auth, lineage, monitoring, environment promotion) compounds over time. Databricks's bet is that consolidation wins on TCO — and given how often I see job descriptions explicitly listing Databricks-plus-DLT-plus-Unity-Catalog, that bet seems to be paying off in the enterprise market.
• Pure-cloud-native stacks (BigQuery + Dataform, Synapse + Pipelines, Redshift + Glue). Each of these works inside its respective cloud. None of them have Databricks's storage-format leverage (Delta is open and portable; the others' formats are not), and none of them have DLT's declarative pipeline ergonomics yet.

The blunt version: I want hands-on with the platform that most enterprise data JDs are asking for, on a project that exercises the full medallion flow rather than just the ingestion edge.

The planned project

A retail / sales-style end-to-end pipeline on a public dataset (with synthetic top-ups where the open data is sparse), demonstrating the full Bronze → Silver → Gold flow with DLT orchestration, Unity Catalog governance, and a Power BI dashboard at the consumption end.

Bronze: raw ingestion

The Bronze layer is where the design discipline starts. The principle is "land everything, transform almost nothing." Specifically:

• Source-of-truth fidelity. Every CSV that lands in object storage gets ingested into a Bronze Delta table with the original column names, original types (or string, when in doubt), and zero business logic applied.
• Audit columns added at ingest. Every Bronze row carries _ingest_timestamp (when this row was written into Delta), _source_file (which physical file it came from), and _ingest_run_id (the DLT run that produced it). Without these three, debugging a downstream data quality issue means guessing; with them, you can trace any anomaly back to a specific batch in seconds.
• Schema-on-read. The bronze tables enforce a schema, but loosely — the goal is to land the data and let schema drift surface as a quality issue downstream, not as an ingest failure that loses the row.