Databricks Interview Questions

"Spark is the open-source engine that distributes computation across a cluster. Databricks is the managed platform around it, created by the same people: it provisions and scales clusters, adds the Delta Lake storage layer, governance through Unity Catalog, collaborative notebooks, job orchestration, SQL warehouses and MLflow, and runs on the main cloud providers. So Spark is the engine; Databricks is the car built around it."

"A data lake gives you cheap storage in open formats for any kind of data, but no transactions or reliable schemas. A warehouse gives you reliability and fast SQL but locks the data in and does not suit unstructured data or machine learning. The lakehouse keeps the data in open formats on object storage and adds a transactional table format, a catalog and fast engines on top, so analysts and data scientists work on the same copy."

"A Delta table is a folder of Parquet files with a transaction log next to them. The log is an ordered record of which files were added and removed in each commit, so readers see a consistent snapshot, writers get ACID guarantees, and a failed job cannot leave partial data visible. On top of that you get schema enforcement, time travel to earlier versions, real updates, deletes and merges, and the same table works as a streaming source and sink."

"Bronze holds data exactly as it arrived, with ingestion metadata, so you can always reprocess from it. Silver applies cleaning, deduplication, typing and joins, producing trustworthy entity tables. Gold holds the aggregates and business-level views that dashboards and models read. Each layer is a Delta table, and quality rules get stricter as you move up."

"Transformations are lazy: they only build a plan. When an action runs, the driver optimises that plan and cuts it into stages wherever data has to be shuffled between nodes. Each stage is split into tasks, one per partition, and the executors run those tasks in parallel while the driver schedules them and gathers results. That is why the number of partitions and the number of shuffles decide most of a job's performance."

"A narrow transformation like a filter works within a partition, so it pipelines with no data movement. A wide transformation like a group-by or a join needs rows with the same key on the same node, so Spark shuffles data across the cluster, writing it to disk and sending it over the network. Shuffles are the expensive part of almost every job, so I try to reduce them, broadcast small tables, and watch for skewed keys."

"Unity Catalog is the governance layer: every table, view, volume, function and model lives under catalog dot schema dot name, and permissions, audit logs and lineage are managed centrally across all workspaces. It replaced per-workspace metastores, so a grant is made once. It also gives fine-grained control like row filters and column masks, and lineage that shows exactly which jobs and columns fed a table when something looks wrong."

"Because the log records which files belonged to every version, I can query the table as of a version number or a timestamp and Delta reconstructs that snapshot. VACUUM removes files that the current version no longer references and that are older than the retention window, to reclaim storage. The catch is that those files are exactly what old versions needed, so after a VACUUM, time travel beyond the retention window stops working. I set retention deliberately and never shorten it casually."

"OPTIMIZE rewrites lots of small files into fewer big ones, which matters because streaming and frequent small writes create thousands of tiny files that slow every read. Z-ORDER goes further by sorting related values into the same files so the per-file statistics let the engine skip most of the table on filtered queries. Liquid clustering is the current recommendation: you declare clustering columns, the layout is maintained incrementally, you can change the keys later, and you avoid the small-partition problems of hard partitioning."

"Auto Loader is a streaming source for files landing in cloud storage. It remembers which files it has processed in its checkpoint, so each file is ingested exactly once, and it can discover new files through notifications instead of listing a huge directory every time. It infers the schema, can evolve it when new columns appear, and captures rows that do not fit. The plain file source re-lists everything and breaks on schema changes, so Auto Loader is the default for bronze ingestion."

"Structured Streaming processes data in micro-batches and writes the source offsets and any state to a checkpoint after each batch. If the job restarts it replays from the last committed offsets. Exactly-once comes from combining that replay with an idempotent sink: a Delta table records the batch id in its transaction, so if the same batch is replayed it is ignored. Triggers control cadence: a fixed interval for an always-on stream, or available-now to drain everything pending and stop, which lets me run a streaming pipeline as a cheap scheduled job."

"With declarative pipelines, the product previously called Delta Live Tables and now part of Lakeflow, I define each table as a query and mark it streaming or materialised, and the service builds the dependency graph, runs it, retries failures and manages compute. Expectations let me attach data-quality rules to a table and decide whether bad rows are logged, dropped or fail the run. I use it for the normal bronze-silver-gold shape because it removes a lot of orchestration code; for unusual processing I write regular jobs."

"All-purpose clusters are for people: notebooks, exploration, shared sessions. Production jobs should run on job clusters that spin up for the run and disappear after, which is cheaper and keeps runs isolated. Serverless removes cluster management entirely and starts in seconds, which suits SQL warehouses and many jobs. Photon is the vectorised execution engine that accelerates SQL and DataFrame operations, so I turn it on for SQL-heavy workloads and measure the difference."

"That pattern is skew: a few keys, like a null or a default customer id, hold most of the rows, so one task does most of the work and spills to disk. First I confirm it in the Spark UI by looking at task durations and shuffle sizes for that stage. Then, in order: make sure adaptive query execution is on so skewed partitions get split, broadcast the smaller table if it fits in memory, and if the big-to-big join is still skewed, salt the hot keys so they spread across partitions. I also check whether the null keys should be in the join at all."

"For upserts I use MERGE INTO with the business key: update rows that match, insert the ones that do not, and delete or expire rows that disappeared if the source is a full snapshot. To propagate changes downstream I enable the change data feed on the table, which lets the next stage read only the inserts, updates and deletes since the last version it processed. For history I keep type-two dimensions with start and end dates, and in declarative pipelines the apply-changes step does that for me."

"I cache a DataFrame only when I will reuse that exact result more than once in the job, for example a cleaned table that feeds three aggregations; it is materialised on the next action and lives in executor memory or spills to disk. Separately, the platform's disk cache keeps copies of remote files on the node's local SSD automatically, which speeds repeated reads with no code. And I unpersist when I am done, because an unused cache is just memory taken from shuffles."

"A SQL warehouse is compute purpose-built for SQL: analysts and BI tools connect to it, it runs Photon, caches results, and the serverless option starts in seconds and scales with concurrency. I point dashboards and ad-hoc SQL at a warehouse and keep clusters for engineering and ML work in notebooks and jobs, so BI traffic never competes with a pipeline."

"I build a job as a graph of tasks with explicit dependencies, so the silver step runs only after bronze succeeds, with retries on transient failures and alerts on failure. Parameters like the run date and the target environment are defined on the job and read inside each task, and tasks can hand values to downstream tasks. Jobs run under a service principal with only the grants they need, and they use job clusters or serverless rather than a shared cluster."

"MLflow is built in. Training runs log parameters, metrics and artefacts to an experiment, mostly automatically through autologging. When a model is good I register it in Unity Catalog, where it gets versions and aliases like champion and challenger, and the same permissions model as tables. From there it can be deployed to a serving endpoint, and lineage shows which tables trained it, which is what makes audits possible."

"I start with the billing system tables to see which jobs, clusters and users the spend comes from, because it is usually a few things. Then the common culprits: interactive clusters left running without auto-termination, production jobs on shared clusters instead of job clusters, and clusters sized for peak all day. After that I look inside the top jobs for wasted work, like full reloads that should be incremental, small-file bloat and skewed shuffles. Fixing habits usually saves more than tuning code."