This page is for data engineers preparing for a first role or a move up. Data engineering interviews usually mix a short screening chat, questions on core ideas like ETL versus ELT, warehouses and lakes, schemas and file formats, a SQL or pipeline design exercise, and stories about pipelines you have built, broken and fixed. Interviewers want to see that you think about correctness, reruns, late data and cost, not just tool names. Each question shows what the interviewer is really checking, a shape for your answer and a sample you could say out loud. Swap in your own stories, and practice the design case out loud before the day.
Search all questions by round, difficulty and level, or save the ones you want to practise.
Start: how you came to the work, in one or two sentences.
What you own: sources, where the data lands, who uses it and how often it runs.
Next: what you want to build more of, and why this role fits.
"I started as a reporting analyst writing SQL against a production database, and I kept hitting the same wall: the data was late, duplicated or shaped wrong for the question. So I started building the fixes myself, first scheduled scripts, then proper Airflow jobs loading into a warehouse. In my current role I own about a dozen batch pipelines. They pull from our app database, a payment provider's API and event logs, and land them as modeled tables the analysts use every morning. The part I enjoy most is making data trustworthy, so people stop double-checking numbers in spreadsheets. Now I'm looking for a team where I can take on streaming work and more of the design side, not just keep jobs running."
Reciting a list of tools with no mention of what data moved, who used it or what went wrong.
Their data: a sensible guess at where their data comes from.
The work: ingestion, modeling, support and on-call, in your own words.
Why you: what about that mix you actually like, plus one question back.
"From the job post and what I could find about your product, I'd guess most of your data comes from the app itself, so events and the main transactional database, plus a few third-party tools like billing and support. So day to day I'd expect to build and maintain ingestion from those sources, model it in the warehouse, and be the person who gets pinged when a dashboard looks wrong. That appeals to me because it's close to the business. A good pipeline here decides whether the product team trusts their numbers. I'd also like to ask how much of the week goes to new work versus keeping existing jobs healthy, because that tells me where I'd add the most."
Describing only exciting new builds and showing no awareness that support and maintenance are a large part of the job.
ETL: transform on a separate layer, load only the clean result.
ELT: load raw first, transform inside the warehouse or lake.
Why ELT: scalable warehouse compute and the raw data kept for rebuilds.
When ETL: sensitive fields that must be masked before landing, or a weak target.
"In ETL you extract data from the source, transform it on a separate processing layer, and only load the cleaned result into the warehouse. In ELT you load the raw data into the warehouse or lake first, then transform it there, usually with SQL. A lot of teams moved to ELT because cloud warehouses can scale compute on demand, so doing the heavy work inside them is simpler than running a separate transform cluster. It also keeps the raw data, so if a business rule changes you can rebuild the tables without pulling from the source again. ETL still makes sense when sensitive fields must be masked or dropped before they land anywhere, or when the target system can't handle heavy transforms."
Saying ELT is simply newer and therefore better, with no trade-off on either side.
Requirements: volume, freshness per consumer, who reads what.
Ingest: collection endpoint to a queue, raw landing kept for replay.
Transform: hourly dedupe, late-data lookback, sessions and fact tables.
Operate: schema checks, freshness and volume alerts, a contract with app teams.
"I'd start with the requirements: event volume, how fresh each view needs to be, and who reads it. For the design, the apps send events to a collection endpoint that writes them to a message queue like Kafka, which absorbs spikes and lets several consumers read. A consumer lands the raw events untouched in the lake as Parquet, partitioned by date and hour, so I can always replay. Every hour an orchestrated job cleans the data, deduplicates on event ID, reprocesses a lookback window for late events, and builds sessions and a fact table. The hourly dashboard and the daily report read aggregate tables built from that. I'd add schema checks at the collection point, freshness and volume alerts, and an agreed event contract with the app teams. If they later need second-level freshness, a streaming job can read the same topic without changing the rest."
Jumping straight to a list of tools without asking about volume or freshness, and keeping no raw copy to replay from.
Scope: what moved and who depended on it.
Find consumers: from query logs and lineage, not memory.
Parallel run: both versions side by side with automated comparison.
Cutover: in groups, with a fallback and clear updates.
"We moved our core orders tables from an old on-premise database with nightly scripts to a cloud warehouse with orchestrated jobs. About forty dashboards and a few finance exports read those tables. I started by listing every consumer from the query logs rather than asking around, because people forget what they use. I built the new pipeline to produce tables with the same names and columns, and ran both systems in parallel for three weeks with an automated daily comparison of row counts and key totals. Most differences turned out to be bugs in the old logic, and I agreed with finance which version was correct before switching. We moved consumers in groups, kept the old tables read-only for a month as a fallback, and sent a short update each week. Nobody lost a report on cutover day."
Switching everything over on one day with no parallel run, no comparison and no way back.
Freshness need: what decision gets better with fresher data.
Cost of streaming: late events, state, delivery guarantees, always-on systems.
Middle ground: micro-batch every few minutes covers many requests.
"I start from how fresh the data really needs to be and what someone does with it. If a report is read once each morning, a nightly or hourly batch is cheaper, easier to test and easier to rerun. Streaming earns its cost when a late answer loses its value, like fraud checks, live inventory or alerting. It brings real extra work: out-of-order and late events, state, delivery guarantees, and a system that has to stay up all the time. Often the honest answer is micro-batch every few minutes, which covers most requests for a real-time dashboard. So I ask the stakeholder what decision they'd make faster with fresher data, and I pick the simplest design that meets that."
Choosing streaming because it is modern, without asking what the freshness is actually used for.
What it is: capture inserts, updates and deletes so copies stay in sync.
Query-based: simple, but misses hard deletes and relies on every writer.
Log-based: reads the database's change log in commit order with little load.
Cost: more moving parts, permissions and log retention on the source.
"CDC means capturing inserts, updates and deletes from a source database as they happen, so downstream copies stay in sync without full reloads. The simple way is query-based: every few minutes, select rows where updated_at is later than the last run. It works, but it misses hard deletes, depends on every writer setting updated_at correctly, can miss rows around the boundary, and puts query load on the source. Log-based CDC reads the database's own change log, like the MySQL binlog or the Postgres write-ahead log, often with a tool such as Debezium streaming into Kafka. You get every change in commit order, including deletes, with very little extra load. The costs are more moving parts, plus the right permissions and enough log retention on the source."
Not realizing that polling an updated_at column never sees rows that were deleted.
Two clocks: event time versus processing time, report on event time.
Batch: reprocess a lookback window sized from measured lateness.
Streaming: watermarks and allowed lateness, a plan for anything later.
Users: recent days can still change, so say so.
"First I separate event time, when it happened, from processing time, when we received it, and I report on event time. In a batch pipeline I don't just load yesterday. Each run reprocesses a lookback window, say the last three days of event dates, and rewrites those partitions, so late events land in the right day. I size that window from how late data actually arrives, which I measure. In streaming, engines use a watermark, a running estimate of how far event time has progressed. A window's result is emitted when the watermark passes its end, and it can still be updated during an allowed lateness. Events later than that are dropped, sent to a side output, or picked up by a later batch correction. Either way, I tell users that recent days can still change, and I sometimes mark them as provisional."
Grouping by the load date so late events are counted on the wrong day, with no way to correct them.
Cause: output written before the read position was committed, then replayed.
Short term: deduplicate the affected hour on the order or event ID.
Long term: idempotent sink, an upsert or merge keyed on a unique ID.
Guard: a uniqueness check on the output.
"This is the classic at-least-once gap. If the job wrote its output and then crashed before committing its read position, on restart it reads those messages again and writes them twice. So I'd check how the job commits offsets relative to its writes, and whether the sink is append-only. For the immediate fix, I'd deduplicate the affected hour on the order ID and correct the report. To stop it for good, I'd make the sink idempotent, writing with an upsert or merge keyed on a unique event ID, so replaying the same message changes nothing. Where the engine and sink support it, transactional or exactly-once delivery can help too, but I prefer idempotent writes because they also protect against duplicates the producer sends. And I'd add a uniqueness check on the output so we catch it next time."
Blaming the message queue and suggesting it be replaced, without understanding how offsets and writes interact.
Warehouse: structured tables, schema enforced on write, fast SQL.
Lake: cheap object storage for any file, schema applied on read.
Lakehouse: a table format on lake files adds transactions and schema control.
Choice: match it to the team's workloads and skills.
"A warehouse stores structured, modeled data in tables with a schema enforced on write, and it's built for fast SQL analytics. A data lake is cheap object storage holding files of any shape, like raw logs, JSON or images, with the schema applied when you read it. Lakes are flexible and cheap, but without discipline they turn into a swamp nobody trusts. A lakehouse adds an open table format like Delta Lake, Apache Iceberg or Apache Hudi on top of lake files. That brings transactions, schema enforcement and time travel, so you get warehouse-like tables on cheap storage that several engines can read. I'd pick a warehouse for a mostly SQL, dashboard-heavy team, a lake for raw landing and data science, and a lakehouse when one copy of the data needs to serve both."
Treating the three as marketing names for the same thing, or claiming a lake needs no schema at all.
Parquet: columnar, compresses well, stores stats so readers skip data.
Avro: row-based with its schema, good for writing records and schema evolution.
CSV: no types or schema, fine only for first landing or exchange.
Partitioning: a column most queries filter on, with few distinct values.
"Parquet is columnar, so a query that needs three columns out of fifty reads only those three, and similar values sit together, so they compress very well. It also keeps min and max statistics per chunk, which lets engines skip data that can't match a filter. Avro is row-based, carries its schema with the data and handles schema changes well, so it suits writing records one at a time, like messages on a queue or raw landing files. CSV has no types or schema, so I avoid it past the first landing. For partitioning, I pick a column most queries filter on that has a fairly small number of values, usually the event date. Partitioning on something like user ID creates huge numbers of tiny files, which hurts more than it helps."
Partitioning on a high-cardinality ID column, or saying Parquet is faster without knowing why.
Star: one fact table with keys to denormalized dimensions around it.
Snowflake: dimensions split further into normalized sub-tables.
Trade-off: fewer joins and simpler queries versus less repetition.
"A star schema has one fact table in the middle holding measurable events, like order lines with quantity and amount, plus foreign keys to dimension tables around it, like customer, product and date. Each dimension is denormalized, so the product table has its category and brand right in it. A snowflake schema normalizes those dimensions further, so product points to a separate category table, which might point to a department table. Snowflaking cuts repetition and keeps one place for each attribute, but it adds joins. For analytics I usually prefer a star, because queries are simpler and reporting tools handle it well, and dimension storage is rarely the bottleneck. I'd only snowflake a dimension that's very large or shared by several others."
Confusing the snowflake schema with the product of the same name, or not knowing what goes in a fact table versus a dimension.
Type 1 vs 2: overwrite and lose history, or add a new row per change.
Columns: surrogate key, natural key, valid-from, valid-to, current flag.
Load: close the changed current row, insert the new version.
Facts: store the surrogate key that was current at event time.
"Type 1 simply overwrites the old value, so you lose history. Type 2 keeps it: instead of updating the row, I close the current one and insert a new one. Each row has a surrogate key, the natural customer ID, the attributes, a valid-from and valid-to date and an is-current flag. When the address changes, the load marks the old row as no longer current and sets its end date, then inserts the new address as the current row. Facts store the surrogate key that was current when the event happened, so an old order still reports against the old address. The part people miss is doing the change check cheaply, comparing a hash of the tracked columns, and making sure a rerun doesn't create a second copy of the same version."
-- 1. Close current rows whose tracked columns changed
UPDATE dim_customer d
SET valid_to = CURRENT_DATE, is_current = FALSE
FROM stg_customer s
WHERE d.customer_id = s.customer_id
AND d.is_current
AND d.row_hash <> s.row_hash;
-- 2. Insert a current row for new or just-closed customers
INSERT INTO dim_customer (customer_id, address, row_hash, valid_from, valid_to, is_current)
SELECT s.customer_id, s.address, s.row_hash, CURRENT_DATE, NULL, TRUE
FROM stg_customer s
LEFT JOIN dim_customer d
ON d.customer_id = s.customer_id AND d.is_current
WHERE d.customer_id IS NULL;Updating the address in place and calling it history, or joining facts to the dimension on the natural key so old orders show the new address.
DAG: the tasks and their order, with a schedule and a start date.
Wait: a sensor in reschedule mode with a timeout and an alert.
Retries: a few retries with a delay on each task for brief failures.
Catchup: switch it on only if every task works on its own logical date.
"A DAG is the set of tasks and the order they run in, plus a schedule. For a daily load I'd have three tasks: a sensor that waits for the day's file, a load into a staging table, and the transforms, chained so each one starts only when the one before succeeds. I'd run the sensor in reschedule mode with a timeout, so it doesn't hold a worker slot for hours, and alert if the file never comes. Each task gets a couple of retries with a delay, because most failures are brief network or source hiccups. Every task works on the run's logical date, never today's date, so a retry or rerun rewrites the same day. Catchup decides whether Airflow creates runs for past intervals it missed. I only switch it on when the tasks are idempotent, and otherwise I backfill on purpose."
Putting the whole pipeline in one giant task, or leaving a sensor holding a worker slot all night with no timeout.
Definition: one run or five runs for the same interval give the same result.
Fixed slice: each run works on its logical date, never on now.
Replace, don't append: overwrite the partition or merge on a key.
Small tasks: a retry repeats a little work, with no side effects inside.
"Idempotent means running the task once or five times for the same interval leaves the data in the same state. That matters because Airflow retries failed tasks, and people rerun and backfill old dates. The main habits are these. Each run works on a fixed slice defined by its logical date, not on the current time. It writes by replacing that slice rather than appending, so a daily load deletes and rewrites that day's partition, or merges on a key, inside one transaction where the warehouse allows it. I keep side effects like sending emails out of the load task, and I keep tasks small so a retry repeats a little work, not the whole pipeline. Then retries with a delay become a safe default instead of a risk."
-- Templated per run: {{ ds }} is the run's logical date
BEGIN;
DELETE FROM fact_orders WHERE order_date = '{{ ds }}';
INSERT INTO fact_orders
SELECT * FROM stg_orders WHERE order_date = '{{ ds }}';
COMMIT;Using plain inserts that append on every run, so each retry quietly duplicates a day of data.
Contain: stop downstream tasks from publishing a partial load.
Diagnose: read the logs, transient or real bug.
Recover: rerun only the failed idempotent tasks, or hold the last good data.
Tell and follow up: warn users early, then close the gap.
"First I'd stop anything downstream from publishing half-loaded data, so I'd pause the dependent tasks or make sure dashboards keep showing the last good state rather than a mix. Then I'd read the logs to find the cause, such as a source timeout, a schema change or a resource limit. If it's transient and the tasks are idempotent, I clear the failed tasks and rerun just those, because a rerun rewrites the same partitions cleanly. If the cause needs a code fix I can't safely make in three hours, I'd leave yesterday's data in place and tell the business before they open that today's numbers will be late, and when to expect them. Afterwards I'd write up what happened and close the gap, for example by making the publish step atomic so a partial load can never be seen."
Rerunning the whole pipeline blindly with appending loads, or saying nothing to the business and hoping it finishes in time.
Size it: affected tables, downstream models, data volume, users.
Warn first: tell stakeholders before historical numbers change.
Run safely: test on a sample, then chunks with limited parallelism.
Swap: build a new version and switch once checked.
"First I'd size it: which tables and downstream models are affected, how much data, and who uses those numbers, because correcting two years of history will change reports people have already shared. I'd tell stakeholders before the numbers move and agree on timing. On the technical side, I'd fix the logic, test it on a sample month, and compare old and new outputs so I can explain the difference. Then I'd run the backfill in chunks, say a month at a time, with limited parallelism so it doesn't starve the daily jobs or blow the compute budget. Ideally I'd write to a new version of the table and swap it in once it's fully checked, rather than overwriting in place. Because the tasks are idempotent and partitioned by date, any failed chunk can simply be rerun."
Kicking off one giant rerun over all history with no testing, no chunking and no warning to anyone.
Key: decide what makes two rows the same event, ideally an event ID.
Rank: ROW_NUMBER over that key, ordered by the column that picks the winner.
Keep: filter to rank one, with a deterministic tie-break.
"I'd first confirm what identifies an event. Ideally the producer sends a unique event ID, and retries reuse it. DISTINCT won't work here because retried rows often differ in something like the ingestion timestamp, so they aren't identical. Instead I use ROW_NUMBER, partitioned by event ID and ordered by ingestion time descending, and keep only row number one. If two copies could share the same ingestion time, I'd add a second column to the ORDER BY so the result is the same on every run. Some warehouses let you write this more briefly with QUALIFY, but the subquery works almost everywhere. In a real pipeline I'd run this per partition on each load rather than over the whole history."
SELECT *
FROM (
SELECT e.*,
ROW_NUMBER() OVER (
PARTITION BY event_id
ORDER BY ingested_at DESC
) AS rn
FROM raw_events e
) ranked
WHERE rn = 1;Using SELECT DISTINCT and assuming it removes retried copies that differ in a timestamp column.
Ingestion: arrival, row counts in range, schema matches.
Transform: unique and not-null keys, valid references, ranges, reconciliation.
Freshness: alert when a table stops updating.
Action: decide which failures block publishing and which only warn.
"I think in layers. At ingestion I check the basics: did the batch arrive, is the row count in the expected range, does the schema match. After transformation I test the rules the business relies on: primary keys are unique and not null, foreign keys find a match in their dimension, values fall in valid ranges, and totals reconcile with the source, like revenue matching the payment system within an agreed tolerance. Then I watch freshness, so someone knows when a table stops updating. The important part is deciding what each failure does. A broken primary key should stop the pipeline before bad data is published, while a small drift in volume might only warn. Tools like dbt tests or Great Expectations help, but choosing the right checks matters more than the tool."
Listing checks with no view on which ones should block a load, or relying on users to report bad numbers.
What went wrong: the bug and who saw the bad numbers.
How you found it: the check or comparison that pinned it down.
Repair: fix, backfill and tell the users which dates were wrong.
Prevention: the test or process you added.
"At my last company I added a promotions dimension to our sales model. A week later someone in merchandising noticed that revenue for a handful of products had roughly doubled overnight. I compared daily totals in the model with the order system and saw the gap start on my release date. The cause was that some products had two active promotions at once, so my join matched each order line twice. I fixed the join to pick one promotion per line using a clear rule we agreed with merchandising, reran the affected days, and sent a short note to dashboard users saying which dates were wrong and that they were now corrected. Afterwards I added a test that the fact table's row count and revenue don't change when a dimension is joined, and a daily reconciliation against the order system."
Blaming the source team or the business users, or fixing the bug quietly without telling anyone who used the bad data.
Clarify: which figure, which dates, which filters, how big the gap.
Definitions: gross versus net, refunds, time zone of the day cut.
Reconcile: count and sum at each stage until the numbers split.
Close: fix, backfill, add a reconciliation check, document the definition.
"First I'd get specifics: which number they're comparing, for which dates and filters, and how big the gap is. Many mismatches are about definitions, not bugs, like gross versus net of refunds, or the day being cut in a different time zone. If the definitions agree, I reconcile step by step down the pipeline: count and sum at the source extract, at the raw landing, after each transform, and in the final table. The step where the numbers split tells me where the problem is, maybe a filter, a join dropping rows, or duplicates. I'd keep finance updated on what I've ruled out. Once it's fixed, I'd backfill and add a daily reconciliation check so the next gap alerts us before they notice. If it was a definition difference, I'd write the agreed definition into the model's documentation."
Assuming the warehouse is right because the pipeline didn't fail, or jumping into code before asking what is being compared.
Measure: read the plan, find bytes scanned and the slow step.
Scan less: partition and cluster filters, only needed columns, filter before joins.
Fix joins: duplicate keys that multiply rows, mismatched types.
Reuse: pre-aggregate or materialize what many people query.
"I'd read the query plan first to see where the time goes, usually how much data it scans, whether a join is multiplying rows, or how much data is being shuffled between nodes. The biggest win is normally scanning less. I make sure the query filters on the partition or clustering column so the engine can prune, select only the columns it needs instead of everything, and filter before joining. Then I check the joins. Duplicate keys on the dimension side can multiply rows, and joining on mismatched types can block optimizations. If many dashboards run the same heavy aggregation, I'd build a pre-aggregated table or materialized view that refreshes incrementally, so the big scan happens once a day instead of on every page load. Only then would I look at resizing compute."
Answering only with a bigger warehouse or more nodes, without looking at what the query scans.
Problem: the job, and why its speed or cost mattered to someone.
Measure: where the time or money actually went.
Change: the specific fix, such as incremental loads or file layout.
Proof: how you showed the output stayed the same.
"We had a nightly job that rebuilt a large events table from scratch, and it had crept past four hours, so analysts sometimes started the day on stale data. When I measured where the time went, most of it was reprocessing two years of history that never changed. I switched it to an incremental load that only processed the last few days, with a lookback for late events, and kept a monthly full rebuild as a safety net. I also rewrote the output from thousands of small files into fewer, larger Parquet files partitioned by date. The run dropped to about twenty minutes and the compute cost for that job fell sharply. Before switching over, I ran the old and new versions side by side for a week and compared totals, so nobody had to take the new numbers on faith."
Describing a speed-up with no measurement before or after, or no check that the results stayed correct.
Growth: what changed and how fast.
First break: the bottleneck, found by measuring.
Redesign: parallel, batched ingestion and incremental transforms.
Early warning: trend alerts on volume and run time.
"At my last company a product launch took our event volume from a few million a day to over fifty million within a couple of months. The first thing that broke was ingestion: a single process reading from an API and writing rows to the warehouse one at a time. Then the daily transform started running past its window. I split ingestion so parallel workers wrote batches of Parquet files straight to object storage, partitioned by hour, and loaded them with the warehouse's bulk load instead of row inserts. For the transforms, I made the heavy models incremental and ran deduplication per hour partition instead of over the whole table. I also added alerts on volume and run-time trends, so next time we'd see the curve weeks before it hurt anyone."
Solving growth only by buying bigger machines, with no idea which step was the real bottleneck.
Find: compare query and compute history with the month before.
Usual causes: frequent dashboard refreshes, accidental full refreshes, idle compute, huge scans.
Fix: the top few items first.
Keep it down: spend alerts, a weekly top-cost report, tags by team.
"I'd start with data, not guesses. Most warehouses expose query history with the compute or bytes used per query, user and cluster, so I'd group last month against the month before and find the biggest movers. It's usually a few things: a new dashboard refreshing every few minutes against a huge table, a job that switched to a full refresh by accident, compute left running when idle, or exploratory queries scanning everything. Then I fix the top items first, like making that job incremental again, pre-aggregating the dashboard's data, setting compute to shut down when idle, or adding partition filters. To stop it creeping back, I'd set spend alerts, send a weekly report of the most expensive queries, and tag jobs by team so every cost has an owner."
Cutting compute size across the board without finding out what drove the increase.
The ask: what they wanted and why it seemed reasonable to them.
Real need: the question that uncovered what they actually needed.
Offer: the simpler option and its trade-offs, in their terms.
Result: how it landed and what you learned.
"A sales leader asked me for a real-time feed behind their forecast dashboard. Before saying yes or no, I asked what they'd do differently with data from one minute ago compared with one hour ago. It turned out the team reviewed the forecast twice a day and mainly needed it to be right before their morning and afternoon calls. The source was a CRM with a strict API rate limit, so real-time would have been fragile and costly. I proposed an hourly sync, a refresh button that triggered the job on demand, and a last-updated time shown on the dashboard. They were happy because it met the actual need, and we avoided building a streaming system to support a twice-a-day meeting. What I took from it is to ask about the decision, not the refresh rate."
Either building exactly what was asked without questions, or refusing flatly without offering an alternative.
Pattern: the repeated breakage and its cost to your users.
Conversation: how you raised it with evidence, not complaints.
Agreement: a data contract, checks on both sides, notice for breaking changes.
Outcome: what changed afterwards.
"Our product team shipped app releases every week, and every few releases an event got renamed or a field changed type. Our pipeline would either fail or, worse, quietly load nulls, and fixing each break was eating a day a week. Rather than complain in a channel, I asked to join their planning meeting and showed three recent incidents and what each had cost the analysts. We agreed on a simple data contract: a file in their repository listing each event, its fields and types. A check in their build failed if a change didn't update the contract, and a matching schema check at our ingestion quarantined events that didn't match. We also agreed breaking changes needed a week's notice. Breaks dropped to almost none, and the relationship improved because they could see the impact of their changes."
Blaming the other team throughout, or only ever patching the pipeline after each break.
Decline the edit: it gets overwritten and can't be traced.
Find the worry: is the data actually wrong, or just unwelcome.
Real fix: correct the logic and rerun if there's a bug.
Deadline: show it with a clear caveat rather than altered numbers.
"I wouldn't hand-edit production data. The next pipeline run would overwrite it anyway, and worse, the chart would show numbers nobody can trace back to a source. But I'd take the worry behind the request seriously. I'd ask what looks wrong, because if the data really is broken, that's a bug I want to fix properly and fast. In that case I'd correct the logic, rerun the affected dates and tell them what changed. If the data is right and the story is just uncomfortable, I'd help them explain it, maybe with a note on the chart or a breakdown that shows the cause. And if there's no time for a proper fix before the meeting, I'd suggest showing the chart with a clear caveat rather than quietly altered numbers."
Making the edit to keep the peace, or refusing coldly without trying to understand what was wrong.
Assume a stranger: someone new gets paged at night.
Make it clear: version control, tests, a README and a short runbook.
Make it safe: helpful alerts and idempotent reruns.
Prove it: hand over and let a teammate fix one failure.
"I assume the next person knows nothing about it and gets paged at night. So the code lives in version control with tests, and each pipeline has a short README saying what it does, where the data comes from, who uses it and what to do about common failures. Alerts say which table and which run failed, not just that a task failed. I keep tasks idempotent so the first fix for most problems is simply a rerun. Tables and columns have descriptions in the catalog or model files, so nobody has to message me to learn what a field means. Before I go on leave, I walk a teammate through what I own and ask them to handle one real failure while I'm still around. If they can fix it from the docs, the docs are good enough."
Treating being the only person who understands a pipeline as job security.
Stay current: release notes and small side experiments.
The test: what problem it solves today, and what it costs to run and learn.
Try it small: one real pipeline, compared with current tools.
Adopt cleanly: migrate and retire the old way.
"I keep up by reading release notes for the tools we already use and trying new ones on small side projects, but I'm careful about bringing tools into the team. My test is simple: what problem do we have today that this solves, and what does it cost to run and to learn? Every new tool is another thing to monitor, upgrade, secure and hire for. So I'd write a short proposal naming the problem, try it on one real pipeline for a few weeks, and compare it with doing the same job using what we already have. If it clearly wins, we adopt it with a plan to migrate and retire the old approach, so we don't end up with three tools doing one job. Often the better answer is using our current tools more fully."
Wanting to adopt whatever is newest, or refusing to consider anything new at all.
ClapAssist is an AI interview assistant for Mac and Windows. It listens to the interview on your computer and shows you what to say, in short lines you can read while you talk. Your resume and notes are never stored on our servers. It stays out of screen share on every plan; only you can see it.