Apache Airflow Interview Questions

"A DAG is a directed acyclic graph, and in Airflow it's the workflow itself: a Python file that declares a set of tasks, how they depend on each other, when the whole thing should run, and shared settings like retries. Directed means each edge has a direction, so task B only starts once task A has done its part. Acyclic means you can never follow the arrows and end up back where you started. That rule is what makes scheduling possible. If extract depended on load and load depended on extract, the scheduler would have no task it could start first, and a run could never finish. Airflow checks for cycles when it parses the file and refuses a DAG that has one. If I genuinely need to repeat something, I rerun the DAG on its next schedule or loop inside a single task, not across tasks."

"There are a few moving parts. The DAG files are parsed on a loop, either by the scheduler or, in newer versions, by a separate DAG processor, and the parsed structure goes into the metadata database. The scheduler reads that, creates DAG runs when they're due, and works out which task instances have their dependencies met. It hands those to the executor, which decides how they run: as local processes, through a queue to Celery workers, or as pods on Kubernetes. Workers run the task code and report the state back; in Airflow 3 they do that through an API server rather than writing to the database directly. The metadata database is the heart of it, holding every run, task state, connection and variable. The web UI reads the same database so I can see runs, logs and clear tasks. If I use deferrable operators there's also a triggerer, which waits on many async events cheaply."

"Airflow doesn't read a DAG file once and remember it. It re-parses every file on a loop, every few seconds to a minute depending on settings, to pick up changes. So any code at the top level, outside a task, runs on every single parse. If that code queries a database or calls an API, I'm hitting that system thousands of times a day even when nothing is scheduled, and parsing gets slow. Slow files delay scheduling for every DAG, and if a file takes too long it hits the import timeout, shows up as a broken DAG, and my changes stop being picked up. The same goes for Variable.get at the top level and heavy imports like big ML libraries. My rule is that top-level code only builds the graph. Anything that talks to the outside world goes inside a task, and variables come in through templates so they're read at run time."

"I set dependencies with the bitshift operators, so extract >> transform >> load, and a list on either side fans out or fans in. For long straight lines I use chain. Each task then has a trigger rule that decides when it may start, and the default is all_success: every direct upstream task must have succeeded. I change it in a few cases. A cleanup task that drops temp tables should run whatever happened, so it gets all_done. An alert task that should only fire when something breaks gets one_failed. The common trap is branching. After a branch operator, the path not taken is skipped, and with the default rule the join task would be skipped too. So the join gets none_failed_min_one_success, which means run if nothing failed and at least one branch actually ran."

"TaskFlow lets me write tasks as normal Python functions with a @task decorator, and the DAG itself with @dag. The big difference is how data and dependencies flow. With classic operators, I'd create a PythonOperator, push a value to XCom, pull it in the next task by task id, and set the order separately with >>. With TaskFlow, a function's return value is pushed to XCom automatically, and when I pass that return value into another task function, Airflow pulls it for me and adds the dependency. So the code reads like ordinary Python. It's still XCom underneath, so the same rule applies: return small things like a count or a file path, not a big dataset. And I can mix styles freely, for example a SQL operator followed by a TaskFlow task that checks its result."

"I test at three levels. First, a cheap check in CI that loads every DAG file, the same way the scheduler would, and fails if there's any import error, a cycle, or a DAG with no owner or no retries. That alone catches most broken deploys. Second, I keep the real logic in plain Python functions or modules that don't depend on Airflow, so I can unit test them with normal test data. Third, I actually run things. The tasks test command runs one task for a given date without recording state, and dag.test runs a whole DAG in a single process, which is great for debugging in an IDE. Before production, changes go to a staging Airflow pointed at test connections. I also add policy checks, for example that every DAG sets catchup explicitly, so risky defaults never slip through."

"An operator is a template for one kind of work, like running a SQL statement, a bash command or a Python function. When I use one in a DAG with a task id, that instance becomes a task. A sensor is a kind of operator whose only job is to wait: it checks a condition, like a file landing in storage or a partition existing, and succeeds when it's true or fails after its timeout. A hook is lower level. It's the client that knows how to talk to an external system, like a database or a storage bucket, and it gets its host and credentials from an Airflow connection. Operators use hooks internally, and I use hooks directly inside my own Python tasks when no operator does quite what I need. So the hook talks to the system, the operator does the work, and the sensor waits."

"In poke mode the sensor starts on a worker and stays there, checking, sleeping for the poke interval, and checking again. For six hours that's a worker slot doing nothing. If twenty DAGs do that, my workers are full of sleeping sensors and real tasks queue behind them. Reschedule mode fixes the slot problem: the sensor checks once, and if the file isn't there it ends that attempt, goes into up_for_reschedule and frees the slot until the next check. That's good for long waits with a check every few minutes. A deferrable sensor goes further. It defers itself to the triggerer, which runs lightweight async triggers for many tasks in one process, and the task only comes back to a worker when the event fires. For a six-hour wait I'd use a deferrable sensor if one exists, otherwise reschedule, with a sensible timeout and an alert."

"I subclass BaseOperator and put the logic in execute, which Airflow calls with the run's context when the task runs. The constructor only stores arguments and passes the rest to the parent, so things like retries and task_id still work. For the database, I use the provider's hook with a connection id, so credentials live in the Airflow connection, not in my code. I add table and partition_date to template_fields, which means someone can pass the run date as a Jinja template and Airflow renders it just before execute. Inside, I run a count with a bound parameter. If the count is below the minimum, I raise an exception, which fails the task and triggers retries or alerts. I return the count, which goes to XCom in case a later task wants it. I keep execute small so the logic is easy to unit test."

"Credentials never go in DAG files. Every hook takes a connection id, and the connection holds the host, login, password and extras. Where the connection lives is a deployment choice. It can be in the metadata database, which encrypts the password field with a key, or as an environment variable, but in production I prefer a secrets backend, like a vault or the cloud provider's secret manager, so secrets are rotated and audited in one place and Airflow just looks them up by name. Variables are for non-secret config like a bucket name. The trap is reading them at the top of a DAG file, which hits the database on every parse. I read them inside tasks or through templates like var.value.bucket_name, which only resolve at run time. Airflow also masks values that look sensitive in logs, but I don't rely on that alone."

"XCom, short for cross-communication, is how tasks pass small values to each other. Each task runs in its own process, maybe on a different machine, so they can't share memory. A task pushes a value, for example by returning it, and a later task pulls it by task id and key. By default those values are serialised and stored in the metadata database, attached to that run. That's why I only pass small things: a file path, a list of partition names, a row count, a status. I never pass a DataFrame or a full query result, because it bloats the database that the whole scheduler depends on, and there are size limits that depend on the database. For big data, the task writes it to storage and passes the path. If a team really needs larger XComs, you can set up a custom backend that stores them in object storage."

"Airflow renders Jinja templates just before a task runs, using that run's context. So in a SQL operator I can write WHERE order_date = '{{ ds }}' and each run gets its own date, which is what makes reruns and backfills load the right slice. The key thing is that only fields listed in the operator's template_fields get rendered, plus template files with extensions it recognises, like a .sql file. If I put a Jinja string into a field that isn't templated, it arrives as literal braces. Inside a Python task, I don't use Jinja at all; I read the same values from the context, like data_interval_start. Useful values are ds for the date, data_interval_start and data_interval_end for the exact window, params for user settings, and var for variables. One catch: on Airflow 3 a plain cron schedule gives a zero-length interval by default, so for window queries like this one I make sure the DAG uses the interval timetable."

"It depends on the timetable, and the default changed between versions. In Airflow 2, a cron schedule like daily uses data intervals: each run covers a window and only starts once that window has closed, because that's when a full day of data exists. So the first interval is January 1 midnight to January 2 midnight, the run starts just after midnight on January 2, and its logical date, the ds macro, is January 1. That's why people say Airflow runs a day late, but the run processes the day it's named after. Airflow 3 switched cron schedules to a trigger-style timetable by default. There the first run fires at midnight on January 1, its logical date is that fire time, and the data interval has zero length. So on Airflow 3, a daily job that loads yesterday either asks for the previous day itself or sets the interval timetable explicitly. I always check which one a project uses before reasoning about dates."

"Catchup is a DAG setting. When it's on and the scheduler sees intervals that have no run, say because the DAG was paused for a week or the start date is in the past, it creates a run for every one of those intervals. When it's off, it only schedules the most recent interval and moves on. A backfill is different: it's something I ask for. I pick a date range and Airflow creates runs for it, regardless of the catchup setting. In Airflow 2 that was a CLI command; Airflow 3 also lets you start one from the UI or the API, and the scheduler runs it. I use backfills after fixing a bug or adding a new DAG that needs history. Because catchup is on by default in Airflow 2 and off in Airflow 3, I always set it explicitly, and I keep max_active_runs low so a big catch-up doesn't flood the workers."

"There are three common ways. My first choice is data-aware scheduling. The producer's load task declares that it updates a dataset, called an asset in newer versions, identified by a URI like the orders table, and my DAG uses that dataset as its schedule. When the producer task succeeds, my DAG runs. It's loosely coupled: they don't need to know each other's DAG ids or schedules. The second option is an external task sensor in my DAG that waits for a specific task in theirs. It works, but it's tied to their task id and assumes the logical dates line up, so if schedules differ I have to map the dates, and a renamed task breaks my DAG silently. The third is having their DAG trigger mine directly, which puts the dependency in their code. One caution with datasets: they fire when the task succeeds, not when data actually changed, so I still validate the data."

"A plain INSERT isn't idempotent. If the task fails after writing, the retry inserts the same rows again. If someone clears the task next week, or runs a backfill, it adds another copy. And if the query picks yesterday using the current date, a rerun next week loads the wrong day entirely. I'd fix two things. First, scope the work to the run's own window using ds or the data interval, so any run always touches exactly its own day. Second, make the write replace that day instead of adding to it: delete the day's rows and insert them again inside one transaction, or overwrite the partition in a lake table, or use a MERGE keyed on the order id. Then I can retry, clear or backfill any date and the table ends up exactly the same."

"I set retry behaviour in default_args so every task in the DAG gets it, and override it per task when needed. For a flaky API I'd give it a few retries with a retry delay of a few minutes, turn on exponential backoff so each wait is longer than the last, and cap it with max_retry_delay so it doesn't wait forever. I also set execution_timeout, because a request that hangs is worse than one that fails: it holds a worker slot and never triggers the retry. Retries only help with transient problems like timeouts or rate limits. If the API says my credentials are wrong or the payload is invalid, retrying just delays the alert, so for those I raise an exception that fails the task immediately without retrying. And because the task may run more than once, it has to be safe to repeat."

"They work at different levels. execution_timeout is per task attempt. If one attempt runs longer, Airflow kills it and marks that attempt failed, and normal retries kick in, so it's my protection against a hung query or API call. A sensor's timeout is about the condition, not one attempt: it's the total time the sensor is allowed to wait for, say, a file to arrive. When that's exceeded, the sensor fails and doesn't use its retries, because waiting again wouldn't make the file appear. That's different from execution_timeout on a sensor, which would limit a single attempt. dagrun_timeout sits on the DAG and limits how long a whole run can stay running before it's marked failed, which also frees a slot when max_active_runs is low. I usually set all three: tight per task, realistic for sensors, and a DAG-level ceiling as a safety net."

"Failure alerts are the easy part: on_failure_callback, or a notifier, posts to chat or pages whoever is on call. Lateness is harder, because nothing fails. Inside Airflow 2 there's an sla argument on tasks with an SLA miss callback, but it had quirks and Airflow 3 removed it, with deadline alerts replacing it in later releases, so I check the version before relying on either. The bigger gap is that any alert that lives in Airflow depends on Airflow working. If the scheduler is down, no run starts, and no callback ever fires. So for important pipelines I add a check from outside: a small monitor that looks at the output, for example the newest date in the target table, and alerts if it's older than expected by a set time. I also monitor the scheduler's heartbeat. That way I hear about late data whatever the cause."

"The executor decides how a task instance gets run. The Local executor runs tasks as separate processes on the same machine as the scheduler. It's simple and fine for a small team, but it can't scale past one box. The Celery executor puts tasks on a message queue, usually Redis or RabbitMQ, and a pool of always-on workers picks them up. Tasks start fast, I can add workers, and I can route tasks to named queues, like a queue of big-memory workers. The downside is running the broker and keeping every worker's dependencies in sync. The Kubernetes executor launches a fresh pod for each task. That gives isolation, per-task images and resources, and nothing running when idle, but each task pays pod start-up time. So for many short tasks I lean Celery, and for mixed, heavy or dependency-clashing workloads, Kubernetes. Newer versions also let you run more than one executor together."

"The tool for a shared resource is a pool. I'd create a pool for that database with, say, four slots, agreed with the DBA, and put every task that queries it into that pool, across all five DAGs. Then no matter how many runs are active, at most four of those queries run at once, and the rest wait in the scheduled state until a slot frees up. A task that's especially heavy can take more than one slot with pool_slots. I'd also look at the DAG-level limits: max_active_runs so a backfill doesn't start thirty days in parallel, and max_active_tasks to cap tasks per DAG. If some queries matter more, like the one feeding a morning report, I give them a higher priority_weight so they get slots first. Finally I'd check the queries themselves, because sometimes the real fix is one bad full-table scan."

"First I check which state they're stuck in, because it tells me where to look. Scheduled means the scheduler knows the task is ready but hasn't handed it to the executor, usually because of a limit: the global parallelism setting, a full pool, or the DAG's max_active_tasks. So I check pools in the UI and count running tasks. Queued means it has been sent to the executor but no worker has started it. With Celery, I check that workers are alive, that their concurrency isn't used up by long sensors in poke mode, and that some worker actually listens on the queue the task was sent to, which is a classic mistake. I also check the broker. With Kubernetes, I look for pods stuck pending because the cluster has no room or can't pull the image. Finally I check scheduler health: its heartbeat, CPU and how long DAG parsing takes."

"This is exactly what dynamic task mapping is for. The first task runs at run time, lists the files for that day, and returns a list of paths. Then I call expand on the processing task with that list, and Airflow creates one mapped task instance per path, all in the same run, each with its own logs and retries. So if one file fails, only that instance retries. A final task takes the mapped results and gets them as a list, so it can add up counts or send a summary. I'd put a limit on how many mapped instances run at once so a day with five hundred files doesn't swamp the workers. The nice part compared to older tricks is that the number of tasks isn't fixed when the DAG is parsed; it's decided by the data each run."

"I'd write one generator file reading a YAML or JSON config deployed with the code, loops over the entries, and builds a DAG for each with a factory function, with a dag_id like ingest_ plus the source name. Each DAG has to be registered where the parser finds it, so I build it in a with block or put it in globals(). Then I'd guard against the known risks. The config must be a local file, not a database or API call, because it's read on every parse. I'd validate the config in CI so one typo doesn't turn the whole file into an import error and take all fifty DAGs down together. dag_ids must never change, because a renamed DAG loses its run history and starts again from its start date, possibly with catchup. And removing an entry leaves an old DAG in the UI that someone has to clean up. If the fifty sources really share one schedule, I'd ask whether one DAG with mapped tasks is simpler."

"At my last company we had a nightly DAG that failed maybe twice a week at the load step, and the usual fix was someone clearing the task in the morning. Retries had been bumped up, which only hid it. I pulled the logs from a month of failed runs and noticed they all failed around the same few minutes, and always with a connection error. That lined up with the warehouse's maintenance window, and on busy nights our run slipped into it because an upstream sensor released late. So the cause wasn't the code, it was timing. I moved the heavy step off that window, added a pool so our loads couldn't open too many connections at once, and made the load a partition overwrite so retries were safe. Failures stopped, and I added a dashboard of task durations so drift like that shows up before it breaks."

"In my last role I led moving about a hundred and twenty DAGs from an old self-managed Airflow to a newer version on a managed platform. First I built an inventory by running every DAG file through the new version's parser in CI and listing the import errors and deprecation warnings: old import paths, the execution_date name, a couple of custom plugins. Owners fixed their DAGs against that list. We ran both environments side by side, with the new one reading from staging connections, and compared outputs for a week. For the cutover, the dangerous part was scheduling: a DAG that appears in a new environment with catchup on will try to run its whole history. So every DAG went across paused with catchup set explicitly, and we unpaused team by team while the old copy was paused. We kept the old deployment for two weeks as a rollback. The only incident was one missed connection, which the run check caught."

"At my last company, about a dozen engineers wrote DAGs and the same mistakes kept coming back: no retries, catchup left to the default, API calls at the top of files, and INSERTs that duplicated data on retry. Instead of catching these one by one in review, I wrote a one-page guide and a starter DAG template with sensible default_args, catchup set explicitly, and an idempotent write pattern. Then I added CI checks that loaded every DAG and failed on import errors, missing owners, missing retries and slow parse times. I walked the team through it in a short session and fixed the three worst existing DAGs myself as examples. People followed it mostly because the template was the easiest place to start. Over the next quarter, duplicate-row incidents stopped and review comments were about logic, not the same basics."

"First I look at which task failed and read its log from the last attempt, not the first. Then I sort the cause: a timeout or connection error is probably transient; a schema or key error means the data changed upstream; a code error means a recent deploy. I also check if the source even delivered. Before I rerun anything, I check what the failed task wrote. If the tasks overwrite their partition, I can safely clear the failed task with its downstream tasks and let them run. If not, I clean up the partial data first. If the source is late or broken, I contact whoever owns it and wait rather than load half a day. By about 6, if I'm not confident it'll be fixed, I message the finance owners so they know the numbers are stale. I never mark the task as success to turn the dashboard green. Afterwards, I write up what happened and fix the root cause."

"First I pause the DAG, which stops new task instances starting, and if the warehouse is struggling I'd also check with its admins whether to cancel running queries. Then I assess. Which runs actually completed, and what did they do? If the tasks only overwrite their own day's partition, old runs may have done little harm. If they appended, sent emails or called external systems, I need a list of affected dates to clean up or to warn people about. Next, I deal with the queued runs, by marking them failed or deleting them in bulk from the UI, so they don't all start when I unpause. Then I fix the DAG: catchup off, a realistic start date, and a max_active_runs limit. If some history really is needed, I run a planned backfill later, in small batches. Finally I'd add a CI rule that every DAG sets catchup explicitly."

"I'd start by asking how big the data is and how much it'll grow, because for a few hundred megabytes a Python task is perfectly fine and simpler. But for large files I'd push back. Airflow is best as the orchestrator: it decides when work runs, retries it and tracks it, while the heavy lifting happens somewhere built for it. Workers are shared, so a pandas job that needs lots of memory can get killed mid-run, and it slows down every other team's tasks on the same worker. So I'd suggest either doing the transform in the warehouse with SQL, which Airflow triggers with an operator, or running it in a separate container or Spark job with its own resources, which Airflow launches and waits for. I'd frame it as a team guideline with a rough size threshold, not a blanket ban, so the choice is easy next time."