Site Reliability Engineer Interview Questions

"I started as a backend developer, and the part of the job I kept volunteering for was the stuff that happened after the code shipped. When something broke at night I was the one who wanted to know exactly why, and I'd end up writing the fix plus a script so it couldn't happen the same way again. Over time my team started sending me the scaling and alerting work, and I realised I liked it more than new features. What pulls me to SRE is that it's still engineering, I'm writing code and designing systems, but the customer is the whole production estate. I want a role where that's the main job, not the side job."

"A traditional operations team often runs systems that developers hand over, and the main way it copes with growth is hiring more people. SRE treats operations as a software problem. If I'm doing the same manual task every week, the expected answer is to automate it, not to get faster at it. The other big difference is how reliability gets decided. Instead of ops saying no to changes and devs pushing to ship, both sides agree an SLO and an error budget, so the numbers settle the argument. Many SRE teams also put a cap on how much of their time goes to tickets and pages, often around half, so there's always room for engineering work that makes the next month easier."

"Your job description talks about moving from one big service to many smaller ones, and that's exactly the stage where reliability gets harder and more interesting. I've been through one of those moves, so I know where things usually crack: alerting, dependencies between teams, and nobody quite owning the whole request path. Before joining I'd want to know a few things. How many pages does an on-call shift get in a normal week? Do product teams own SLOs, or does SRE set them alone? And when a postmortem produces action items, how often do they actually get done? The answers tell me whether I'd be building things or just firefighting."

"An SLI is the thing I measure, like the fraction of requests that succeed, or the fraction served in under 300 milliseconds. An SLO is the target I set for that SLI over a window, say 99.9 percent of requests succeed over 30 days. The engineering team lives by the SLO; it drives alerting and decisions about risk. An SLA is a contract with customers that says what happens, usually credits or refunds, if we miss a level. Sales and legal care most about that one. The key point is that the SLA should be looser than the SLO. If my internal target is 99.9, I might promise customers 99.5, so I get warned and can react long before I'm paying penalties."

"An error budget is one minus the SLO. If the SLO is 99.9 percent over 30 days, the budget is 0.1 percent. Thirty days is 43,200 minutes, so that's about 43 minutes of full downtime, or in request terms one failed request in every thousand. If we serve ten million requests a month, around ten thousand can fail before we've missed the target. The budget is useful because it turns reliability into something we spend. If there's budget left, teams can ship faster and take risks. If it's gone, the error budget policy kicks in, usually a pause on risky launches and engineering time moved onto reliability fixes until we're back inside the SLO. That policy has to be agreed with product ahead of time, or it becomes an argument in the moment."

"I'd start with what the customer cares about: they press pay, it works, and it doesn't take forever. So I'd pick availability, the share of valid checkout requests that don't return a server error, and latency, the share completed under a threshold like 500 milliseconds. I'd leave most 4xx responses out of the valid count, because a bad card number isn't our failure, but I'd count our own throttling as bad. For checkout I'd also want a correctness signal, like orders that actually reach the order store. Where I measure matters. The app's own metrics miss requests that never reach it, so I'd take the numbers from the load balancer, and add a synthetic probe that runs a real test checkout every minute from outside."

"Burn rate is how fast I'm spending the error budget compared with a pace that would use it exactly by the end of the window. A burn rate of one means I'll land right on the SLO after 30 days. A fixed threshold like 'page at one percent errors' ignores that, so it's either too twitchy for short blips or too slow for a steady leak. With burn rates, I might page when the one-hour burn rate is over 14.4, because that spends two percent of a 30-day budget in a single hour, page again for a steadier burn of 6 over six hours, and just open a ticket for a slow leak over about three days. I pair each long window with a short one, say five minutes, so the alert only fires if it's still happening now and resets fast once it's fixed."

"A team wanted to launch a new search feature before a big sales weekend. Their load test only covered half the traffic we expected, and the feature added a heavy query to a database we already knew was near its limit. I didn't want to be the person who just says no, so I brought the numbers to the product lead: expected peak, what the test showed, and what would happen to checkout if the database slowed down. Then I offered options. We could launch to a small share of users behind a flag, add a cache in front of the query first, or launch the week after. They picked the flag and the cache. It went out a few days late but held up fine, and that team now asks us for a review before big launches."

"First I'd go back to the error budget policy we agreed with product, because the point of having one is that this conversation isn't a fresh argument. Usually it says risky launches pause until we're back inside the SLO, with reliability work taking priority. Then I'd look at what burned the budget. If it was one bug or one dependency, fixing that might be quick, and it's the fastest way to earn the launch back. I'd also look at how risky the feature really is. If it's isolated, maybe it ships behind a flag to a small group with a quick kill switch. If the product lead still wants a full launch, that's a business call above both of us, so I'd take it to whoever the policy names, with the data, and respect the decision."

"A page should mean users are hurting, or about to be, and a human needs to act now. Symptoms like error rate and latency against the SLO capture that directly. Causes don't. There are hundreds of possible causes, most of them don't hurt anyone, and you'll never list them all, whereas a symptom alert catches the failure you didn't predict. A classic bad alert is paging whenever any host goes over 90 percent CPU. At my last company that fired most nights during a batch job while users saw nothing. We deleted the page, kept CPU on the dashboard for debugging, and replaced it with an alert on the checkout SLO burn rate. Pages dropped and the ones left were real."

"On my previous team, on-call got around thirty pages a week and people had started acknowledging them without looking, which is dangerous. I pulled a month of pages and sorted each one: did someone need to act, and was it urgent? Most weren't. Several fired on disk usage that cleaned itself up, some duplicated each other, and a few were for a service nobody used any more. We deleted the dead ones, turned the non-urgent ones into daytime tickets, merged duplicates, and replaced a set of per-host alerts with one SLO burn-rate alert per service. We also made a rule that every page gets a quick note: was it useful? Within two months we were at about five pages a week, and they were almost all real."

"I wouldn't assume the alert is wrong just because the dashboard looks fine. Dashboards often show totals, and a totally broken region or one big customer can disappear into an average. So I'd open the alert's own query and break it down by region, endpoint and customer. I'd send a real request myself and check the synthetic probes. Users not complaining yet doesn't mean much, since complaints lag by minutes or hours. If I find real impact, it's an incident. If I prove it's a false alarm, say the metric source restarted and sent a burst of stale data, I'd silence it for a short, set time, and then fix the alert itself, and note what happened so the next person doesn't repeat my investigation."

"Toil is work that's manual, repetitive and could be automated, that you do in reaction to something, that leaves nothing better behind, and that grows as the service grows. Restarting a stuck worker by hand every few days is toil. A planning meeting isn't toil, it's overhead, and debugging a brand-new failure isn't toil either, because you learn something. To pick what to automate first, I list the toil with rough numbers: how often it happens, how long it takes, and how likely a tired person is to get it wrong. Something that takes ten minutes but happens twenty times a week, often at night, beats a two-hour job that happens once a quarter. I also look for the fix that removes the need entirely, not just a script that does the same step faster."

"For me it's mostly about the next person. A good teammate writes a proper handoff at the end of the shift: what fired, what's still open, what to watch. They update the runbook when they find it wrong, instead of just remembering the fix themselves. They file a ticket for every page that wasn't useful, so the rotation gets quieter over time. They also don't play hero. If they're stuck, they escalate early and say so plainly, because a long silent struggle at night makes outages longer and burns people out. And they're generous about swaps when someone has a family thing. On-call only works long term if everyone trusts that the load is fair and the rotation keeps getting better."

"The main lever is making reliability something both sides measure and feel. I'd start with SLOs agreed jointly with the dev team and product, plus an error budget policy, so reliability becomes part of how they plan, not our opinion. I'd also have developers on call for their own services, at least as a second tier, because nothing improves code faster than getting woken by it. Then I'd make the right thing easy: a short production readiness review before launch, dashboards and alerts generated from a template, and good defaults for timeouts and retries in the shared libraries. And I'd keep it a partnership. I'd join their planning, share postmortems widely, and give credit loudly when a team's own fix removes a class of pages."

"Metrics are numbers aggregated over time, like request rate, error rate and latency. They're cheap to keep, so they're what I alert on and what shows me the shape of a problem. Traces follow a single request across every service it touches, with timing for each hop, so they tell me where the time or the error is. Logs are the detailed record of individual events, which is where I find the actual error message or the odd input that caused it. So when something breaks, I usually go metrics first to see scope and timing, then traces to narrow it to a service or a call, then logs for that service and time range to understand why. Linking them with a shared request ID is what makes that path fast."

"The average hides the slow requests. If most calls take 50 milliseconds and one in a hundred takes four seconds, the mean still looks fine while those users are having a terrible time. And the tail hits more people than it seems: if a page makes twenty backend calls, close to one page load in five will hit at least one slow call. The p99 shows me the experience of that unlucky slice. The trap is averaging. If I take each server's p99 and average them, the number I get isn't the p99 of anything, and it can hide one bad server completely. The right way is to collect latency as histograms, add the bucket counts from all servers together, and then calculate the percentile from the combined histogram."

"First I'd scope it. Is it every endpoint or one, every region or one, all hosts or a few, p50 or only the tail? That usually halves the search. Then I'd ask what changed, because no deploy doesn't mean no change. I'd check config pushes, feature flags, traffic volume and mix, a dependency's own dashboard, and anything scheduled at two, like a batch job or a backup. Next I'd look at saturation: CPU, memory and garbage collection pauses, and the things that queue quietly, like connection pools and thread pools. Traces help a lot here, they show which hop grew. Once I had a good host and a bad host, I'd compare them. In one case like this, the cause was a reporting job hitting the shared database every afternoon."

"Load shedding means deliberately turning some requests away, fast and cheaply, so the ones you do accept still get a good answer. Graceful degradation means giving a reduced answer instead of an error, like serving cached results or hiding the recommendations panel. Without either, an overloaded service queues everything, latency climbs past every client's timeout, and you end up doing work for people who've already given up. To build them in, I'd put a concurrency limit at the front, so beyond a certain number of requests in flight we return a quick 503 with a retry-after hint. I'd tag traffic by priority so checkout survives and background jobs are shed first. I'd pass deadlines along so a server drops work that's already expired. And I'd add switches for expensive features, so we can turn them off during a spike."

"When a service slows down, clients time out and retry, so it gets more traffic at the exact moment it can handle less. If three layers each retry three times, one user click can become dozens of calls at the bottom. That's how a brief hiccup becomes an outage that won't recover on its own. To design retries safely, I use exponential backoff with a cap, plus random jitter so clients don't all retry in step. I set a retry budget, for example retries can't exceed a small fraction of normal traffic, so they stop when things are clearly broken. I let only one layer retry, usually the one closest to the user. And I only retry operations that are safe to repeat, or I send an idempotency key so a repeated payment isn't charged twice."

"I'd start with the peak, not the average. Say today's peak is 4,000 requests a second, so the plan is 8,000. Then I'd load test one instance to find where latency breaks the SLO. If it breaks at 600 requests a second, I'd plan to run each one at about 400, which leaves room for spikes. So 8,000 divided by 400 is 20 instances. But if we run across three zones and must survive losing one, those 20 have to fit in two zones, so that's 10 per zone and 30 in total. Then I'd check everything behind the app: database connections and write load, cache memory, third-party rate limits, and cloud quotas. Last, I'd look at lead times, because some of those take weeks to raise."

"Chaos engineering is running controlled experiments where you inject a failure on purpose, like killing an instance or adding latency to a dependency, to check that the system behaves the way you believe it does. It's a test of a hypothesis, not random breakage. For a first experiment I'd pick something we think we already handle, say losing one instance behind the load balancer. I'd define the steady state, like the error rate and p99 for that service, and write the hypothesis: users see no change. I'd run it in staging first, then in production on a tiny slice, during working hours, with the on-call team told and a clear abort rule, like stop at once if the error rate climbs past a level we agreed beforehand. Whatever happens, I'd write up what we learned and fix any gap before growing the blast radius."

"First I'd make sure their slowness can't become my outage. I'd set a timeout well inside my own request deadline, allow at most one retry with an idempotency key so nobody is charged twice, and add a circuit breaker so when they're clearly down we fail fast instead of piling up threads. I'd give their calls their own connection pool, so product pages and login keep working even if payments are struggling. Then I'd look at fallbacks with the business: maybe we accept the order, queue the payment and confirm by email, or route to a second provider. I'd also measure their success rate and latency from our side, because that's the evidence I take to the vendor. And I'd be honest in our SLO: checkout can't be more reliable than that provider unless we have a fallback."

"My default is to stop the harm first and understand it later, and a rollback that we've practised is usually the quickest way to do that. If the bad release went out an hour ago and errors started with it, I roll back and debug on the old version in peace. Fixing forward makes sense when rolling back isn't safe. The classic case is a release that ran a database migration the old code can't read, or that already wrote data in a new format. It also makes sense when the fix is tiny, obvious, and can ship faster than a rollback. Before either, I check whether the change is behind a feature flag, because switching that off is often the fastest mitigation of all."

"At my last company our login service started failing for about a third of users on a weekday morning. I was on call, so I declared an incident in the first couple of minutes and took the commander role. I asked one engineer to lead the technical investigation and another to handle updates to support and the status page, so I wasn't trying to debug and talk at once. I set updates every fifteen minutes. Within ten minutes we saw errors only from one region, which had just picked up a certificate change. Instead of debugging it live, I had traffic shifted to the healthy regions, and errors dropped straight away. Then we fixed the certificate calmly. The postmortem added an expiry and rollout check to that pipeline."

"I once pushed a load balancer config change that I'd tested in staging, but in production it pointed a small share of traffic at a pool that had been drained for maintenance. Errors went up within a couple of minutes. I saw the alert, said in the incident channel straight away that it was probably my change, and rolled it back. The whole thing lasted about eight minutes. Afterwards I wrote the postmortem myself. The real gap wasn't my carelessness, it was that staging didn't mirror which pools were drained, and config changes went out everywhere at once. We added a check that refuses to route to drained pools, and moved config pushes to a staged rollout, one zone at a time."

"I'd acknowledge the page first, so it doesn't escalate while I'm reading. Then I'd read what the alert actually says and open the runbook and dashboard it links to. My first question is whether users are hurting: I'd check the error rate and latency against the SLO and see if it's stable or getting worse. Then I'd check recent changes, a deploy, a config push, a flag. If something went out in the last hour and the runbook says rollback is safe, I'd roll it back. If I can't tell what's going on within ten minutes and users are affected, I'd page the owning team's escalation contact. Waking someone up is far better than a long outage while I learn a system from scratch. I'd keep notes in the incident channel the whole way through."

"First I'd confirm the rollback really happened everywhere. It's common to find a few instances still on the new version, or a second region that didn't roll back. If it did complete, I'd think about what the release left behind that a rollback doesn't undo. A database migration, data already written in a new format, a poisoned cache entry, or a feature flag or config change that shipped alongside the code but lives somewhere else. Any of those keeps failing on the old version. And I'd be willing to drop my theory: maybe the timing was a coincidence and the real cause is a dependency or a traffic change. So I'd go back to the error messages and traces, and tell the incident channel plainly that rollback didn't fix it and what we're checking next."

"The one I remember best was a queue backlog that delayed order emails for about three hours. I wrote the postmortem with a timeline built from chat logs and graphs, then the impact in plain terms: how many customers, how long. For causes, I avoided stopping at the first answer. The trigger was a bad message that crashed consumers, but the contributing factors were that we had no dead-letter queue, our alert watched consumer CPU instead of queue age, and the runbook was out of date. I also wrote down what went well and where we got lucky. We ended with four action items, each with an owner and a date, tracked in the normal sprint board. Three were done within a month; the fourth I raised at our monthly review until it shipped."

"I'd take the question seriously but move it. Something like: the person who pushed it is in the timeline and did what anyone could have done with the tools they had, so the more useful question is why our system let that config reach production. Then I'd walk through the gaps: no validation step, no staged rollout, and an alert that took twenty minutes to fire. Those are things we can fix. If the manager pressed, I'd explain privately afterwards that if people get named and blamed, they stop reporting near misses and start hiding mistakes, and our next incident gets longer. Blameless doesn't mean nobody is accountable. It means we're accountable for fixing the system, and the action items carry owners and dates."