This page is for anyone interviewing for a cloud engineer job, whichever provider the team uses. Most rounds check the same things: that you understand how cloud services are split up, can design something that keeps running when part of it fails, keep access tight, move old systems without drama, and watch the bill. Expect a few questions on your path, a set of design and knowledge questions, and stories from real incidents and migrations. Each question shows what the interviewer is listening for, a shape for your answer and a sample you could say out loud. Swap in your own stories and the services you have used.
Search all questions by round, difficulty and level, or save the ones you want to practise.
Start: where you began, such as support, sysadmin, development or a degree project.
Proof: one or two real things you built or ran in the cloud.
Direction: the kind of cloud work you want more of, and why this role fits it.
"I started on the support desk looking after a handful of Linux servers, and the part I enjoyed most was automating the boring jobs. When my team began moving services to the cloud, I volunteered for the migration of our internal wiki and a reporting app. I learned networking and access control the hard way there, and I wrote the infrastructure as code so we could rebuild it. Since then I've run the monitoring and backups for about a dozen services. In two years I'd like to be the person who designs the landing zone and the standards other teams build on, not just the one who fixes things. This role looks like a step towards that, because you're building a shared platform rather than one-off projects."
Listing certificates and service names without a single thing you actually built, ran or fixed.
What the job is: building and running the base other teams deploy onto.
Why it suits you: the part of that work you enjoy, with a small example.
Honest trade-off: what you give up, such as less feature work, and why that's fine.
"I like the middle ground. A developer mostly thinks about features, and a network engineer mostly thinks about links and routing. A cloud engineer touches both, plus identity, storage, cost and reliability, and the result is something lots of teams depend on. In my last job the moment that hooked me was building a template that let any team spin up a new service with logging, backups and sensible network rules already in place. Deploys got faster and the security review got shorter, and I could see the effect across the whole company. I know it means fewer shiny features and more on-call and quiet plumbing work, but I enjoy making other engineers faster and keeping things running."
Saying cloud is where the jobs are, with nothing about the actual work.
Depth: name the provider and what you ran in production versus only tried.
Transferable ideas: networks, identity, compute, storage and managed data exist everywhere.
Ramp-up plan: how you would map what you know onto the new one in the first weeks.
"Most of my production work is on one provider: I've run virtual machines, managed databases, object storage, load balancers and the identity setup for three environments. On a second provider I've only done a sandbox project, so I wouldn't call myself experienced there. What carries over is the model. Every provider has private networks with subnets and firewall rules, a way to give workloads an identity, managed databases, object storage and a monitoring stack. To ramp up on a new one, I'd map each concept I know to its name there, read how its identity and networking differ, because that's where the surprises hide, and then rebuild something small I've already built elsewhere. I'd expect to be useful on routine work within a few weeks."
Claiming equal expert depth on every provider, then struggling with a basic follow-up.
How you learn: release notes filtered to what you use, hands-on in a sandbox, sharing with the team.
Adoption test: it solves a real problem, it's generally available, the team can support it and there's a way out.
"I don't try to follow everything. I read the release notes for the services we actually use, since changes there affect us directly, and skim the rest for anything that solves a problem we already have. When something looks useful, I try it in a sandbox account for an afternoon, then give the team a short demo with what I liked and what worried me. Before it goes near production, I ask a few questions. Does it fix a real pain for us, not just a nice-to-have? Is it generally available and not still in preview? Can the team monitor, secure and support it at two in the morning? And if it doesn't work out, how hard is it to move off? If the answers are good, we start with one low-risk service and grow from there."
Pushing the newest service into production because it's new, or having no way of keeping up at all.
IaaS: you rent virtual machines, disks and networks; you run everything on top.
PaaS: the provider runs the servers, OS and runtime; you bring code, config and data.
SaaS: you use finished software; you still own users, settings and data.
"Take a small online shop. With IaaS, we rent virtual machines and a network and install everything ourselves: the operating system, patches, the web server, the database and the shop code. We control a lot, but we also look after a lot. With PaaS, we push the shop's code to a managed app platform and use a managed database, so the provider handles the servers, the OS patching and the runtime, and we look after the code, its configuration, scaling settings and the data. With SaaS, we'd just subscribe to a hosted shop product. We don't run any software, but we still own who has access, how it's configured and the customer data inside it. The further you go from IaaS to SaaS, the less you run and the less you can change."
Saying that with SaaS the customer has no responsibility at all, including for data and access.
Provider side: buildings, hardware, the hypervisor and the provider's own network.
Your side always: data, identities, access rules and configuration.
The moving part: OS, runtime and engine patching shift to the provider as services get more managed.
"The provider is responsible for the security of the cloud itself: the data centres, the physical hardware, the virtualisation layer and their backbone network. We're responsible for what we put in it and how we set it up. On a virtual machine that's a lot: we patch the operating system, configure the firewall rules, manage the software and handle encryption and access. On a managed database the provider patches the engine and the OS underneath, and runs the hardware, but we still decide who can connect, whether it's reachable from the internet, whether encryption and backups are switched on, and what data goes in. The things that never move to the provider are our data, our identities and our configuration. In practice, the cloud security incidents I hear about most often come from that side, like an open storage bucket or a leaked key."
Believing that because a service is managed, the provider is also responsible for how it is configured and who can reach it.
Region: a separate geographic area with its own set of zones.
Zone: one or more isolated data centres in a region with separate power and networking, linked by fast connections.
Placement: zones for high availability, regions for disaster recovery, user latency and data rules.
"A region is a geographic area, like a city or part of a country, and inside most regions there are several availability zones. A zone is one or more data centres with their own power, cooling and networking, far enough apart that one failing shouldn't take the others with it, but close enough for low-latency links. So I spread an app across at least two zones in a region: instances behind a load balancer and a database with a standby in another zone. That covers a data centre failure without much latency cost. Going across regions protects against a whole-region problem and can put the app closer to users, but the latency is much higher, so database replication is usually asynchronous, data transfer costs more, and some data may be legally required to stay in a particular country."
Treating two zones as protection against a regional outage, or thinking a region is a single data centre.
Edge: DNS and a CDN in front of a load balancer that spans at least two zones.
App tier: stateless instances in an autoscaling group across zones, with health checks.
Data tier: a managed database with a synchronous standby in another zone, files in object storage.
Failure walk-through: what fails over, how long it takes and whether the remaining zones have enough capacity.
"Users hit DNS, then a CDN for static files, then a load balancer that spans two or three zones. Behind it, the app runs as stateless instances in an autoscaling group spread evenly across those zones, with health checks that remove a bad instance. Sessions live in a replicated cache or the database, and uploads go to object storage, so no instance holds anything unique. The database is a managed one with a synchronous standby in a different zone and automatic failover. If a zone dies, the load balancer stops sending traffic there, and autoscaling launches replacements in the healthy zones. The database fails over to the standby, which usually means a short interruption, often a minute or two depending on the service, so the app needs connection retries and short DNS caching. I'd also size it so the surviving zones can carry full load, otherwise failover just becomes an overload. A whole region failing is a separate disaster recovery plan."
Spreading only the web servers across zones while the database, cache or a single NAT or proxy sits in one zone.
Why: any instance can serve any request and can be killed or added without losing anything.
Sessions: a shared cache or database, or signed tokens the client carries.
Files: object storage, not the instance's local disk.
"In the cloud, instances come and go all the time. Autoscaling adds and removes them, a rolling deploy replaces them, and a failed health check kills one. If a server holds something unique, like a user's login session or a file they just uploaded, that thing disappears with it, and the next request may land on a different server that has never heard of the user. So I keep the app servers stateless. Sessions go into a shared store such as a replicated cache or the database, or into a signed token the client sends with each request. Uploaded files go straight to object storage, often with a short-lived upload link. Sticky sessions on the load balancer can hide the problem for a while, but users still get logged out when that instance goes away, and load spreads unevenly."
Relying on sticky sessions as the fix and not seeing what happens when that instance is replaced.
Real benefits: negotiating power, a best-fit service, customer or regulator demands, resilience to a provider-wide failure.
Real costs: double the skills, identity, networking and security tooling; the lowest common denominator; data transfer charges.
Where lock-in lives: mostly in data and managed services, not in the virtual machines.
Recommendation: a primary cloud, portable layers where cheap, and a written exit plan.
"I'd start by asking what problem we're solving. There are real reasons to use two providers: leverage in pricing talks, one service that's clearly better elsewhere, a customer or regulator that demands it, or protection against a rare provider-wide failure. But it isn't free. The team needs two sets of skills, two identity setups, two network designs and two sets of security tools. If you insist on running everywhere, you often end up using only the features both clouds share, which means giving up the managed services that save the most work. Moving data between clouds costs money and adds latency too. Lock-in mostly lives in your data and the managed services you use, not in virtual machines. So I'd usually recommend one primary cloud, portable choices where they're cheap, like containers, infrastructure as code and open database engines, and a written, costed exit plan. A second cloud then gets used on purpose, for specific workloads."
Agreeing that multi-cloud is always safer and cheaper, or dismissing it without asking what the leader is worried about.
Find the limit: which part broke first, such as the app tier, database, connections or an outside dependency.
Scaling speed: how long new instances take to boot and warm up, and what the scaling metric is.
Hidden caps: account quotas, instance limits, connection limits.
Changes: pre-scale for launch, faster startup, better metrics, caching, and a graceful way to shed load.
"First I'd find what actually broke first, because autoscaling only helps if the app tier is the bottleneck. If the database hit its connection or CPU limit, more app servers would make things worse. If the app tier was the problem, I'd look at timing. New instances might take several minutes to boot, install and warm up, while the traffic spike arrived in seconds. I'd also check whether we were scaling on the right metric, whether health check grace periods were killing slow starters, and whether we'd hit an account quota on instances. For the launch itself, I'd pre-scale by raising the minimum capacity before the expected peak, bake images so instances start faster, and scale on request count rather than CPU. For the database, I'd add caching or read replicas. And I'd add a way to shed load gracefully, like a queue or rate limit, then rerun the test."
Answering only with "raise the maximum instance count" without finding the bottleneck.
Address range: a private range that won't overlap with the office, on-premises or other cloud networks, with room to grow.
Subnets: public for the load balancer, private for app servers, private with no internet route for data, one of each per zone.
Paths and rules: inbound only through the load balancer, outbound from app via NAT, each tier accepts traffic only from the tier in front.
"First I'd pick a private address range that doesn't clash with our office network, on-premises data centre or any other cloud network we might connect later, and leave room to grow. Then per zone I'd create three subnets. A public one holds the load balancer and the NAT. A private one holds the app servers, with no public addresses. A data subnet holds the database, with no route to the internet at all. Inbound traffic only comes through the load balancer. App servers reach the internet for updates or outside APIs through the NAT, so nothing can connect to them directly. Firewall rules are chained: the load balancer accepts HTTPS from anywhere, the app accepts traffic only from the load balancer, and the database accepts its port only from the app. For the provider's own services, I'd use private endpoints so that traffic doesn't cross the internet."
10.20.0.0/16 app network, clear of office and on-prem ranges
10.20.0.0/24 public zone A load balancer, NAT
10.20.1.0/24 public zone B load balancer, NAT
10.20.10.0/24 private zone A app servers
10.20.11.0/24 private zone B app servers
10.20.20.0/24 data zone A database
10.20.21.0/24 data zone B database standby
Putting the database in a public subnet or allowing it from anywhere inside the network "because it has a password".
VPN: encrypted tunnels over the internet, quick and cheap, but bandwidth and latency vary.
Dedicated link: a private connection through the provider or a partner, steady and fast, slower to set up and costs more.
Details that bite: redundancy, dynamic routing, overlapping address ranges and name resolution both ways.
"The quick option is a site-to-site VPN: encrypted tunnels over the internet between our firewall and the provider's gateway. It can be up in a day and it's cheap, but the throughput is limited and latency changes with the internet. That's fine for admin traffic or a small migration. It stops being enough when we're moving large amounts of data, when an app needs steady low latency to a database on the other side, or when the business wants a guaranteed bandwidth. Then I'd use a dedicated private connection through the provider or a partner, which can take weeks to arrange. Those links are private but usually aren't encrypted by default, so sensitive traffic may still need encryption on top, like a VPN over the link or link-level encryption where it's offered. Either way I'd build two paths, use dynamic routing so failover is automatic, make sure address ranges don't overlap, and set up DNS so each side can resolve the other's private names."
Offering a single VPN tunnel for a critical production link with no second path and no thought about routing or DNS.
The problem: long-lived keys leak into code, logs and images, rarely get rotated and work from anywhere.
Inside the cloud: attach an identity to the machine, container or function; the platform hands out short-lived credentials.
Outside the cloud: federate the CI system so it swaps a signed token for short-lived access.
Leftover secrets: third-party keys live in a secrets manager, never in code.
"A long-lived key is just a string. It ends up in a config file, a container image, a build log or someone's laptop, it rarely gets rotated, and whoever finds it can use it from anywhere until someone notices. So inside the cloud I don't create keys for workloads at all. I give the virtual machine, container or function its own identity with a narrow set of permissions, and the platform issues it short-lived credentials that rotate automatically. The code never sees a stored secret. For a CI pipeline that runs outside the cloud, I set up identity federation: the pipeline gets a signed token from the CI system, and the cloud trusts that token, limited to one repository and branch, and exchanges it for short-lived credentials. The secrets we still need, like a third-party API key, go into a secrets manager that the workload reads through its own identity."
Suggesting keys in environment variables or an encrypted config file as the end state, without mentioning platform-issued identity.
Boundaries: separate accounts or projects per environment and per major workload, under one organisation.
Identity: single sign-on from the company directory, groups mapped to roles, no shared admin users.
Guardrails: organisation-wide policies that block risky actions and keep audit logs in a place workload admins can't touch.
Production access: read-only by default, time-limited elevation with approval and a sealed break-glass account.
"I'd use separate accounts or projects for dev, staging and production, and often per major workload too, all under one organisation. That boundary is the strongest one the cloud gives you, so a leaked dev credential or a bad script in staging simply can't reach production. People sign in through single sign-on from the company directory, and access comes from group membership mapped to roles, so when someone leaves, one change removes everything. At the organisation level I'd add guardrail policies that no account admin can override, such as blocking unused regions and stopping anyone from switching off audit logging. Those logs flow to a separate security account. In production, engineers get read-only access by default and request time-limited elevated access with a reason, which is logged. There's also a break-glass account with strong MFA, kept sealed and alerting whenever it's used."
Putting all environments in one account and relying on naming conventions or tags to keep people out of production.
What you found: how wide the access was and why it mattered.
Evidence: how you worked out which permissions were really used.
Rollout: new roles, team by team, with a quick path for anything you missed.
Result: what access looks like now and how it stays that way.
"When I joined my last team, almost every engineer was in one group with full admin rights in production, because it had been easier early on. That meant any stolen laptop session could delete anything. Rather than guess, I pulled several months of audit logs and the last-used data to see which actions each team actually performed. Most people only read logs and restarted services. I built three roles: read-only, operator for routine actions, and a time-limited admin role that needed a reason and was logged. I rolled it out one team at a time, kept a channel open for anything I'd missed, and fixed gaps within hours. After a month, full admin was used a few times for real emergencies instead of every day. I also set a quarterly review so access didn't creep back."
Removing access overnight with no data and no fallback, then blaming people for complaining.
Rehost: move the servers as they are; fastest, least cloud benefit.
Replatform: small changes to use managed services, such as a managed database; code mostly unchanged.
Refactor: redesign for cloud-native services; the most effort and the most benefit.
Deciding: business value, remaining life, deadline, team skills and licences, and whether to retire or replace instead.
"Rehost is lift and shift: we copy the servers into cloud virtual machines with almost no changes. It's the fastest and the least risky, but we keep all the old running costs and chores. Replatform means a few targeted changes, like moving the database onto a managed service or the app onto a managed runtime, while the code stays mostly the same. Refactor means redesigning the app to use cloud-native pieces, like containers, queues or serverless, which takes the most time but pays off for systems that change often or need to scale. The longer list also includes retire, retain, and repurchase, which means replacing the app with a SaaS product. To decide, I'd sort the portfolio by business value and how long each app will live, then look at the deadline. If we have to leave a data centre by a fixed date, most apps get rehosted first and the valuable ones get improved afterwards."
Saying every app should be refactored to microservices, or that lift and shift is always wrong.
Copy and sync: a full initial copy, then continuous replication of changes until cutover.
Prove it: check replication lag, compare row counts and checksums, run the app against the copy, rehearse.
Cutover: stop writes, wait for zero lag, promote the target, repoint the app, smoke test.
Rollback: keep the old database intact or replicate back until you're confident.
"I wouldn't try to copy it in one downtime window. I'd take a full copy into the cloud database first, then keep it in sync with continuous replication, either the engine's own replication or a change data capture tool, while production carries on as normal. Over the following days I'd watch replication lag, compare row counts and checksums on key tables, and point a test copy of the app at it to check versions, character sets and performance. I'd rehearse the cutover at least once and lower the DNS time-to-live early if clients connect by name. On the night, we put the app in maintenance mode so writes stop, wait for lag to hit zero, promote the cloud database, switch the connection settings and run smoke tests. The old database stays untouched, or we replicate back to it, so rolling back is one switch if something looks wrong."
Planning a single dump and restore of a large, busy database in the cutover window with no replication and no way back.
The app: what it was and which migration approach you chose.
What worked: the planning or testing that paid off.
The surprise: the thing you didn't see coming and how you handled it.
Lesson: what you now do first on every migration.
"I helped move an internal reporting app that ran on two servers in our data centre. We rehosted the app server and replatformed the database onto a managed service. The replication-based database move went well because we rehearsed the cutover twice. The surprise came from dependencies nobody had written down. A nightly job read files from a network share in the office, and one config file had a hard-coded IP address for an old authentication server. The first night in the cloud, the nightly report failed. We fixed it by moving the files to object storage and switching the hard-coded address to a proper DNS name. Since then, before any migration, I watch network traffic from the servers for a couple of weeks to map every connection, rather than trusting the documentation or people's memory."
Describing a migration where nothing surprising happened and no lesson came out of it.
Say it early: share the estimate and the reasons with the person who owns the decision.
Options: rightsize before moving, replatform a costly part, commitment discounts, or accept it for now.
Context: the deadline may be worth the cost, for example a data centre exit.
Plan: if it goes ahead, a dated plan to optimise after the move.
"I'd raise it straight away with the person who owns the decision, with the estimate and the reasons. Usually it's that the old servers are sized for peak and running around the clock, and we'd be paying for that same size by the hour. Then I'd give options. We could rightsize from real usage before moving, which often closes much of the gap. We could replatform the most expensive part, like moving the database onto a managed service, if that fits the six weeks. We could buy commitment discounts once usage is stable. Or we could accept a higher bill for now, because the real goal might be leaving a data centre whose lease is ending, and that's a valid business choice. If we go ahead, I'd attach a dated plan to cut the cost after the move, so it doesn't stay the expensive version for ever."
Either doing the move silently and letting the bill surprise everyone, or refusing to migrate because of the cost.
RPO: the most data you can afford to lose, measured in time.
RTO: how long the system can be down before it must be working again.
Patterns: backup and restore, pilot light, warm standby, active-active, from cheapest and slowest to most expensive and fastest.
Who decides: the business sets the targets per system; you show what each costs.
"RPO, the recovery point objective, is how much data we can afford to lose, expressed as time. An RPO of fifteen minutes means that after a disaster we may lose up to the last fifteen minutes of changes. RTO, the recovery time objective, is how long it can take to get the service back. RPO drives how often we back up or replicate. RTO drives how much we keep running and how automated the recovery is. So for a system that can be down for a day, backups copied to another region and a tested rebuild from code is enough. With a pilot light, the data is replicated and only the core is running, then we scale up. Warm standby is a smaller full copy that's always running. Active-active across regions gives near-zero numbers but costs the most. I'd get the targets from the business for each system and show the cost of each option."
Mixing up the two terms, or picking active-active for everything without asking what downtime actually costs.
The test: what you restored or failed over, and where.
Measured against targets: how long it really took versus the promised RTO and RPO.
Gaps: the missing steps or wrong assumptions it exposed.
Fixes: what you automated or redesigned, and how often you test now.
"We had a plan saying our main database could be restored in two hours. I ran the first real test by restoring the latest backup into a separate, isolated account. The backup itself was fine, but the whole thing took close to six hours. The database had grown a lot since the plan was written, so the restore alone took most of that time. Then the runbook skipped two steps: updating the DNS name the app used, and copying the secrets the app needed into the recovery account. We fixed it in two ways. For that database, we moved to a replica in a second region, which brought recovery down to well under an hour. And we turned the runbook into a script and scheduled a restore test every quarter, with the measured times reported to the service owners."
Saying backups are fine because the backup job reports success, without ever having restored one.
Confirm: check your own metrics and the provider's status, and declare an incident.
Communicate: tell users and the business what you know and when you'll update them.
Decide: compare the expected recovery time with how long a restore elsewhere would take, and whether you have what you need there.
Afterwards: agree targets with the business and build a tested plan to match.
"First I'd confirm it's really the provider and not us, by checking our own metrics against the provider's status page and other teams' reports, and I'd declare an incident with a clear lead so people don't all poke at things. Then I'd get a status message out to users and the business, with the next update time. The big decision is whether to wait or rebuild elsewhere. That depends on whether we have backups copied to another region, whether our infrastructure is in code so we can stand it up there, and how long that would honestly take compared to the provider's expected recovery. I'd avoid risky changes in the failing region because its control plane may be unreliable too. Afterwards, I'd push for the business to set real recovery targets for this app, then build to match, with cross-region backups at a minimum, and test it."
Jumping straight into a rushed rebuild in another region with no plan, or doing nothing and waiting without telling anyone.
Different jobs: replication survives hardware and zone failures, backups survive mistakes and attacks.
The risk: a bad delete, a bad migration or ransomware is copied to every replica at once.
Cheaper option: shorter retention, cheaper storage tiers or less frequent backups, not none.
"I'd agree that replication is great, but explain that it protects against a different problem. If a disk or a whole zone fails, the replica takes over. But if someone deletes a table by mistake, a bad migration corrupts data, or an attacker encrypts it, the replica copies that damage within seconds. Only a backup lets us go back to how things were at a point in time. So stopping backups means one bad command could lose the data for good. Then I'd try to meet the cost concern. We could keep backups for fewer days, move older ones to a cheaper storage tier, or back up less often if the business can accept losing a bit more data. I'd also suggest keeping at least one copy somewhere the main account can't delete it."
Agreeing that replicas are the same as backups, or arguing for backups without offering any way to reduce the cost.
See it: tag everything with an owner and environment, set budgets and anomaly alerts.
Size it: rightsize from real usage, switch off non-production out of hours, clean up leftovers.
Buy it smart: commitment discounts for a steady baseline, interruptible capacity for work that can restart.
Routine: a monthly review with the teams who own the spend.
"It starts with visibility. Every resource gets tags for owner, team and environment, enforced by policy, so the bill can be split by team and nobody's spend is anonymous. I set budgets and anomaly alerts so a spike shows up the next day, not at month end. Then I rightsize from real usage, because most servers and databases are sized for a guess. Development and test environments get switched off at night and at weekends. I look for leftovers like unattached disks, old snapshots, idle load balancers and forgotten public IP addresses. Once usage is stable, I buy commitment discounts for the steady baseline only, and use interruptible capacity for batch jobs that can restart. I also watch data transfer, since traffic leaving the cloud or crossing zones can quietly add up. Finally, a short monthly review with each team keeps it from creeping back."
Reducing cost control to buying reserved capacity once, with no tagging, ownership or ongoing review.
Finding it: how you broke the bill down and what stood out.
Changes: what you changed, in what order, and who you agreed it with.
Safety: how you checked nothing got slower or broke.
Result: what it saved, in plain words, and how you stopped it creeping back.
"At my last job the bill had grown every month for half a year and nobody knew why. I first got tags onto everything so we could split spend by team and environment. That showed two things: our development and test environments ran all day and night, and the biggest databases were sized for a load they never saw. I agreed a schedule with each team to switch non-production off at night and at weekends, with an opt-out tag for anyone who needed it. For the databases, I looked at two weeks of usage, dropped one size at a time during quiet hours and watched latency before going further. I also added lifecycle rules to move old logs to cheaper storage. Together that cut the monthly bill by about a third, and nobody saw a slowdown. A monthly review with the teams kept it there."
Cutting resources without talking to owners or watching the effect, or claiming big savings with no sense of how they were measured.
User signals: latency, traffic, errors and saturation, plus checks from outside.
Underneath: database, queues, dependencies, certificate expiry, disk, with logs and traces to dig in.
Paging rule: page on symptoms that hurt users and need action now; everything else becomes a ticket or a dashboard.
Every page: has an owner and a short runbook.
"I start with what users feel: request rate, error rate, latency at the slow end, not just the average, and how close things are to their limits, like CPU, memory, connections or queue depth. I add synthetic checks from outside the cloud that hit the login and main pages, because internal health can look fine while users can't get in. Underneath, I watch the database, queues, outside dependencies and certificate expiry, and send logs and traces somewhere central so we can dig in. For paging, my rule is that a page must mean users are hurting, or soon will be, and a person needs to act now. So I'd page on the error rate or latency breaking our service level objective, not on one server's CPU hitting ninety. Everything else becomes a ticket or a dashboard. Every page has an owner and a short runbook."
Paging on every resource metric with a fixed threshold, with no link to what users actually experience.
Situation: what users saw and how you found out.
Debugging: the evidence you checked, in order, and what pointed to the cause.
Fix: how you restored service first, then fixed the real cause.
After: the alert, test or design change that stops it happening again.
"At my last company our API started timing out during a morning traffic peak, about an hour after a routine release. The app servers looked healthy on CPU, but the error logs were full of database connection timeouts. The database dashboard showed it had hit its maximum connections. The new release had raised the connection pool size per instance, and autoscaling had added instances for the peak, so together they asked for more connections than the database allowed. We rolled back the release to restore service within about fifteen minutes. Then we fixed the pool size, added a connection pooler in front of the database, and set an alert on connection count at a safe level below the limit. The bigger lesson was that pool size times maximum instances has to fit the database, so we added that check to our release review."
A story with no evidence trail, where the fix was a lucky restart and nothing changed afterwards.
Measure: pull recent alert history and ask of each one whether anyone needed to act.
Cut: delete, downgrade to a ticket, or merge the ones that didn't need action.
Rebuild: alert on user symptoms with sensible durations; every page gets an owner and a runbook.
Keep it clean: track pages per shift and review them regularly.
"I'd treat it as a real risk, because the day a true alert fires, nobody will look. I'd start by pulling the last month of alerts and going through them with the on-call engineers, asking for each one whether someone had to act. Usually a large share never needed anything. Those get deleted or turned into tickets or dashboard items. Alerts that fire in bursts for the same cause get grouped. Thresholds that trip on brief spikes get a duration, so they only fire if the problem lasts. Then I'd make sure the remaining pages are about what users feel, like error rate or latency, rather than one machine's CPU. Each page gets an owner and a short runbook. Finally, I'd track the number of pages per on-call shift and review it every few weeks, so noise doesn't creep back."
Suggesting the on-call person just needs to be more disciplined about reading every alert.
Tone: assume people acted sensibly with what they knew; ask how, not who.
Content: a timeline, what made it worse or better, and why the system allowed the mistake.
Actions: a few specific fixes with owners and dates, tracked until done, and the write-up shared widely.
"I start the meeting by saying plainly that we're here to fix the system, not to find a culprit, and I mean it. If someone ran a command that took production down, the useful question is why the system let one command do that, and why they had reason to think it was safe. We build a timeline together from logs and chat, including what helped us recover and what slowed us down, like a missing dashboard or an unclear runbook. Then we pick a small number of concrete fixes, each with an owner and a date, for example a guardrail, an alert or a safer default. I track them to completion, because a review with open actions is just a story. And I share the write-up beyond the team, since other teams often have the same gap."
Ending the review with "the engineer should be more careful" as the main action.
ClapAssist is an AI interview assistant for Mac and Windows. It listens to the interview on your computer and shows you what to say, in short lines you can read while you talk. Your resume and notes are never stored on our servers. It stays out of screen share on every plan; only you can see it.