AWS Interview Questions for 10+ Years Experience (Senior)

“I think of an SCP as the fence around an account, not a key to any door. For a request to succeed, the SCPs on every level from the root down to the account have to allow it, and then IAM has to allow it too. So an SCP that allows everything grants nothing on its own. The catch people miss is scope. SCPs do restrict the root user of a member account, which is why they're the right place for things like 'nobody leaves the organization' or 'no resources outside our approved regions'. But they don't apply to the management account at all, and they don't restrict service-linked roles. That's why I keep nothing running in the management account and treat access to it as the most sensitive thing we have. I also roll any new SCP out to a sandbox OU first, because a bad deny can break every pipeline in the org at once.”

“The deputy here is the vendor's service. It holds permission to assume roles in lots of customer accounts. If the only check is 'trust the vendor's account', then customer A could type customer B's role ARN into the vendor's settings page, and the vendor would happily assume it, because from AWS's side the vendor is allowed. The fix is an external ID. The vendor generates a unique value per customer and always passes it when assuming the role, and the customer's trust policy only allows the assume call when that exact value is present. Customer A can't supply B's external ID because the vendor controls it. The same idea applies when an AWS service assumes a role for you: I add a source ARN or source account condition so the service can only act for my own resources. And the role itself still only gets the read actions the tool needs.”

“The risk with letting teams create roles is simple escalation: create a role with admin rights, then assume it. Permission boundaries solve that. A boundary is a managed policy that sets the maximum a role can do; the role's effective permissions are the overlap between its own policies and the boundary. So I write one boundary per team, covering their services and nothing in IAM or billing. Then the team's deploy role may create roles only if the create call attaches that exact boundary, which I enforce with a condition on the permissions boundary key. The part people forget is the escape hatches: I also deny changing the boundary policy itself, deleting it, and removing the boundary from a role. And I scope their role names to a path, so they can't edit the platform's roles. With that, teams ship on their own and the worst they can build is a role no stronger than the boundary.”

“The classic problem is server-side request forgery. If an app will fetch any URL a user gives it, an attacker points it at the metadata address and gets back the temporary credentials of the instance's role. IMDSv1 answered a plain GET, so that worked. IMDSv2 is session based: you first send a PUT with a TTL header to get a token, then send that token as a header on every read. Most forgery bugs can only make a GET with no custom headers, so they fail. There's also a hop limit on the token response, which stops it reaching further than it should, for example into containers that shouldn't have the host's role. To enforce it at scale, I set tokens to required in every launch template and AMI pipeline, add an SCP that denies launching instances unless the metadata tokens condition says required, and use a Config rule to list the old instances still allowing v1 so teams can fix them.”

“SSE-KMS uses envelope encryption. When I put an object, S3 asks KMS for a data key, encrypts the object with it, and stores the data key encrypted next to the object. When I read it, S3 sends that encrypted data key to KMS to decrypt. So at high request rates, every S3 call is also a KMS call, and KMS has request-rate quotas per account and region. That's where the throttling comes from. The first fix is S3 Bucket Keys: S3 gets a bucket-level key from KMS and uses it to make data keys locally for a while, so the KMS call count drops massively. It applies to new objects, so old ones still hit KMS until they're rewritten. Then I check what else in the account shares the quota, because one noisy job can starve the rest, and only then ask for a quota increase. I'd also make sure clients retry throttles with backoff and jitter.”

“KMS is different from most services because the key policy is the main gate. IAM permissions only count for a key if its key policy lets the account delegate access through IAM. The default key policy does that by trusting the account, so people get used to IAM just working. But if someone wrote a custom key policy naming only one application role, an admin with full KMS rights in IAM still gets denied. So I check the key policy first, not the user. The bigger risk is lockout. KMS has a safety check that rejects a policy which would stop the person setting it from changing it again, but it can be bypassed, and it doesn't protect anyone else. If every admin is locked out, only AWS support can help, and that's slow. So key policies live in code, get reviewed like production changes, and always keep a break-glass admin role.”

“Every AWS service has a control plane, which creates, changes and deletes things, and a data plane, which serves what's already running. Data planes are simpler and built to be more available. During a big incident the control plane is often what struggles, because everyone is trying to launch capacity at once. So if my recovery plan is 'when an AZ dies, Auto Scaling launches more instances elsewhere', I'm betting on the weakest part at the worst time. Static stability means the system keeps working through a failure without having to change anything. For AZ loss, I run in three AZs and give each enough capacity for half the peak, so losing one still leaves the full peak covered. That costs more, and I'd show leadership exactly what it buys. For failover between regions I prefer health-checked DNS records that flip on their own, and we rehearse it on a normal weekday.”

“The danger is that retries multiply. If the edge, the API and the service layer each try three times, one click can become twenty-seven calls on the database, right when it's slowest. The retries also arrive in waves, so when the service recovers, the backlog knocks it over again. That's how two minutes becomes twenty. My defences: retry at one layer only, usually the one that knows whether the work is safe to repeat, with capped exponential backoff and jitter so clients spread out. I add a retry budget, so retries can never be more than a small share of normal traffic. Circuit breakers stop calls to a dependency that's clearly failing, and the service sheds load it can't handle instead of queueing forever. Timeouts get shorter as you go down the stack. And since the AWS SDKs already retry with backoff, I check we aren't wrapping their retries in our own.”

“Right now everyone shares one fleet and one database, so any failure is everyone's failure. I'd move to cells. A cell is a complete, independent copy of the service, with its own compute, queues and data, often in its own account, and sized to a known maximum. A thin routing layer looks up which cell a customer belongs to and forwards the request. It does almost nothing else, so it can be the most stable part of the system. Now a poison request, a runaway customer or a bad config hurts one cell, not the whole product. Deploys go cell by cell, starting with a small one, with automatic rollback if its alarms fire. Big customers get per-customer limits, or a cell of their own. The costs are real: more copies to run, cross-cell features like global search get harder, and moving a customer between cells needs a data migration plan. I'd present it to leadership as how many customers any one incident can touch.”

“This is a nice trap. If the health check calls the database, a database blip makes every instance fail at the same moment, because they all share it. The load balancer itself is fairly forgiving here: when every target in every enabled zone is unhealthy, an ALB fails open and keeps sending traffic to all of them, since sending nowhere helps no one. The bigger danger is the Auto Scaling group. If it uses ELB health checks, it sees the whole fleet unhealthy and starts terminating and replacing instances. Now a thirty-second database blip turns into many minutes of cold starts, warming caches and a thundering herd on the database when it recovers. So I keep the load balancer check shallow, meaning the process is up and can serve, and I watch dependencies through metrics and alarms. If a deep check is needed, I make it degrade softly instead of failing the whole node.”

“DynamoDB only guarantees uniqueness on the primary key. A GSI on email will happily hold two items with the same email, and it's updated asynchronously, so 'query the index, then write' has a race: two requests both see no match and both write. The pattern I use is a guard item. Next to the user item, I write a second item in the same table whose key is EMAIL plus the address. Both puts go in one TransactWriteItems call, each with a condition that the key doesn't already exist. If the email is taken, the whole transaction is cancelled, and I return 'already registered'. The costs: transactional writes use twice the write capacity of normal ones, and changing an email means deleting the old guard and adding the new one in the same transaction. I also normalise the email first, or two spellings of one address slip through.”

“The first surprise is that a new replication rule only covers objects written after it's created. Everything already in the bucket stays put unless I run S3 Batch Replication. Second, deletes. Delete markers are only replicated if I turn that on, and deleting a specific version is never replicated. That's useful protection if someone wipes the source, but it means the replica isn't a mirror. Third, prerequisites: both buckets need versioning, and objects encrypted with KMS keys are skipped unless the rule opts in and the replication role can use a key in the destination region. Before I'd trust it, I'd check the replication status on a sample of objects, watch the replication metrics for failures and lag, and in a drill point a real reader at the replica. Bucket policies, lifecycle rules and access points don't come along, and the app needs those there too.”

“Standard RDS is an engine on an instance with its own disk, and Multi-AZ adds a standby with its own copy. Aurora separates compute from storage. One cluster volume keeps six copies of the data across three AZs, and a write is durable once four of the six acknowledge it. So it can lose an AZ and keep writing, and it repairs copies in the background. Readers attach to that same volume instead of replaying changes into their own storage, which is why their lag is usually far lower than a classic read replica's. It isn't zero, so read-after-write still goes to the writer. On failover, Aurora promotes a reader based on the promotion tiers I set, and moves the cluster endpoint to it, so I keep a reader sized like the writer in the top tier. And clones use copy-on-write on the shared volume, so a team can get a full-size copy of production data quickly, and only pages that change take new space.”

“Iterator age is how far behind the newest record the consumer is, so one shard falling behind points at that shard. There are two usual causes. The first is a record the function can't process. With streams, Lambda keeps order per shard, so when a batch fails it retries that same batch, and by default it keeps going until the records expire from the stream. Everything behind it waits. The fixes live on the event source mapping: bisect batch on error, so Lambda splits the batch to isolate the bad record, a maximum retry count and maximum record age, so it gives up, and an on-failure destination, so details of the failed batch land somewhere we can replay from. Partial batch responses work here too. The second cause is a hot shard, one partition key carrying too much traffic. The parallelisation factor lets several batches from one shard run at once while keeping order per key, but the lasting fix is a better key.”

“Asynchronous invokes go into a queue that Lambda runs for you, and the caller gets a success as soon as the event is queued. If the function throws, Lambda retries twice by default, with a wait between attempts. If the function was throttled instead, Lambda keeps retrying for longer, up to the maximum event age, which can be as long as six hours. Once retries or age run out, the event is dropped, and nothing tells you unless you set it up. So after an hour of errors, most of those events used up their retries and are gone. My design: an on-failure destination, usually an SQS queue, with an alarm on its depth, so each failure is kept with the original event and the error. Handlers are idempotent, because an async event can occasionally run more than once. And we keep a small tool that replays events from that queue after a fix. On EventBridge rules, I also set a dead-letter queue for events it couldn't deliver at all.”

“They solve different problems, so at that scale I use all three. Transit Gateway is a routing hub. VPCs attach to it, separate route tables keep production, non-production and shared services apart, and it's where on-premises links land. It gives broad network reach, which also means a broad blast radius, and it needs address ranges that don't overlap. Shared VPCs, through Resource Access Manager, let a central network account own the VPC and subnets while teams launch resources into them from their own accounts. That cuts the number of VPCs and keeps network changes with the network team, but teams share one network boundary. PrivateLink is the narrowest: a consumer gets an endpoint for one service behind a load balancer, connections only start from the consumer side, and it works across overlapping ranges. For team-to-team APIs I prefer it, because it exposes a service, not a network. And I'd put address planning in one place, with IPAM, so ranges stop colliding.”

“Auto Scaling doesn't know an instance is halfway through a job, so on scale-in it just picks one and ends it. There are two tools. A terminating lifecycle hook puts the chosen instance into a wait state; the worker stops taking new work, finishes the current job, then completes the lifecycle action so termination goes ahead. The other option is scale-in protection set by the worker itself: it protects itself when it picks up a job and removes protection when it's idle, so only idle instances are chosen. I'd also fix the scaling signal. CPU is a poor fit for a queue of jobs. Backlog per instance, meaning messages waiting divided by running workers, tracks the real need. Even with all that, instances can still disappear, for example on a Spot interruption, so the job itself should be safe to retry or able to resume from a checkpoint.”

“By default a pod can call the instance metadata service just like the node, and get the node role's credentials. So if one app needs to write to a bucket and that permission goes on the node role, every other pod on that node can write there too. The fix is per-workload identity. With IAM roles for service accounts, the cluster has an OIDC provider, a Kubernetes service account is annotated with a role, and the SDK in the pod swaps its projected token for that role's credentials. EKS Pod Identity does the same job with an agent on each node and associations managed through the EKS API, which is simpler across many clusters. Either way, the role's trust policy names one namespace and service account, not anything in the cluster. Then I close the fallback: IMDSv2 required on nodes with a hop limit of one, so pods on the pod network can't reach the node's metadata, and a node role cut down to what the node itself needs.”

“I'd aim for rules that are enforced by the pipeline, so the platform team isn't reviewing every pull request. First, a paved road: versioned modules for the common shapes, like a service behind a load balancer or a queue with a dead-letter queue, with logging, encryption and tags built in. Teams using them get through faster, which is the real incentive. Second, stack boundaries. Networking, data stores and the app live in separate stacks, so a routine app deploy can't touch the database, and stateful resources keep a retain policy. Third, policy-as-code checks in the pipeline for things like public buckets and open security groups, with SCPs underneath as the floor nobody can get around. Fourth, nothing changes by hand in production, and drift detection runs on a schedule. Teams own their own stacks and on-call. My team reviews only exceptions, and when one keeps coming up, we turn it into a module.”

“I treat the account as the strongest boundary AWS gives me, so each team gets separate production and non-production accounts, and developers get sandboxes. Organizational units follow policy, not the org chart: security, infrastructure, workloads split into production and non-production, a sandbox OU with looser rules, and a suspended OU for accounts being closed. Guardrails attach at the OU, so a new production account gets production rules automatically. A few central accounts do shared jobs: a log archive that receives an organization-wide CloudTrail and Config data, which almost nobody can touch, a security tooling account set as delegated admin for services like GuardDuty and Security Hub, and a network account that owns the hub and shared subnets. The management account runs nothing else. Accounts come from a vending pipeline, so each arrives with the baseline, single sign-on groups mapped to roles, and a budget alert. People sign in through single sign-on, never as IAM users.”

“At my last company I led moving about sixty applications and their databases out of a data centre whose lease was ending. We spent the first month on discovery, because the dependency map everyone believed in was wrong in several places; network flow data showed apps talking that nobody had listed. We sorted each app: most were rehosted as they were, a few databases moved to managed RDS, and around ten apps were retired outright. We ran it in waves, starting with internal tools so we could fix our runbook on low stakes. For databases we did a full load and then ongoing change capture, so the final cutover was only minutes of catch-up. Each cutover had a rehearsal, a go or no-go meeting and a rollback we'd actually tested. One wave did roll back because of a licensing check tied to hardware. We finished two weeks before the lease ended.”

“I led a move of our main API from containers to Lambda, mainly to cut the time we spent on patching and scaling. On paper it fit. In practice, a big share of our traffic was long-running report requests and some endpoints kept warm in-memory caches, and I hadn't looked closely enough at the traffic mix. Latency got worse, the database got hammered by connections, and after two months we'd moved only a third of the endpoints. I called a stop, which was hard because I'd championed it. We kept the event-driven parts on Lambda, where they worked well, and moved the rest back. Afterwards I changed how we start projects like this: a one-page decision record, a small proof on the nastiest endpoint rather than the easiest, and exit criteria written down at the start, so stopping is a planned outcome, not a failure.”

“First I make the roles explicit, because arguing usually means nobody's in charge. I take incident lead, ask one person to own communications, and name one lead per affected service. Then I move the goal from 'why' to 'stop the bleeding': what changed recently, can we roll it back, can we move traffic away from a bad zone or turn off a non-essential feature to shed load? The root cause debate goes to a side thread while we mitigate. I also check the AWS Health Dashboard early, so we don't spend an hour debugging our code for a problem on the provider's side. For the VP, I'd say politely that the communications owner will post updates every fifteen minutes in the leadership channel, and we stick to that clock, even when the update is 'no change, trying this next'. Most leaders relax once updates are predictable. Afterwards we run a blameless review with a clear timeline and actions that each have an owner and a date.”

“For hiring, I care more about judgement than memory. In interviews I give a small design problem with a twist, like a database that can't go down during a migration, and a broken setup to debug together, and I listen for how people reason and when they ask questions. Service trivia tells me very little. For speed, the trick is making mistakes cheap. We have sandbox accounts, guardrails from SCPs and pipeline checks, and nothing changes in production by hand, so a new engineer can ship in their first week without scaring anyone. For growth, I hand people ownership of a real system early, pair them with a senior on their first incidents, and rotate who leads design reviews. My own goal is to become unnecessary for approvals. When I find myself the bottleneck on something, that's a sign we're missing a guardrail or I haven't trusted someone enough yet.”