This page is for backend developers and architects facing a microservices round, whatever language they build in. Most rounds open with when microservices make sense at all and how to draw service boundaries, then move to sync and async communication, gateways and discovery. The middle of the round tests failure handling: timeouts, retries, circuit breakers and what happens to data when one database becomes many. Senior rounds add sagas, idempotency, tracing across services and changing an API without breaking its callers. Each question shows what the interviewer is really checking, the shape of a strong answer and a short answer you can say out loud.
Search all questions by round, difficulty and level, or save the ones you want to practise.
Definition: a small service you deploy on its own, owning one business capability and its data.
Modular monolith: clear internal modules, but one deployable and usually one database.
Trade: independent deploys and scaling versus network calls, partial failure and more operations work.
"A microservice is a service built around one business capability, like orders or payments, that a team can change and deploy without touching anything else. It owns its own data and talks to other services only over the network, through an API or messages. A modular monolith can have equally clean modules inside, but it ships as one unit, runs as one process and usually shares one database. So the real difference isn't size, it's independence. With microservices I can deploy payments ten times a day, scale search on its own and keep a crash in one area from taking down the rest. The price is that calls that used to be in-memory now go over the network and can fail, data is spread out, and I need solid monitoring and automation. For a small team I'd usually start with a modular monolith and split out services when the pain is real."
Defining a microservice only by lines of code, or claiming microservices are simply the modern and better choice.
Definition: separate services that still have to change, deploy or fail together.
Signs: lockstep releases, a shared database, long chains of sync calls, shared domain libraries.
Fix: redraw boundaries, give each service its data, and cut the chatty sync calls.
"A distributed monolith is when you've split a system into services, but they're still tightly coupled, so you pay for network calls and extra operations without getting independent releases. The first sign I look for is release coordination: if shipping a feature means deploying four services in a set order, they aren't independent. Another is a shared database, where one team renaming a column breaks three other services. Long chains of synchronous calls are a third sign, because one slow service then stalls everything. A fourth is a shared library holding the domain models, so every service has to upgrade at the same time. To fix it, I'd find which services always change together and consider merging them, move each table under one owner, and replace some sync calls with events or local copies of the data a service needs."
Saying more services always means less coupling, or not recognising a shared database as coupling.
Pick: a capability at the edge with a clear boundary and real value in splitting.
Route: put a routing layer in front and move traffic for that capability gradually.
Data: give the new service its own data, keep both sides in sync during the move.
Retire: delete the old code path once traffic and data have fully moved.
"I'd use the strangler fig approach. First I'd pick something with a clean edge, like notifications or invoicing, rather than the core that everything touches. I'd put a routing layer, usually the gateway or a proxy, in front of the monolith so I can send requests for that one capability somewhere else. Then I'd build the new service and switch traffic over gradually, maybe a small slice of users first, with the monolith still there as a fallback. Data is the hard part. The new service needs its own store, so for a while I'd sync data from the monolith, often through events or change data capture, until the new service becomes the owner. Once all traffic runs through the new service and nothing reads the old tables, I delete the old code. Each step is small and reversible, which is the whole point."
Proposing to rewrite the whole monolith at once, or ignoring how the data moves.
Why: the pain that justified the split.
How: routing, data migration and how you reduced risk.
Result: what improved, with evidence.
Hindsight: one honest thing you'd change.
"In my previous role, notifications lived inside our main monolith, and every change to an email template meant a full release of the whole application, which we could only do weekly. We decided to pull notifications out as the first service. I added an event in the monolith whenever something needed a notification, and the new service consumed those events. For a couple of weeks both sent in parallel, with the new one writing to a log instead of actually sending, so we could compare outputs. Then we switched over one notification type at a time. Afterwards, the notifications team could ship several times a day. What I'd do differently is set up tracing and a dead-letter queue before the switch, not after. For the first week, when a message failed, we had no easy way to see which user was affected or replay it."
Claiming the migration had no problems, or having no concrete reason for the split.
Business first: split by business capability and bounded context, not by layer.
Cohesion test: things that change together stay together.
Data and teams: one owner for each piece of data, one team for each service.
"I start from the business, not the database. I look for bounded contexts from domain-driven design, which are areas where a word has one clear meaning. 'Order' in checkout means a basket being paid for, while in the warehouse it means a parcel to pack, so those are probably different contexts. Then I test a candidate boundary. Do the things inside it change together for the same reasons? Can this service answer most requests with its own data? Could one team own it end to end? If two services keep needing each other's data for every request, the line is in the wrong place. I also avoid splitting by technical layer, like a separate service for validation or for data access, because every feature then touches all of them. And I'd rather start with slightly bigger services and split later than start too small."
Making one service per database table, or per technical layer like 'validation service'.
Different meanings: each context cares about different customer facts.
One owner: a single service is the system of record for identity and contact details.
Local copies: others keep only the fields they need, updated by events.
Trade-off: accept brief staleness, and decide which reads truly need the latest value.
"First I'd point out that 'customer' means different things to each of them. Billing cares about payment methods and billing addresses, support cares about tickets and contact history, orders care about delivery addresses. So I'd have one service, say a customer or account service, be the system of record for identity and core contact details. Billing would own billing data itself. The others shouldn't query the customer service on every request, because that makes it a bottleneck and a single point of failure. Instead, the customer service publishes events like 'customer updated', and each service keeps a local copy of just the fields it needs. That means copies can be slightly stale for a moment, so for anything where that matters, like checking an account is still active before charging, I'd do a direct call at that moment. What I'd never do is let all three share one customers table."
Letting every service read and write one shared customers table, or making every request call the owner synchronously.
Evidence: how often they change together, and why.
Cause: shared data, a chatty API, or one concept split in two.
Options: merge, or move the boundary so each owns a full capability.
Reason to keep apart: very different scaling, security or release needs.
"I'd look at the evidence first. Over the last few months, how many changes touched both services, and why? If almost every feature does, that's a strong sign the boundary is in the wrong place, and we're paying for a network hop and two pipelines without getting independence. Then I'd find the cause. Sometimes one business concept has been split across the two, so the logic for, say, pricing rules lives half in each. Sometimes one keeps calling the other for data it really should own. I'd recommend merging them, or moving the boundary so each owns one complete capability. I'd only keep them apart if there's a real reason, like very different scaling needs, a security boundary around sensitive data, or different teams owning them. Merging services isn't a failure. It's correcting a line we drew before we understood the domain."
Insisting services must never be merged, or proposing a shared library to keep them in sync.
Synchronous: the caller needs the answer to continue, like a price check.
Asynchronous: the work can happen later or fans out to many, like emails or indexing.
Costs: sync couples availability; async brings eventual consistency and harder debugging.
"I ask one question: does the caller need the answer right now to finish what it's doing? If checkout needs to know whether a coupon is valid before showing the total, that's a synchronous call, usually HTTP or gRPC with a timeout. If placing an order should also send an email, update the search index and notify analytics, none of that has to happen before the user sees 'order placed', so I publish an event and let those services react in their own time. Async is great for decoupling: if the email service is down, orders still go through and the messages wait in the queue. It also absorbs traffic spikes. But it costs something. Data becomes eventually consistent, the flow is harder to follow, and consumers must handle duplicate messages. Sync is simpler to reason about, but every sync call ties my availability to the other service's."
Saying async is always better, or not mentioning that consumers must cope with duplicates.
gRPC: HTTP/2, Protocol Buffers, a typed contract with generated clients, streaming.
REST with JSON: readable, works everywhere, easy to debug and cache.
Decision: internal, high-volume, many languages leans gRPC; public or browser-facing leans REST.
"Both work, so it depends on the traffic and the teams. gRPC runs over HTTP/2 and uses Protocol Buffers, so messages are compact and binary, and the contract lives in a .proto file that generates typed clients in each language. That catches mismatches at build time, and it supports streaming and deadlines built in. For busy internal calls between services written in different languages, that's a strong fit. REST with JSON is readable, every tool understands it, I can test it with curl, and it plays well with HTTP caching. Browsers can't speak plain gRPC, so they need gRPC-Web and usually a proxy in between, which is why anything public or browser-facing I'd keep as REST. The honest answer in many companies is both: REST at the edge, gRPC inside where performance and strict contracts matter. What I wouldn't do is switch just for speed without measuring that serialisation was actually the bottleneck."
Claiming gRPC is always faster so it's always better, or not knowing browsers can't call plain gRPC.
Orchestration: one coordinator tells each service what to do next and tracks state.
Choreography: services react to each other's events, with no central brain.
Choice: choreography for short, simple flows; orchestration once steps, branches and failures multiply.
"In orchestration, one component, the orchestrator, runs the process. It tells payment to charge, waits, tells inventory to reserve, waits, and knows exactly where each order is. In choreography, there's no boss. Orders publishes 'order placed', payment reacts and publishes 'payment taken', inventory reacts to that, and so on. Choreography keeps services loosely coupled and is lovely for two or three steps. But as the flow grows, nobody can see the whole process in one place, and questions like 'why is this order stuck?' mean piecing together events from five services. For order fulfilment, with payment, stock, shipping and refunds when something fails, I'd lean towards orchestration, because the steps and the compensation logic live in one readable place and the order's state is easy to query. I'd still use events for side effects like emails that nobody needs to wait for."
Not knowing either term, or not seeing that choreography makes the overall flow hard to see.
Latency: hops add up, and the slow tail of each hop makes the whole request slow more often.
Availability: it multiplies, so the chain is less available than any single service.
Fixes: call in parallel, cache or copy data locally, move non-urgent work to async, merge chatty services.
"Two things get worse. Latency adds up, because every hop brings its own network time and processing, and each service has occasional slow requests, so the chance that at least one hop is slow grows with every service you add. The chain's slow tail ends up much worse than any single service's. Availability multiplies the other way. If each service has three nines, about nine hours of downtime a year, five in a row give you roughly five times that, close to two days a year. To fix it, I'd first check whether the calls really depend on each other. Independent ones can run in parallel. If a service only needs reference data, like product names, it can keep a local copy updated by events. Anything the user doesn't need to wait for can move to a message. And if two services always call each other, that's a sign they belong together."
Treating the chain's availability as equal to the weakest service, or offering only 'add more servers'.
One entry point: clients call one address; the gateway routes to the right service.
Edge concerns: authentication, rate limits, TLS, request logging in one place.
BFF: a separate edge layer per client type, shaped to what that client needs.
"Without a gateway, a mobile app would need to know the address of every service and call each one directly, which exposes your internals and makes every change a client release. The gateway gives clients one entry point and routes each request to the right service. It's also the natural place for things every request needs, like checking the access token, rate limiting, TLS termination and request logging, so each service doesn't reinvent them. Backend-for-frontend takes it a step further. Instead of one general gateway, you give each kind of client its own edge layer, say one for the mobile app and one for the web app, often owned by the team building that front end. The mobile BFF can combine three service calls into one response and trim fields to save data. The main thing I'd keep out of any gateway is business logic, or it turns into a new monolith."
Putting business rules in the gateway, or saying services can trust any request just because it came through the gateway.
Registry: instances register themselves and are removed when health checks fail.
Client-side: the caller asks the registry for instances and load balances itself.
Server-side: the caller hits a stable name or load balancer that picks the instance.
"In a system with autoscaling and frequent deploys, IP addresses change constantly, so you can't hard-code them. Service discovery solves that with a registry, a list of which instances of each service are running and healthy. Instances register when they start, and they drop out when they stop or fail their health checks. With client-side discovery, the calling service asks the registry for the list of instances and picks one itself, often with a client library doing the load balancing. That saves a hop but puts logic in every client. With server-side discovery, the caller just calls a stable name, and a load balancer or the platform routes it to a healthy instance. Kubernetes works this way: you call a Service by its DNS name and the platform forwards the traffic. Server-side is simpler for callers, so it's what I'd default to unless we need custom routing in the client."
Suggesting hard-coded addresses in config files, or not mentioning health checks.
Closed: calls pass through; failures are counted.
Open: past a failure threshold, calls fail at once for a cool-down period.
Half-open: a few trial calls go through; success closes it, failure opens it again.
"A circuit breaker sits around calls to another service and stops you from hammering something that's already failing. It starts closed, which means calls go through normally while it keeps count of failures and timeouts. When failures pass a threshold, say too many in a rolling window, it trips to open. Now every call fails immediately without touching the network, so my threads aren't stuck waiting and the struggling service gets breathing room to recover. After a cool-down period, it moves to half-open and lets a small number of trial calls through. If they succeed, it closes again and traffic resumes. If they fail, it goes straight back to open. The breaker only helps if I decide what to do when it's open, like returning cached data, a default value or a clear error, so I always pair it with a fallback."
Describing it as a retry mechanism, or not knowing the half-open state.
Retry only temporary failures: connection errors, timeouts, 502, 503, 504.
Never blindly: no retries on errors like 400 or 404, or on non-idempotent calls without a key; a 429 means wait for Retry-After.
Backoff with jitter: waits grow each attempt and are randomised so callers don't sync up.
Limits: a small attempt cap and a maximum delay.
"I'd only retry failures that are likely to go away: connection errors, timeouts, and 502, 503 and 504. A 400 or 404 will fail the same way every time, so retrying just adds load. A 429 means slow down, so I'd honour its Retry-After header rather than use my own schedule. I'd also only retry operations that are safe to repeat, like reads, or writes that carry an idempotency key. For timing, each wait doubles, capped at a maximum, and I pick a random delay between zero and that value. That randomness is the jitter, and it matters: without it, hundreds of clients that failed together all retry at exactly the same moment and knock the service over again. I keep the attempt count small, three or four, and in a chain of services I'd retry at one layer only, otherwise retries multiply at every hop."
import random, time
RETRYABLE = {502, 503, 504} # a 429 waits for Retry-After (not shown)
def call_with_retry(send, max_attempts=4, base=0.2, cap=5.0):
for attempt in range(1, max_attempts + 1):
try:
resp = send()
if resp.status not in RETRYABLE:
return resp
except (ConnectionError, TimeoutError):
if attempt == max_attempts:
raise
if attempt == max_attempts:
return resp # last retryable response; caller decides
delay = min(cap, base * 2 ** (attempt - 1))
time.sleep(random.uniform(0, delay)) # full jitter
Retrying every error, including 400s, with a fixed delay and no attempt limit.
Always set one: some HTTP clients wait forever by default.
Measure: base it on the callee's real latency, a bit above its slow tail.
Budget: the whole request has a deadline; each hop gets what's left, minus its own work.
Retries fit inside: the time for all attempts must fit in the caller's own timeout.
"First, I make sure there is a timeout, because some HTTP clients will wait forever by default, and one hung dependency can then use up every thread. To pick the value, I look at the downstream service's real latency and set the timeout a little above its slow tail, like its 99th percentile, so normal slow requests succeed but a hung one is cut off. Then I think end to end. If the user-facing request has two seconds in total, that's the budget, and each service gets whatever is left minus its own work. The best setups pass the deadline along with the request, which gRPC supports, so a service deep in the chain knows to give up if the caller has already stopped waiting. Retries have to fit inside the budget too. If the outer timeout is shorter than the inner timeouts plus retries, the inner work keeps running for a caller that already gave up."
Leaving library defaults in place, or giving every hop the same large timeout.
Cause: a shared pool, where slow calls hold threads until none are left.
Bulkhead: give each dependency its own limited pool or concurrency limit.
Together with: timeouts to free threads, and fast rejection when a compartment is full.
"That's the classic case for a bulkhead, named after the walls in a ship's hull that stop one leak from flooding the whole ship. The problem is that every endpoint shares one pool of worker threads or connections. When one dependency slows down, requests to it pile up and hold threads while they wait, until there are none left for anything else. A bulkhead gives each dependency its own compartment: a separate small thread pool, connection pool or a concurrency limit like a semaphore. Say calls to the reporting service may use at most twenty concurrent slots. When those are full, extra calls to reporting are rejected at once with a clear error or a fallback, but the rest of the service keeps serving normally. I'd combine it with tight timeouts, so threads are released quickly, and a circuit breaker, so we stop trying altogether when the dependency is clearly down."
Answering 'increase the thread pool size', which only delays the same failure.
Setting: what you configured and what you expected.
Surprise: what actually happened under real load.
Fix: how you changed it and proved the change.
Habit: how you test these settings now.
"At my last company, we added three retries to every call from our orders service to inventory, thinking it would smooth over blips. Then inventory had a slow patch during a sale. Because the API gateway also retried and a middle service retried as well, every user request turned into many calls to inventory, which was already struggling, and it fell over completely. The retries meant to help had multiplied the load. We fixed it by retrying at one layer only, adding jitter, capping retries with a budget so they couldn't exceed a small share of normal traffic, and putting a circuit breaker in front of inventory. Since then, I don't trust these settings until I've tested them. We run a load test that makes a dependency slow on purpose and watch what the callers do. That's caught two similar problems before they reached production."
Blaming the other team's service without examining what your own settings did.
Classify: recommendations are optional; checkout must never depend on them.
Now: a short timeout plus a fallback of nothing or a cached list.
Better: load recommendations separately from the checkout response.
Protect: a circuit breaker so a dead service costs almost nothing.
"The core problem is that an optional feature has become a hard dependency of the most important flow we have. Recommendations are nice to have; checkout is revenue. So the rule I'd set is that checkout must succeed even if recommendations don't. The quick change is to wrap that call with a short timeout and a fallback, so if it fails or is slow, we show an empty section or a cached, generic list. I'd add a circuit breaker too, so when recommendations is fully down, checkout stops waiting on it at all. The better change is to take it out of the checkout request entirely and have the page load recommendations in a separate call, so a failure there just leaves a blank box. Then I'd review checkout's other dependencies with the same question: which ones are truly needed to take the order, and which can fail quietly?"
Only proposing to make the recommendations service more reliable, without removing the hard dependency.
Why: schema changes, deploys and scaling stay independent; no hidden coupling through tables.
Freedom: each service can choose the store that fits its data.
Cost: no joins or single ACID transactions across services; data syncs through APIs and events.
"If two services share tables, they're coupled through the schema even if their code is separate. One team renames a column and another team's service breaks in production. Giving each service its own database means only that service reads and writes its data, and everyone else goes through its API or listens to its events. That lets the team change the schema, deploy and scale on their own schedule, and even pick a different kind of store, like a document store for a catalogue and a relational one for payments. It doesn't have to mean a separate server each time. Separate schemas with separate credentials on a shared server can be enough to start. What I give up is real: no SQL joins across services and no single transaction covering two services' data. So I need patterns like sagas for multi-step updates and local copies or read models for queries that span services."
Saying services can share a database as long as they're careful, or not naming any downside.
Idea: a chain of local transactions, one per service, instead of one big transaction.
Compensation: each step has an action that undoes its effect if a later step fails.
Realities: in-between states are visible, and every step and compensation must be idempotent.
"A saga replaces one distributed transaction with a series of local transactions, one in each service, where every step has a compensating action. For the order, step one: the order service creates the order as 'pending'. Step two: inventory reserves the stock. Step three: payment charges the customer. Step four: delivery books a slot. If delivery fails, we walk back: payment issues a refund, inventory releases the reservation, and the order is marked 'cancelled'. Two things matter. Compensation isn't a database rollback. The charge really happened, so we refund it, and the customer might even see both on their statement. And in-between states are visible to other readers, which is why the order sits in 'pending' until everything succeeds. Because messages can be retried, every step and every compensation must be safe to run twice. I'd also run steps likely to fail early and hard-to-undo ones late, so in a real build I'd only hold the card at step three and capture the money once delivery is booked."
Describing a saga as a rollback, or ignoring that other users can see the in-between state.
How it works: a coordinator asks every participant to prepare, then tells all to commit or abort.
Blocking: participants hold locks while they wait, and a coordinator crash leaves them in doubt.
Coupling: every participant must be up at once, and many brokers and stores don't support it.
Instead: sagas, the outbox pattern and idempotent steps.
"Two-phase commit has a coordinator that asks every participant to prepare, meaning to promise it can commit, and then tells them all to commit or abort. It does give atomicity, but at a price that hurts in microservices. While a participant waits for the final decision, it holds its locks, so other requests to those rows stall. If the coordinator crashes after the prepare phase, participants can be stuck in doubt, unable to commit or abort on their own, until it recovers. It also couples availability: the transaction only succeeds if every service and database is up at the same moment, which undoes the independence we split services for. And in practice, many message brokers, NoSQL stores and third-party APIs don't take part in it at all. So instead, I'd use a saga with compensating steps, make each step idempotent, and use an outbox so a database change and its event can't get out of step."
Saying only that it's slow, without mentioning blocking, in-doubt participants or the availability cost.
Naturally idempotent: prefer 'set status to shipped' over 'add one' where you can.
Dedupe key: a message ID or business key recorded once processed.
Same transaction: record the key and apply the change together, guarded by a unique constraint.
Races: the unique constraint handles two copies processed at once.
"First, I'd make the operation naturally idempotent where I can. Setting an order's status to 'shipped' twice is harmless; adding to a counter twice isn't. For everything else, I use a dedupe table. Each message carries a unique ID, and when the handler runs, it inserts that ID into a processed-messages table with a unique constraint, in the same database transaction as the real change. If the insert finds the ID already there, I skip the work and just acknowledge the message. Doing both in one transaction is the key part: if the change commits, the ID is recorded, and if it rolls back, neither is. The unique constraint also covers the race where two consumers get the same message at once. The second insert waits for the first transaction to finish, and if that one committed, it finds the ID and skips. I'd keep the IDs for longer than the broker could ever redeliver, then clean them up."
BEGIN;
INSERT INTO processed_messages (message_id, processed_at)
VALUES ('msg-7f3a', now())
ON CONFLICT (message_id) DO NOTHING;
-- app checks the row count: 0 means already handled,
-- so it runs ROLLBACK, acks the message and stops here
UPDATE stock SET reserved = reserved + 2 WHERE sku = 'A-100';
COMMIT;
Checking 'have I seen this ID?' in one step and doing the work in a separate, non-atomic step.
API composition: call each service and join in memory, fine for small, simple queries.
Read model: a separate store built from events, shaped for the query (the CQRS idea).
Analytics: heavy reporting often belongs in a warehouse fed from all services.
Trade-off: freshness versus load and complexity.
"It depends on size and freshness. For a small view, say one customer's last ten orders with payment status, API composition is fine: a service or the gateway calls orders, customers and payments and joins the results in memory. That breaks down for a dashboard that filters and sorts across thousands of rows, because you end up pulling huge lists from each service and joining them yourself. For that, I'd build a read model. A small service listens to events like 'order placed', 'customer updated' and 'payment captured', and writes one table already shaped for the dashboard. That's the core idea behind CQRS: separate the model you write to from the one you query. The cost is that the view lags slightly behind and I own another pipeline. If it's really analytics, like monthly trends, I'd stream the events into a data warehouse instead. What I wouldn't do is let the dashboard connect straight into each service's database."
Querying every service's database directly, or looping over one service and calling another once per row.
Understand: why they want it: speed, latency or a missing endpoint.
Risk: hidden coupling to a schema another team owns, and bypassed business rules.
Options: an endpoint, a local copy from events, or a batch call.
Agree: involve the owning team and write down the decision.
"I'd first ask what's driving it. Is the API too slow, or is the endpoint they need missing? Then I'd explain the risk plainly. Reading their tables ties our code to a schema the inventory team can change at any time without telling us, so a harmless rename on their side breaks our checkout in production. We'd also skip any rules inside their service, like how reserved stock is counted, and get wrong numbers. Then I'd offer a way to hit the deadline. If the endpoint is missing, I'd talk to the inventory team today. Often it's a small change they can ship quickly. If latency is the worry, we can call once for a batch of items instead of one by one, or keep a local copy of stock levels updated from their events. If we truly have to cut a corner, I'd want it agreed with the owning team and tracked as debt with a date."
Either agreeing without a word, or refusing flatly without offering a way to meet the deadline.
Trace and spans: one trace per request; each unit of work is a span with a parent.
Propagation: the trace ID travels in headers on every call and every message.
Link to logs: write the trace ID in every log line so you can jump between them.
"When a request enters the system, the first service starts a trace with a unique trace ID. Every piece of work after that, like an HTTP call, a database query or handling a message, becomes a span that records its start, its duration and which span called it. The trick is propagation: each service has to pass the trace context along on every outgoing call, usually in a header. The W3C traceparent header is the common standard, carrying the trace ID and the parent span ID. It has to go on messages too, in the message headers, or the trace breaks at the queue. Tools like OpenTelemetry do most of this automatically for common frameworks and clients. Then a tracing backend puts the spans together into one timeline, so I can see that a slow checkout spent most of its time waiting on one service. I also put the trace ID in every log line."
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
version - trace id (whole request) - parent span id - flags
Confusing tracing with logging, or forgetting that the context must cross async messages too.
Liveness: is this process stuck? Check only the process itself.
Readiness: can this instance take traffic right now? Warm-up, its own pools.
Dependencies: don't fail liveness on them; be careful even with readiness.
Instead: report dependency health through metrics and alerts.
"I split it into two questions. Liveness asks: is this process stuck and needs a restart? That should only check the process itself, like whether it can answer at all. If I put the database in the liveness check, a short database blip makes the platform restart every instance at once, and now I have two outages. Readiness asks: should this instance get traffic right now? That's where I'd check things like whether it has finished warming up. For dependencies I'm careful even there. If a shared database is down and every instance fails readiness, the load balancer has nowhere to send traffic, and users get connection errors instead of a clean, fast error message from the service. So I usually keep the service up, fail those requests quickly with a sensible error or fallback, and track dependency health through metrics and alerts instead."
Making the liveness check call every downstream service and the database.
Situation: the symptom, where it showed up and the impact.
Trail: the traces, metrics and logs that led you across services.
Fix: the immediate fix and the lasting one.
Lesson: what you monitor or design differently now.
"At my last company, checkout started timing out for some users in the evening. Checkout's own metrics looked fine apart from the timeouts, so the first guess was a checkout bug. I pulled traces for the slow requests and saw they all spent most of their time in one span: a call to the pricing service. Pricing's CPU was low, which was confusing, but its database connection pool was maxed out. Its logs showed one query had gone from fast to slow after a release that morning added a filter on a column with no index. Under evening load, those slow queries held every connection, so pricing calls queued and checkout timed out. The quick fix was rolling back pricing. The lasting fix was the index, a shorter timeout from checkout to pricing with a cached-price fallback, and an alert on pool usage. The lesson for me was to follow the trace before guessing."
A story where the fix was restarting things until it stopped, with no evidence of how the cause was found.
Config outside the build: one image, different settings per environment, kept in version control.
Secrets separate: a secrets manager or vault, injected at runtime, never in git or images.
Per service: each service can read only its own secrets.
Rotation and audit: secrets can change without a rebuild, and access is logged.
"The rule I follow is one build, many environments. The same image goes to test and production, and only the configuration changes. Ordinary settings, like feature flags, timeouts and URLs of other services, live in version control per environment and reach the service as environment variables, mounted files or from a central config service, so every change is reviewed and traceable. Secrets are different. Database passwords, API keys and certificates go in a dedicated secrets manager or vault, never in git, never in an image and never in a log. The platform injects them at runtime, and each service's identity only lets it read its own secrets, so if one service is compromised, the attacker doesn't get everyone's keys. I'd also make sure secrets can be rotated without rebuilding, and where the platform supports it, use short-lived credentials that expire on their own."
Putting secrets in the repository or baking them into the container image.
Additive first: add optional fields and new endpoints; never rename or remove in place.
Tolerant readers: callers ignore fields they don't know.
Breaking change: a new version beside the old, track who still calls the old one, then retire it.
Catch it early: consumer-driven contract tests in the pipeline.
"Most changes can be made backward compatible, so that's my first choice. Adding an optional field or a new endpoint doesn't break anyone, as long as callers ignore fields they don't recognise, which I'd make a team rule. Removing or renaming something, or changing what a field means, is breaking. For that I use expand and contract. I add the new field or a new version of the endpoint beside the old one, move callers across, watch the logs or metrics to see who still uses the old one, and only remove it when that drops to zero. For a truly new shape, I'd publish a new version, in the path or a header, and run both for a while. To catch accidents before release, I like consumer-driven contract tests: each calling team records what it relies on, and my pipeline fails if a change would break one of them."
Changing the API in place and asking every team to deploy at the same time.
ClapAssist is an AI interview assistant for Mac and Windows. It listens to the interview on your computer and shows you what to say, in short lines you can read while you talk. Your resume and notes are never stored on our servers. It stays out of screen share on every plan; only you can see it.