AI Engineer Interview Questions

"An ML engineer usually owns training and serving a model on the team's own data. An AI engineer mostly starts from a model someone else trained and owns everything around it: the prompts, the retrieval layer, tool calls, evaluation and the cost and latency budget. There is overlap, but the centre of gravity is the application, not the weights."

"The prompt is split into tokens, each token becomes a vector, and a stack of transformer layers updates those vectors using self-attention, so each position pulls in information from the tokens before it. At the end, the model outputs a probability for every token in its vocabulary, one is chosen by the decoding settings, it is appended to the input, and the loop runs again until a stop token."

"The context window is the token budget for one call, prompt and answer together. It decides almost every architecture choice: whether I can paste a document or need retrieval, how I trim chat history, and what a call costs. And a large window is not a free pass, because recall is worse for content buried in the middle, so I still rank and place context deliberately."

"I start with prompting, because it is the fastest to iterate and to evaluate. If the failures are about missing knowledge, that is a retrieval problem, so I add RAG. If the failures are about behaviour, like format, tone or following a complex procedure, and prompting has plateaued, then fine-tuning makes sense. Fine-tuning is a poor way to teach facts that change, so those two are not substitutes."

"Offline, I load documents, split them into chunks that keep their meaning, embed each chunk and store the vector with metadata like source and date. At query time I embed the question, run a hybrid search over vectors and keywords, filter by permissions, rerank the candidates, and put the best few into the prompt with citations. Then the model answers only from that context, and I log which chunks were used so I can debug misses."

"Fixed-size chunks are easy but they cut through sentences and tables and strip away the heading that gave the paragraph its meaning. I split on document structure first, keep a small overlap so ideas at boundaries survive, and prepend the section title to each chunk so it stands on its own. Then I pick the size by measuring recall on a set of real questions, because the right size depends on the corpus."

"An embedding is a numeric vector for a piece of text where texts with similar meaning end up close together, so I can find relevant passages by distance instead of exact words. I choose the model by running a small retrieval benchmark on our own documents, and I look at language support, vector size for storage, and whether I need to self-host. Embeddings are weak on exact codes and part numbers, so I pair them with keyword search."

"I keep a golden set of real questions with expected answers or a scoring rubric, and it grows every time production surprises us. Deterministic checks cover format and facts we can verify; for open answers I use a model as a judge with a written rubric, and I regularly check the judge against human ratings so I trust it. Every prompt or model change runs the suite in CI, and after launch I watch thumbs-down rate, task completion and cost per task."

"Hallucination is the model stating something false with the same fluency it uses for true things. I attack it in layers: ground the answer in retrieved documents, instruct the model to cite and to say it does not know, keep temperature low for factual work, and then run a check that every claim is supported by the provided sources. Anything that fails the check gets a safer fallback rather than reaching the user."

"Temperature controls how spread out the next-token choice is: near zero the model picks the likeliest token almost every time, higher values let unlikely tokens through. Top-p limits sampling to the most probable tokens that together reach a probability mass, which trims the tail in a different way. Max tokens caps the answer length, and I always check the finish reason so a cut-off answer is not shown as complete."

"I use the model's schema-constrained mode or a tool definition, so the output is generated against the JSON schema rather than hoped for. On my side I still parse and validate every response, and if validation fails I retry once with the validation error in the prompt. I also keep schemas small and use enums, because the fewer ways there are to be wrong, the more reliable it is."

"First I look at the numbers: input tokens, output tokens and time to first token per call, because the fix depends on which one is big. Usually the prompt has grown, so I shorten it, retrieve fewer chunks and cache the fixed prefix. Then I stream so the user sees words early, route simple requests to a smaller model, and cache repeated requests. Each change is checked against the eval suite so speed does not silently cost quality."

"LoRA keeps the base model frozen and trains small low-rank adapters that are added to specific weight matrices, so you train a tiny fraction of the parameters and ship an adapter instead of a full copy. It is worth doing when I have a stable task and a few thousand clean examples, and the goal is to make a smaller model behave like a larger one for that task. If the task is still moving or prompting already passes, the maintenance cost is not justified."

"Prompt injection is text, from the user or from a retrieved page, that tells the model to ignore its instructions. I assume it will sometimes work, so the real defence is limiting blast radius: the model only gets tools and data it truly needs, secrets never sit in the prompt, untrusted content is labelled as data, and anything destructive needs a human click. On top of that I run filters and keep injection cases in the eval suite so regressions show up."

"Prompts are files in the repository with an id that is logged with every request, so I can tie any bad answer back to the exact prompt and model version. A change has to pass the eval suite in CI, then it goes to a small share of traffic while I compare feedback and cost with the previous version. Rollback is a config flip, not a deploy."

"Every call gets a trace: which prompt and model version, the tokens in and out, time to first token, the finish reason, which chunks were retrieved and which tools ran. I sample traffic through the eval graders daily so quality has a number, and I alert on cost, latency tails, provider errors and sudden changes in refusal rate. With that, a bad answer from last week can be replayed and explained."

"A base model only continues text, so if you ask it a question it may keep asking questions. An instruction-tuned model has been trained on request and response pairs, so it does what you ask. A chat model goes one step further with conversation roles and preference training, which is what you normally call through an API. For most applications I want the chat or instruct version."

"We launched a support assistant that answered confidently and was often wrong on billing questions. The traces showed retrieval was returning the right document but a chunk without the pricing table, because chunking split the table from its heading. I changed the chunker to keep tables whole with their heading, added citations, and built an eval set from the failed tickets. Accuracy on that set went from poor to nearly all correct and the complaint tickets stopped."

"I decide with evals and constraints, not preference. If the data cannot leave our environment, or volume is high and steady enough that per-token pricing dominates, self-hosting an open-weights model is worth the operational cost. If the frontier hosted model is clearly better on our eval set and volume is modest, the API wins and the team ships features instead of running GPUs. Either way I put an abstraction layer in so switching later is a config change."

"Quantization stores the weights in lower precision, for example eight or four bits instead of sixteen, so the model fits in less memory and runs faster. The trade is a possible drop in quality, which is usually small for large models and can be noticeable for small ones or for tasks that need precise reasoning. I treat a quantized model as a new model and run the full eval suite before switching."