Generative AI Interview Questions

"A discriminative model learns to map an input to an output, like an email to spam or not spam. A generative model learns the distribution of the data well enough to produce new samples from it, such as a paragraph, an image or a voice. Language models and diffusion models are the two families most people mean by generative AI today."

"It predicts one token at a time. Given the prompt, the model outputs probabilities for the next token, one is picked according to the decoding settings, it is added to the sequence, and the process repeats until a stop token or a length limit. That is why generation is sequential, why long outputs cost more, and why an early wrong turn can steer the rest of the answer."

"Pre-training is the expensive part: predict the next token across an enormous corpus, which gives the model language and knowledge but no sense of how to behave. Supervised fine-tuning shows it examples of good responses to instructions. Preference tuning then compares candidate answers, learns what people prefer, and pushes the model toward that, which is what makes it helpful and safer to use."

"During training the model sees images with increasing amounts of noise added and learns to predict that noise. To generate, it starts from pure noise and removes a little noise at a time over many steps, and a text encoder steers each step toward the prompt. Most practical systems do this in a compressed latent space rather than on raw pixels, which is what makes it fast enough to use."

"GANs pit a generator against a discriminator; they sample in one pass and produce sharp results but are hard to train and can collapse onto a few modes. Diffusion models train stably by learning to denoise and give strong diversity and quality, at the cost of many sampling steps, which distillation and better samplers have cut down. Autoregressive models generate a token at a time and dominate text; they are easy to condition and to scale but sequential by nature."

"An autoencoder learns to squeeze an input into a small vector and rebuild it. A variational autoencoder makes that latent space well behaved by encoding to a distribution and regularising it toward a simple prior, so nearby points decode to similar, valid outputs and you can sample new ones. That property is why modern image models run diffusion inside a VAE's latent space instead of on pixels."

"Language models make things up fluently, agree too readily with the user, change answers with small prompt changes, and lose track of content in the middle of a long context. Image models still struggle with hands, readable text and prompts with several distinct parts. Across both, models can regurgitate training data, reproduce biases and give no signal about how sure they are, which is why evaluation and grounding matter."

"For text, perplexity tells you about the model, not about task quality. Overlap metrics like BLEU and ROUGE only work when there is a reference answer, which most real tasks lack. So in practice I build a task-specific eval set, use a model as a judge with a written rubric, and calibrate it against human ratings. For images, FID compares distributions and alignment scores check the prompt was followed, but a human preference study is still what decides a release."

"Tokenization breaks text into sub-word pieces learned from frequency, so common words are one token and rare ones several. It matters because everything, cost, context and speed, is counted in tokens, and some languages and scripts use far more tokens for the same meaning. It also explains odd behaviour like trouble spelling or counting letters, because the model never sees characters directly."

"Attention computes weighted sums over all tokens, and a sum does not care about order, so the model has to be told where each token sits. Early transformers added a fixed or learned position vector to each token embedding. Most current models use rotary position embeddings, which encode relative distance inside the attention computation and extend more gracefully to longer contexts."

"When generating, each new token attends over all previous tokens, and their keys and values do not change, so recomputing them every step would be wasted work. The KV cache stores them per layer, so each step only computes the new token's keys and values. The price is memory, which grows with context length and the number of parallel requests, and that memory is what really limits throughput on a GPU."

"RAG means searching your own documents for the passages that match the question and putting them into the prompt, so the model answers from evidence instead of memory. It is the default because it needs no training, reflects new data immediately, lets you cite sources and enforce permissions, and it is much cheaper than fine-tuning. Its weakness is that the answer is only as good as the retrieval."

"A vision encoder splits the image into patches and produces an embedding for each. A projection layer maps those embeddings into the same space as text tokens, so from the language model's point of view the image is just a run of extra tokens. The model then attends across image and text jointly, and it learned that alignment from large sets of image and caption pairs."

"Zero-shot is just the instruction, few-shot adds a handful of examples so the model copies the pattern and format, and chain-of-thought asks it to work through the steps before answering, which helps on arithmetic and multi-step logic. Examples are the most reliable lever for format; reasoning is the lever for correctness on hard problems, and newer models do it natively."

"I would spend most of the effort on the data: a smaller set of clean, varied examples with a real held-out test split. Then I would use a parameter-efficient method like LoRA on an already instruction-tuned model, with a low learning rate and few epochs to avoid overfitting. If I need more data I generate candidates with a strong model and have people review them, but I never let synthetic data into the test set."

"If a model is trained mostly on outputs of earlier models, rare and unusual examples fade and the model converges toward the average, which people call model collapse. Any errors in the generated data get baked in as truth. Synthetic data is still useful, but it needs verification, a strong share of real data, and records of where each example came from."

"Three concrete things. I check the licence and terms of any model and dataset, and I add checks against long verbatim reproduction. I keep a bias evaluation set that probes outputs across demographics and scenarios and I run it on every change. And I put safety classifiers on input and output, keep red-team prompts in the test suite, and have a process for handling incidents, because some will get through."

"I ask whether the output is naturally text or media, whether some variation is fine, and whether mistakes are cheap and reviewable. Drafting, summarising and classifying messy input are good fits. Computing a tax figure or making an irreversible decision alone is not. My practical test is whether I can write down what a good output looks like clearly enough to evaluate it; if I cannot, the project is not ready."

"Greedy decoding picks the single most probable token each step, which is fast but often repetitive. Beam search keeps a few candidate sequences and picks the best overall; it suits tasks with one right answer like translation, but for open conversation it produces dull text. Sampling draws tokens from the distribution with temperature and top-p controlling the spread, and that is what chat models use by default."