Data Annotator and AI Trainer Interview Questions

"I got into it in my final year of college. A friend was doing text labelling on a freelance project, and I tried the qualification test out of curiosity. I passed, and my first project was tagging product reviews by topic. What surprised me was how much judgement it took. The easy items take seconds, but the tricky ones make you read the guideline three times. I found I liked that, and my reviewer scores stayed high, so I moved onto image work and then some answer-rating tasks. I want to do more of it because I'm patient and careful with rules, and I'd like to grow toward reviewer or quality roles where I help other annotators stay consistent."

"A model learns from examples, and the labels are the answers it's told are right. If I box a dog and call it a cat a few times, or I'm sloppy about where a box ends, the model learns that mistake as if it were true. And it's not only single errors. If I label a case one way and my teammate labels the same case another way, the model gets mixed signals and becomes unsure exactly where it matters most. Labelled data is also used to test models, so bad labels can make a weak model look good or a good one look weak. That's why I treat consistency with the guideline as the real job, not just speed."

"I've done three main kinds. Image work, mostly bounding boxes and some polygons on street scenes. Text work, which was sentiment and topic tagging on customer reviews, plus a short named-entity project. And for the last few months, rating and comparing chatbot answers. I'm strongest on the text side. My reviewer accuracy there was consistently high, and I was asked to help write examples for the edge-case section of the guideline. I'm less experienced with pixel-level segmentation. I've done practice tasks but not a full project, so I'd expect to be slower there at first and I'd lean on the examples and the first round of feedback."

"On a review-tagging project, the guideline said to mark any mention of delivery as a shipping complaint. But lots of reviews said things like 'arrived on time, love it', which wasn't a complaint at all. I didn't just start ignoring the rule. I kept following it for the clear cases, and for the positive delivery mentions I used the unsure flag the tool gave us and wrote a short note. Then I posted in the project's question channel with three example IDs and asked whether positive mentions should count. The lead agreed it was a gap and added a line to the guideline the next day. Because of my notes, they could find and fix the earlier items instead of relabelling everything blind."

"First I'd search the guideline and the project's question log, because hidden objects are a common case and it might be covered under a different heading. If it truly isn't there, I'd look at how the examples treat partly hidden objects. Projects usually do one of two things: box only the visible part, or estimate the full outline, and often there's an occluded tick box as well. If I still can't tell, I'd label it the way the nearest rule suggests, tick any occlusion flag the tool has, and post the question with the image ID. The key thing is that I apply the same answer to every similar car afterwards, and if the lead's answer differs, I go back and fix the ones I've done."

"For an item that fits none, I'd first check whether the project has an 'other', 'none' or 'unclear' option and read what it's for, because some projects want 'other' used rarely. If there's no such option, I flag the item rather than forcing the closest label, since a forced label teaches the model something false. For an item that fits two, I look for a priority rule. Many guidelines say something like: if it's both a billing and a technical issue, choose the one the customer asked to fix. If the task allows more than one label, I use both. If it's single-label with no rule, I pick using the nearest example, flag it and ask. Then I note the answer so I'm consistent next time."

"In my second week on an image project, a quality check flagged about a third of one batch. The boxes were too loose. I'd been leaving a small margin around each car because I thought it was safer. My first reaction was a bit defensive, but I read every comment and pulled up the examples in the guideline, and they clearly wanted tight boxes touching the object's edges. I wrote the rule on a sticky note, redid the flagged items, and for the next two days I zoomed in on every edge before saving. My next audit came back clean. I also thanked the reviewer and asked one follow-up question about side mirrors, which turned out to be a common miss."

"On a named-entity project, I realised on day three that I'd been including titles like 'Dr.' inside person names, and the guideline said to leave them out. That meant a couple of hundred of my earlier items were probably wrong. I didn't wait to see if the audit would catch it. I messaged the lead that afternoon with exactly what the mistake was, roughly how many items it touched and the date range, and offered to fix them if the tool let me reopen them. They reopened that set, I corrected it the next morning, and the lead used my message to add a clearer example to the guideline. It was a bit embarrassing, but hiding it would have been much worse for the data."

"The first thing I'd do is check myself. I'd re-read the rule and the examples, because the most likely explanation is that I've drifted. If I still think my reading is right, I'd collect three or four concrete examples with item IDs and the exact guideline line, and send them to the lead or the project channel, framed as a question: we seem to be splitting on this, which way is right? I wouldn't message other annotators telling them they're wrong. Once there's an answer, I'd apply it, and offer to relabel my earlier items if I was the one who was off. A split like this is exactly what drags agreement down, so raising it early helps everyone."

"Inter-annotator agreement is how consistently different people label the same items. Plain percent agreement can mislead, because when one label is very common, people agree by chance. Say two of us each mark 90 of 100 messages as not spam. Even labelling at random with those rates, we'd expect to agree on about 82 items. So if we actually agree on 82, it looks strong but it's no better than chance. Cohen's kappa corrects for that. It's observed agreement minus chance agreement, divided by one minus chance agreement, and in that example it comes out at zero. For more than two annotators, teams use Fleiss' kappa or Krippendorff's alpha. And low agreement usually points to an unclear guideline, not just careless people."

"There are a few common ways, and most projects mix them. One is gold or test items: questions with known correct answers hidden in the normal queue, so your accuracy on those is a direct score. Another is review, where a senior annotator or the quality team checks a sample of your work and marks errors, often split into major and minor. Then there's agreement, where the same item goes to several people and your answers are compared with theirs or with the final agreed label. For specific task types there are specific scores, like how well your box overlaps the reference box, or whether your text spans match exactly. Speed is tracked too, but on good projects it's read alongside accuracy, never alone."

"IoU stands for intersection over union. You take your box and the reference box, measure the area where they overlap, and divide by the total area the two boxes cover together. A perfect match scores one, and boxes that don't touch score zero. A reviewer or an automatic check compares your box with a gold box, and if the score is below the project's threshold, it counts as an error. What surprises people is how quickly it drops. Take two 40-pixel squares where one is shifted 10 pixels right and 10 pixels down: they score only about 0.39. That's why a loose box can fail even when it clearly contains the object, and why I zoom in on the edges."

"The longest one I did was about six weeks of tagging short customer messages into roughly twenty intent categories. The risk with that kind of work is that by the third hour you're pattern-matching instead of reading. So I worked in blocks of about fifty minutes with a short break, and I did the hardest categories in the morning when I was sharpest. Every Monday I re-read the guideline's edge-case section, because rules I'd learned in week one had drifted a bit in my head. I also kept a small log of tricky items and how they were resolved, so I didn't have to re-decide them. My audit scores stayed steady the whole project, and went up slightly near the end."

"Near the end of a project, the client moved the delivery date up and we were asked to roughly double our daily count. I tried it for one morning and then checked twenty of my own items, and I'd made mistakes on four, all on the hard edge cases. So I went back to the lead with that number. I said I could raise my speed a lot on the easy items, but for the ambiguous ones I'd rather flag them for a second pass than guess. We agreed on that, I hit a much higher count than before, and the flagged items went to a senior annotator. The batch passed the client's check. I think the lead appreciated a fact instead of a complaint."

"I'd be honest that I've been putting accuracy first while I learn the guideline, and then show I have a plan to close the gap. I'd look at where the time actually goes. Usually it's a few things: re-reading the guideline for the same edge cases, using the mouse where a shortcut exists, or overthinking easy items. So I'd build a short cheat sheet of the rules I keep looking up, learn the main shortcuts, and give myself a time limit on each item before flagging it instead of agonising. I'd also ask whether some slow item types should go to a flag queue. Then I'd tell the lead what speed I expect by the end of next week, and track it daily so they don't have to chase me."

"I moved from text tagging to a video project with two days' notice. The tool was new to me, and so was tracking objects across frames. First I read the whole guideline once without touching the tool, then went through every example slowly. Then I spent an hour on the practice set learning the keyboard shortcuts, because with video the mouse alone is far too slow. I kept a one-page cheat sheet of shortcuts and rules beside me. For the first day I deliberately went slower than the target and asked two questions in the project channel. By the end of the week I was at target speed, and my first audit passed with only minor notes."

"I've used Label Studio for text and image tasks, CVAT for image and video work, and a couple of clients' own in-house tools. Honestly the habits matter more than the tool. For speed, keyboard shortcuts are the biggest win: hotkeys for each label, next item and undo. In video, CVAT's interpolation saves a lot of time. You set the box on keyframes, it fills in the frames between, and you correct it wherever it drifts. For accuracy, I use zoom constantly, I treat any model pre-labels as a starting point and check every one, and I use the flag or comment feature on anything I'm unsure about. I also learn how the tool saves, so I never lose a half-finished item."

"I was rating answers to programming questions, and one answer to a Python question looked excellent. It was well explained, neatly formatted, and it confidently said the code would sort a list of records by date. When I actually ran it, it sorted the dates as plain text, so dates written day first came out in the wrong order. The other answer was plainer but correct. I ranked the plain one higher, and in my written reason I quoted the exact line, the input that broke it and what the output should have been. The reviewer later told me several raters had preferred the polished one. That's the value of subject knowledge: looking right and being right are different things."

"First I'd check the request really is harmful under the project's policy, because not every sensitive-sounding question is. Asking how lock picking works for a novel is different from asking how to break into a neighbour's house. If it genuinely is harmful, the answer that helps is the worse one, however polished it is. So the rude refusal ranks higher. But I wouldn't give it top marks. In my written reason I'd say it's safe but preachy, and that a good answer would decline briefly and respectfully, and offer something useful where it can, like a safer related resource. And if the rubric has separate scores for safety and tone, I'd score them separately rather than blending them in my head. The flip side matters too: refusing a harmless request is a flaw of its own, and I'd mark that down."

"I start with the prompt itself, because a great answer to the wrong question still loses. I note exactly what was asked, including limits like length or format. Then I check each answer in a rough order. Is it correct? Does it follow every instruction? Is it safe and honest, including admitting uncertainty where it should? Only then do I look at clarity, tone and length. A short, correct answer beats a long, confident one with an error in it. I make my choice, and if the scale allows, I say how strong the preference is, since slightly better and much better mean different things to the team. Then I write a specific reason, like 'B ignores the word limit and gets the second date wrong'."

"I break the answer into the specific claims it makes: names, dates, numbers, quotes, citations, and any 'this function does that' statements. That's where invented details hide. General statements are usually fine. Then I check the risky claims against reliable sources the project allows, like official documentation or a well-known reference, not another chatbot. A few patterns make me extra careful: a citation with a very plausible-looking title, an exact figure with no source, a quote from a real person, or a confident answer about something very recent. If I can't verify a claim either way, I say so in my rating instead of guessing. And in my reason I point to the exact sentence that's wrong and what the correct fact is."

"First I make sure I understand what the person actually wants, including anything implied, like the level of detail a beginner would need. If the prompt is ambiguous, I follow the project's rule, which is usually to answer the most likely reading or ask a clarifying question. Then accuracy: every fact, step or line of code has to be right, so I check anything I'm not certain of. After that comes the project's style guide, which often covers tone, length, formatting and when to use lists. The answer should get to the point, without filler openings or needless warnings. Before submitting, I re-read it as if I were the reviewer and check it against every instruction in the prompt, because this answer will be copied as the standard."

"First I'd check whether the project expected this. Safety and moderation projects often do, and they usually have guidance and options like opting out of certain categories or taking extra breaks. I'd keep labelling what I can handle accurately, because rushing through upsetting content is exactly when mistakes happen. If it's getting to me, I'd take a proper break away from the screen and tell my lead honestly rather than going quiet. And if something turned up that the project didn't warn about, especially anything involving a child or a real threat to someone, I'd report it straight away through the channel the project gives us, not just label it and move on."

"I'd follow the project's personal data rules, which usually say one of three things: tag it as personal information, redact it, or flag the item. If the guideline covers it, I do exactly that. If it doesn't, I'd flag the item and ask the lead rather than decide alone. What I wouldn't do is copy it anywhere, screenshot it or look the person up out of curiosity. It stays inside the tool. Privacy laws differ from country to country, and the client may have stricter rules on top, so my habit is to treat any real person's details as confidential by default. If it looked like the data shouldn't have been in the batch at all, I'd mention that too, because the client may need to know."

"No, not unless the project clearly allows a specific tool. There are two reasons. First, the data belongs to the client. Pasting it into an outside service can break the confidentiality agreement I signed, and I don't control where it goes after that. Second, on AI training projects the whole point is human judgement. If I rate a chatbot's answers using another chatbot, the model ends up learning a machine's opinion dressed up as mine, and projects treat that as a serious breach. What I do use, where it's allowed, is ordinary research to check facts, like official documentation or a reliable reference, and I note my sources when the task asks for them."

"I treat it like a job with hours, even when the project is flexible. I set fixed working blocks, usually two longer ones, and I tell my lead when I'll be online so they know when to expect my work. I work at a desk, not the sofa, with notifications off and the guideline open beside the tool. I take a short break every hour, because my accuracy is better after them. At the end of each day I write two lines: what I finished and any open questions. If I'm going to miss hours or a deadline, I say so early, not after the fact. Trust in remote work comes from being predictable, so that's what I aim for."

"It means a few concrete habits. I work only on my own device, with a password and screen lock, and nobody else uses it while I'm working. I keep tasks inside the tool: no screenshots, no downloads, and no copying text into notes or other apps unless the project allows it. I don't talk about the client, the project or the content, including on social media or in job forums, which is also why I describe past projects only in general terms in interviews. I avoid public Wi-Fi for work, and use a VPN if the project provides one. And if I think data has leaked, even through my own mistake, I report it straight away rather than hoping nobody notices."

"A bounding box is the quickest: a rectangle around the object. It's right when the model only needs to find and count things, like cars or people, and the exact shape doesn't matter. A polygon traces the outline with points, so it fits irregular shapes like a sign seen at an angle or a piece of clothing, and it's slower. Pixel-level segmentation labels every pixel. Semantic segmentation gives each pixel a class, like road, sky or car, without separating one car from another. Instance segmentation also tells each individual object apart. It's the most precise and the slowest, and it's used where shape really matters, like medical scans or driving scenes. The guideline decides, but that's the trade-off behind it."

"Using a common label set, I'd tag 'Maria Lopez' as a person, 'Northwind Bank' as an organisation, and 'Chicago' as a location. The time expression depends on the guideline. Some projects tag 'last March' as a date, some tag only 'March', and some don't label dates at all. The main thing I'd be careful about is the boundaries. The span should cover the whole name, so both words of 'Maria Lopez' and all of 'Northwind Bank', but not the word 'in' or any space or punctuation next to it. I'd also check whether the project wants 'Chicago' as a city or a general location, because label sets vary. Consistent boundaries matter as much as the right type."

"The first one I'd mark positive. It's not excited, but 'arrived fast' and 'works fine' are both satisfied, so it's mildly positive if the scale has strength. The second is negative. It starts with 'great', but that's sarcasm: 'another charger dead in a week' tells you they're annoyed, and it hints at a repeated problem. I'd never label on the first word. The third is mixed: a clear positive about the screen and a clear negative about the battery. If the project has a mixed label, I'd use it. If it only allows positive, negative or neutral, I'd check the guideline's rule for mixed reviews, whether that's overall feeling or the stronger complaint, and follow it the same way every time."