Pandas Interview Questions

"A Series is a one-dimensional labelled array. It has values of a single dtype and an index that gives each value a label. A DataFrame is a table, and the easiest way to think of it is a collection of Series that all share the same row index, one Series per column. Each column can have its own dtype, so I can have text, integers and dates side by side. That's also why selection behaves the way it does: df['amount'] gives me a Series, while df[['amount']] with a list gives me a DataFrame with one column. It matters because some methods only exist on one of them, like the .str and .dt accessors, which live on Series."

"Pandas aligns on the index before it does any arithmetic. So when I add two Series, it matches values by label, not by position, and the result has the union of both indexes. Any label that exists in only one of them has nothing to add to, so it comes out as NaN. This often happens after filtering or sorting one of the Series, because the labels no longer line up the way the rows look on screen. If missing labels should count as zero, I use s1.add(s2, fill_value=0). If I really wanted positional addition, I reset both indexes first. The same rule applies when you assign a Series into a DataFrame column: it lines up by index, which surprises people."

"Method chaining means writing a transformation as one expression, where each step returns a new DataFrame and the next method runs on it. I wrap it in parentheses and put one step per line, so it reads like a recipe: load, filter, add columns, group, sort. assign is what makes it work for new columns. If I pass a lambda, it receives the DataFrame as it is at that point in the chain, so I can use columns created earlier in the chain. pipe lets me slot in my own function, like a cleaning step I reuse. The benefit is no half-finished intermediate variables and no inplace calls. The cost is debugging, so I comment out lines from the bottom, or pipe in a small function that prints the shape."

"loc works with labels, so row index labels and column names, and it also takes a boolean mask. iloc works with integer positions, like a list. Slicing is different too: df.loc[2:4] includes label 4, while df.iloc[2:4] stops before position 4. The case where people get burned is after a sort or a filter. The default index keeps its original labels, so after sorting by amount, df.loc[0] is still the row that was originally first, while df.iloc[0] is the row that's now on top. My habit is loc whenever I'm selecting rows by a condition or columns by name, and iloc only when I truly mean position, like taking the first row of each sorted group."

"I build one boolean Series per condition and combine them. For the region I use isin with a list, which is cleaner than chaining equals checks with or. Then I combine that with the amount condition using the ampersand, and each condition goes in its own parentheses, because the ampersand binds more tightly than the greater-than sign. Python's and doesn't work because it tries to turn the whole Series into a single True or False, and pandas raises a ValueError saying the truth value of a Series is ambiguous. For readability with many conditions, query is a nice alternative, and for a not condition I use the tilde."

"The warning comes from chained assignment, like df[df.amount > 100]['flag'] = True. That's two steps: the first selection builds a new object that may or may not share data with df, and the second sets a value on it. If it's a copy, the original never changes, and the warning is the only hint. The fix depends on what I meant. To change the original, I use one loc call with the mask and the column together. If I want a separate subset to work on, I call .copy() when I create it, so it's clearly independent. From pandas 3.0, Copy-on-Write is the default: every selection behaves like a copy, so chained assignment never updates the original, and the old warning is replaced by a ChainedAssignmentError warning. One loc call is correct in every version."

"First I measure: df.isna().sum() gives the count per column, and dividing by the length gives the share. isna catches NaN, None and NaT for dates, and I never test with double equals, because NaN isn't equal to itself. Then I decide per column. If a row is useless without a key field, I drop with dropna and a subset, rather than dropping any row with any gap. If a sensible default exists, I fillna with a dict so each column gets its own value. For time-ordered data, forward fill can make sense. Two things I keep in mind: an integer column with a missing value becomes float unless I use the nullable Int64 dtype, and sum and mean skip missing values by default, so a mean may cover fewer rows than I think."

"First I look before deleting anything. duplicated on customer_id with keep=False marks every row that shares a key, so I can see whether they differ in real fields or are exact copies. Then I need a rule for which row wins, and here it's the newest. I sort by updated_at and call drop_duplicates on customer_id with keep='last', so the latest row survives. An alternative is idxmax on the timestamp per group, which gives the index of the newest row. Afterwards I check that customer_id is unique. Without the subset argument, drop_duplicates only removes rows that match on every column, which usually isn't what the business means by a duplicate customer."

"I'd clean the text first with the .str methods, which work on the whole column at once: strip the spaces and remove the commas. Then pd.to_numeric with errors set to coerce converts everything it can and turns the rest into NaN. The step people skip is the audit. I compare the converted column with the raw one and pull out values that became NaN but weren't empty to start with. That tells me whether it's just 'N/A' placeholders, or something like a currency code or a range I should handle properly. Only then do I replace the column. I don't call astype(int) directly, because it fails on the first bad value and can't hold NaN anyway."

"I start by reading a few rows with nrows to see the real shape, then set the options on purpose. Anything that looks like a number but is really a code, such as an ID or a postal code, I load as a string with dtype, otherwise leading zeros disappear and long IDs can get mangled. Date columns go in parse_dates. If the export uses placeholders like 'N/A' or a dash for missing, I list them in na_values, and if numbers have thousands separators I set thousands to a comma. If the file is wide, usecols loads only what I need, which saves memory and time. And if I get a decode error, I check the encoding and the separator before anything else."

"groupby splits the rows by region, applies a function to each group and combines the results into one table. For several metrics with readable names, I use named aggregation, where each output is written as a name equal to a pair of source column and function. Orders is nunique of order_id, because one order can have several lines, and revenue is the sum of amount. Average order value is then revenue divided by orders, which I add with assign. Taking the mean of amount would give the average line, not the average order, which is an easy slip. Named aggregation also avoids the multi-level column names you get from a dict of lists, and reset_index makes region a normal column again."

"The difference is the shape of what comes back. agg reduces each group to one row, so I get one line per category. transform returns something the same length as the original, aligned to its index, so I can assign it straight back as a new column. That's how I fill missing prices with the median price of their own category, or compute each order's share of its region's total. filter works on whole groups: it keeps every row of groups that pass a test, like customers with at least three orders, and drops the rest. apply can return anything, but it calls Python once per group, so I only use it when agg, transform or filter can't do the job."

"I first aggregate to one row per region and product with the summed revenue, because the raw data has many lines per product. Then I sort by region, and by revenue from highest to lowest, and take groupby region head three. head keeps the first rows of each group in their current order, so the sort does the ranking. If ties matter, like two products with the same revenue in third place, I'd use rank with method dense, descending, within each region, and keep ranks up to three. That can return more than three rows for a region, which is the honest answer, and I'd say so to whoever asked."

"merge is the general SQL-style join. I give it the key columns with on, or left_on and right_on if the names differ, and choose how: inner is the default, and left, right and outer are the others. join is a convenience method on a DataFrame that matches on the index by default and does a left join, so it's handy when both tables are already indexed by the key. concat doesn't match keys at all. It stacks tables, by default one under the other, lining up columns by name, so it's what I use for monthly files with the same layout. If the columns don't match, concat fills the gaps with NaN, which is a quick way to spot a renamed column."

"A left merge keeps every left row, but if the key appears more than once in the right table, each left row is repeated once per match. So the row count grows and any sum over the merged table is inflated. I find it by checking which keys are duplicated in the right table. To guard against it, I pass validate, for example many_to_one when I expect each order to match one customer. pandas then raises an error instead of quietly duplicating. I also assert the row count didn't change, and use indicator=True to count matched and unmatched rows. The opposite surprise, rows not matching, usually comes from keys that differ in dtype, stray spaces or case, so I normalise keys before the merge."

"pandas doesn't have an anti-join method, so there are two common ways. The quickest is isin: take the customers whose ID is not in the orders table's customer_id column, using the tilde to negate. The more explicit way is a left merge with indicator set to True, then keeping rows where the merge column says left_only. I like the merge version when I also want to report how many customers matched, and I deduplicate the order keys first so the merge doesn't repeat customers. Either way, I'd check the key dtypes match, because an integer ID in one table and a text ID in the other would make every customer look like they never ordered."

"pivot only reshapes. It takes one column for the new rows, one for the new columns and one for the values, and every row and column pair must appear exactly once. If the data has two rows for the same region and month, pivot doesn't know which value to put in the cell, so it raises an error about duplicate entries. pivot_table handles that by aggregating: I choose aggfunc, and if I don't, it takes the mean, which is a classic surprise when someone expected totals. It also has fill_value for combinations with no data, and margins to add totals. So for a sales-by-region-and-month report I use pivot_table with aggfunc set to sum."

"I'd use melt. The store column stays as it is, so it goes in id_vars, and every month column gets folded into two new columns: month, holding the old column name, and sales, holding the value. Now each row is one store in one month. That long layout is what most pandas work wants, because I can group by month, filter to a quarter, or feed it to a plotting library without listing column names. To go back to the wide layout for a report, I pivot with store as the index, month as the columns and sales as the values. After melting, I'd usually convert the month text into a real date so it sorts in calendar order, not alphabetically."

"apply with axis=1 looks vectorised but it isn't. It runs a Python function once per row, and builds a small Series for each row to pass in, so on millions of rows most of the time is overhead. Vectorised operations run on whole columns in compiled code. So if I'm labelling orders as high or low, I use np.where with the condition on the whole column. For several bands, np.select takes a list of conditions and a matching list of labels, checked in order, with a default. For text I use the .str methods, for dates the .dt accessor, and for looking values up I use map with a dict or a merge. I keep apply for logic that really can't be expressed on columns, and I measure before and after."

"I start with memory_usage with deep set to True, because without it pandas can undercount text columns, and text is usually where the memory goes. Then I go column by column. A text column with a few distinct values repeated millions of times, like status or country code, becomes the category dtype, which stores each value once plus small integer codes. Numeric columns default to 64-bit, so I downcast with pd.to_numeric, and use float32 only where the lost precision doesn't matter. The biggest win is often not loading data at all: usecols to skip columns I don't need and dtype at read time. I measure again after each change to see what actually helped."

"First I load less. With usecols I read only the user ID and amount, with compact dtypes, and sometimes that alone fits. If not, I read in chunks with chunksize, which gives an iterator of DataFrames. For each chunk I group by user and sum, then add that into a running total with fill_value zero, so users seen in only some chunks still count. The thing to get right is which aggregates combine: sums and counts add up across chunks, but a mean doesn't, so for an average I keep sum and count and divide at the end. If this job runs often, I'd convert the file to Parquet once so later reads can pick columns, or move it to an engine that works out of memory."

"I use pd.to_datetime and give it the format explicitly. Without a format, a value like 03/04 is ambiguous, and I don't want pandas guessing whether that's March or April. With errors set to coerce, anything that doesn't match becomes NaT, the missing value for dates, instead of stopping the whole conversion. Then I check how many NaTs appeared where the text wasn't empty, and look at a few, because a cluster of failures usually means a second format hiding in the data. Once the column is a real datetime, the .dt accessor gives me year, month, day name or a month period for grouping, and date comparisons and sorting work properly."

"resample is groupby for time. It needs a datetime index, or a datetime column passed with on. Resampling to 'W' and summing gives one row per week, and by default those weekly bins end on Sunday and are labelled with that Sunday, which I'd confirm with whoever reads the report. A useful difference from a plain groupby is that resample creates every week in the range, even ones with no sales, so gaps show up as zero for a sum or NaN for a mean instead of just being missing. For weekly totals per store, I use groupby with two keys: the store column and a pd.Grouper on the date with weekly frequency. That gives one row per store per week, though a week with no sales for a store is simply absent, and I reset the index to get a flat table."

"shift moves values down by one row, so the change is the value minus the shifted value. A plain df['sales'].shift() has two problems. It works on row order, so if the data isn't sorted by date I'm comparing random days. And across stores, the first row of store B would pick up the last value of store A. So I sort by store and date, then do groupby store and shift, which restarts the shift for each store and leaves the first day as NaN. The subtler issue is gaps: if a store was closed on a Tuesday, Wednesday gets compared with Monday. If the question really means yesterday, I reindex each store to a full daily range first, or I compare on dates rather than rows."

"At my last company I built a monthly revenue-by-segment report, and the finance lead said one segment looked too high against their system. I reconciled step by step: the raw orders total matched finance, but after I merged in the customer table, the total went up. The customer table had duplicate IDs for a few hundred customers after a system migration, so those customers' orders were counted twice. I fixed the source by keeping the newest customer record, and in the code I added validate set to many_to_one on the merge plus an assertion that the row count and revenue total don't change after joining. Since then I reconcile the headline total against a trusted source before I share any new report."

"At my last company we had a daily script that calculated a fee for each transaction and it took about forty minutes. Instead of guessing, I timed each step on a sample of a hundred thousand rows. Almost all the time was in one apply with axis=1 that looked up a rate from a dictionary and then used if and else on three columns. I replaced the lookup with a merge against a small rates table, and the if and else chain with np.select. The script went from forty minutes to under two, and I compared the old and new outputs row for row before switching it over."

"In one analysis of customer survey data, about a fifth of the responses had no age. The marketing manager wanted results by age group. If I dropped those rows, the overall satisfaction score would shift, because the people skipping the age question were mostly newer customers. If I filled ages with an average, I'd be inventing data. I showed the manager a small table: the score with all responses, and the score with only responses that had an age. Then I explained in one sentence that dropping changes who we're describing. We agreed to keep everyone in the overall score, show the age breakdown with an 'age not given' group, and add a note on the chart."

"First I'd spend five minutes sizing it. I count the rows that became NaT but had text, and look at the most common raw values. Very often it's a second date format from one source, like year first instead of day first. If so, I parse those with their own format, fill the gaps, and check the failure count drops to near zero. That's usually quicker than it sounds. If it's something I can't fix in time, I don't drop those rows quietly, because they'd vanish from the totals. I'd send the report on time with a clear note: how many rows had unreadable dates, which figures they affect, and when I'll send the corrected version."

"Before changing anything, I'd restart the kernel and run it top to bottom, because notebooks often only work thanks to cells run out of order. If it reproduces last month's figure, I save that output as my baseline. Then I find where the rule is applied and pull that logic into a small function, so I can test it on a few hand-made rows. After the change, I rerun and compare against the baseline. I expect differences only in the rows the rule touches, so I check which rows changed and that I can explain each one. pd.testing.assert_frame_equal is handy to prove everything else is identical. Finally, I'd write down the rule change and the difference in the headline number for whoever signs it off."

"I'd hold the rewrite until we've measured. First I'd find where memory peaks. It's often one step, like a merge that multiplies rows or text columns loaded as full strings. Then I'd try the cheap fixes: read only the needed columns with tight dtypes, filter as early as possible, switch the input from CSV to Parquet, and delete big intermediate frames once used. Those often cut memory by a large factor in a day's work. Then I'd look at the growth trend. If the data will outgrow a single machine within months even after that, a distributed rewrite is the right call, and I'd plan it with the current output as the test baseline. If not, a rewrite adds cost and complexity for nothing."