Linux Interview Questions

"Everything hangs off one root, slash, and other disks get mounted somewhere inside that tree. The folders I use most are /etc, where configuration lives, like the SSH server config or a web server's site files, and /var, which is for data that changes: logs under /var/log, package caches, mail spools, often database files too. Installed programs sit in /usr/bin with their libraries in /usr/lib, while /opt is where a vendor app often drops its whole folder, and /usr/local is for things someone built or installed by hand. User homes are under /home, and root's home is /root. Then there are the virtual ones: /proc and /sys aren't real files on disk, they're the kernel showing live information about processes and hardware, and /dev holds device files like disks."

"First, df -h to see which filesystem is actually full, because /var or /home might be a separate mount. Then I walk down with du, staying on that one filesystem with -x so it doesn't count other mounts, one level deep, sorted by size. I go into the biggest folder and repeat until I find the culprit. It's usually logs under /var/log, a package cache, old core dumps, or an app that writes uploads or temp files and never cleans them. For quick, safe space I'd vacuum the journal, clean the package cache and compress or remove old rotated logs, never a log file an app still has open. If a big file is still in use, I truncate it instead of deleting it. Then I fix the reason it grew, like adding a logrotate rule, and make sure the alert fires well before the disk is completely full next time."

"The usual cause is a file that was deleted while a process still had it open. du walks the directory tree, so it only counts files that still have a name. df asks the filesystem how many blocks are in use, and a deleted file's blocks stay in use until the last process closes it. It often happens when someone deletes a huge log by hand while the app keeps writing. I'd find it with lsof +L1, which lists open files with no links left, and it shows me the process and the size. The clean fix is to restart or reload that process so it lets go. If I can't restart right now, I can truncate the file through the process's file descriptor under /proc, which frees the space at once. Rarer causes are files written into a folder before a disk was mounted over it, and the blocks ext4 keeps back for root."

"First I run lsblk to find the new device, one with no partitions and no mount point, and I check the size so I don't format the wrong disk, because mkfs won't ask twice. Then I create a filesystem, usually ext4 or xfs depending on what the team runs, make a folder like /data and mount it there. To make it survive a reboot I add a line to /etc/fstab, using the UUID from blkid rather than a name like sdb, because device names can change between boots, especially on cloud VMs. The step people skip is testing: I run mount -a, or findmnt --verify, before any reboot. A bad fstab line can drop the server into emergency mode at boot, which on a cloud VM with no easy console is a very bad day. For a data disk the boot doesn't need, I add nofail too."

"I'd write find with the folder as the start point, -type f so only files match, a -name pattern for the log files, and -mtime +14. find counts age in whole days and rounds down, so +14 really means fifteen days or more. The first run just prints, and I read the list and check the count. Then I add -delete as the last thing on the line. Order matters, because find reads its expression left to right, so a -delete placed before the tests would delete everything it walks into. If I need a different action, like listing sizes or compressing, I use -exec with a plus so it batches the files. For names with spaces, I stick to -delete or -exec rather than piping plain output into xargs, or I use -print0 with xargs -0. Once it's correct I'd schedule it, though for logs a logrotate rule is usually the cleaner answer."

"The first dash tells me it's a regular file. Then it splits into three sets of three. The owner has rwx, so they can read it, change it and run it. The group has r-x, so group members can read and run it but not edit it. Everyone else gets nothing. In numbers, read is 4, write is 2 and execute is 1, so the owner is 7, the group is 5 and others are 0, which is chmod 750. I could also do it symbolically, like chmod o-rwx, which is handy when I only want to change one part. Modes only work alongside ownership, so I'd check the owner and group too and fix them with something like chown appuser:appgroup on the file. On Linux only root can give a file to another user."

"Setuid on an executable means it runs with the rights of the file's owner rather than the person running it. The classic example is passwd: it's owned by root with setuid, so an ordinary user can run it and it can still update the shadow file that only root can write. In ls you see an s where the owner's x would be. Setgid does the same for the group, but I use it more on directories: set it on a shared team folder and new files inherit that folder's group, so the whole team can keep working on them. The sticky bit is the t you see on /tmp. Everyone can write there, but you can only delete or rename your own files. On the security side, setuid-root programs are a common privilege escalation path, so in a hardening review I'd list them and question anything unexpected."

"I'd create the account with useradd -m so they get a home directory and -s to give them bash, or use adduser on a Debian-style box because it walks you through it. Since we log in with keys, I'd put their public key in their authorized_keys rather than set a password. For groups I use usermod -aG with the group name. The -a is what people forget: without it, -G replaces all their extra groups instead of adding one. For sudo, the simple route is the admin group, which is sudo on Debian and Ubuntu and wheel on Red Hat family systems. If they only need a few commands, I write a small rule in /etc/sudoers.d with visudo, because it checks the syntax before saving, and a broken sudoers file can stop everyone using sudo. Then I run id on the user to confirm."

"I generate a key pair on my laptop, these days ed25519, with a passphrase. The public half goes into authorized_keys in the .ssh folder of my account on the server, usually with ssh-copy-id. When I connect, the server checks my public key is listed and asks me to prove I hold the matching private key. My client signs some data with it and the server checks the signature with the public key, so the private key never leaves my machine. If it doesn't work, it's usually permissions: sshd ignores authorized_keys if the home folder, .ssh or the file itself is writable by others. Once key login works, I set PasswordAuthentication no and PermitRootLogin no in sshd_config, test the file with sshd -t, and reload. The habit that matters is keeping my current session open and testing a fresh login from a second terminal before closing anything."

"I'd treat it as something to fix today, not a style point. 777 means every user on the box, and any compromised process, can rewrite the app's code and config and read whatever secrets are in there. First I'd find the real cause. I'd check which user the service runs as, from the unit file or ps, and look at the original error, which usually names the file. Often the deploy was done as root, so the files ended up owned by root, or one upload folder needed group write. Then I'd set the right owner and group with chown and put back sensible modes, like 750 for folders, 640 for files and 600 for secrets, with write access only where the app writes. I'd talk to the teammate one to one, show how I found the actual file, and suggest we add the ownership step to the deploy script."

"I wouldn't just say no, because an outage is real pressure and blocking them makes it worse. I'd ask what they're trying to do. Usually it's reading logs, checking a config or restarting their service. Reading logs can be covered by adding them to the group that can read the journal or the app's log folder, and a restart can be one sudo rule for that one unit. Often the fastest option is getting on a call and running the commands together while they watch. If they truly need broad access, I'd use our break-glass process: approved, time-limited, with sudo logging, and removed afterwards. Then, after the incident, I'd look at why they needed the server at all. If the logs were in a central place they could search, most of these requests would go away."

"For a live view I open top, or htop if it's installed. At the top I read the load average, then the CPU line: us is time in application code, sy is kernel time, id is idle, wa is time waiting on disk, and st is time the hypervisor gave to other guests, which matters on cloud VMs. In the process list I press shift-M to sort by memory or shift-P for CPU. For memory I look at RES, the resident memory the process really holds, not VIRT, which counts all the address space it has mapped and is often huge without meaning much. A process showing well over a hundred in the CPU column is simply using more than one core. When I want a snapshot for a ticket, I use ps with the columns I care about and a sort, keeping in mind that its CPU figure is an average over the life of the process, not this second."

"Plain kill with no number sends SIGTERM, which is signal 15, so kill and kill -15 do the same thing. SIGTERM asks the process to stop, and a well-written program catches it, finishes the request it's on, closes files and database connections, removes its lock file and exits. SIGKILL, signal 9, is different: the process never gets to react, the kernel just removes it. That's why it's a last resort. You can be left with half-written files, stale lock files, or a database that has to recover on the next start. So my habit is to send TERM, wait some seconds, check with ps, and only then use -9. Many daemons also treat SIGHUP as reload your config, so for those a HUP is gentler than a restart. systemd follows the same order when it stops a service: TERM first, KILL after a timeout."

"When a process dies without a word in its own log, I suspect something outside killed it, and the OOM killer is top of the list. The kernel logs every kill, so I search dmesg -T or journalctl -k for out of memory or killed process, which names the victim and how much memory it held. Other clues: an exit code of 137, which is 128 plus 9, meaning SIGKILL, and systemctl status often shows the unit failed with the result oom-kill. Then I work out the scope. If the whole machine ran out, I look at what grew, using memory graphs if we have them. If the service or container has a memory limit through its cgroup, it can be killed while the host still has plenty free, so I compare the limit with real usage. The fix is finding the leak or the spike, then setting honest limits and sizing the machine."

"I'd write a unit file under /etc/systemd/system, say myapp.service. In the Service section I set ExecStart to the full path of the binary and its arguments, because systemd doesn't use my shell's PATH. I set User to a dedicated account that owns only the app's folder, plus WorkingDirectory, and I load settings from an EnvironmentFile with tight permissions rather than putting secrets in the unit itself. Restart=on-failure with a short RestartSec gives crash recovery without a tight loop. In the Unit section, After and Wants on network-online.target make it wait for the network. The Install section with WantedBy=multi-user.target is what lets enable start it at boot. Then systemctl daemon-reload so systemd reads the new file, systemctl enable --now to start it and turn it on for boot, and I check status and journalctl for the unit."

"First I run systemctl status on the service. It tells me whether it failed or is stuck restarting, shows the exit code, and prints the last few lines of its log, which is often enough. If not, I use journalctl -u with the unit name and -e to jump to the end, or --since with something like ten minutes ago to cut the noise, and -b to stay within the current boot. After a config change, the message is usually a syntax error on a given line, a port already in use, or permission denied on a file. Many daemons have a config test mode, like nginx -t or sshd -t, so I run that before trying again. If I still can't see it, I diff the config against the backup or the last version in git, and if the service matters, I roll back first and debug after."

"On a modern server there are two layers. systemd's journal collects the output of every service, and I read it with journalctl. Then there's /var/log with plain text files, often written by rsyslog: general messages in syslog on Debian and Ubuntu or messages on Red Hat family systems, logins and sudo in auth.log or secure, plus folders for things like nginx. To stop growth, logrotate runs daily with rules in /etc/logrotate.d: how many copies to keep, compress the old ones, rotate by size or age. The trap is an app writing its own log files with no rule, so I add one for each. The journal has separate limits, like SystemMaxUse in journald.conf, and journalctl --vacuum-size frees space straight away. If an app keeps its log file open, I use copytruncate or a reload after rotating, or it keeps writing to the old file."

"I use ss -tulpn. The letters mean TCP, UDP, listening sockets only, show the process, and numeric ports instead of service names. I run it with sudo, because otherwise I can't see the owners of processes that belong to other users. The column I read carefully is the local address. If a service is on 127.0.0.1, it's only reachable from the machine itself, which is a very common reason an app answers curl on the box but not from outside. If it's 0.0.0.0 or the IPv6 equivalent, it's listening on every interface. When I only care about one port, like an 'address already in use' error on 8080, I filter for it or run lsof -i on that port to get the PID, and I check what that process is before stopping anything. netstat does the same job, but it comes from an older package many distros no longer install by default."

"I work from the inside out. First, is the service up? systemctl status and its journal. Then, is it listening, and where? ss -tlnp gives me the port and address, and if it's bound to 127.0.0.1 only, that's the answer. Next I curl it locally with -v, first on localhost and then on the server's own IP, so I see whether the app answers and with what status. If that works, the host firewall is next: ufw status, firewall-cmd --list-all or the nftables ruleset, depending on the distro. After that it's outside the server: the cloud security group or network rules, the load balancer's health checks, and whether the DNS name actually points here, which I check with dig. ip addr and ip route show the addresses and default gateway if I suspect the network setup itself. Each step rules out a whole layer."

"dig is a DNS tool, so it asks a DNS server directly and skips everything else. Most programs don't do that. They call the system resolver, and on Linux that follows /etc/nsswitch.conf, which normally says check /etc/hosts first and DNS after. So the first thing I'd look for is a stale line in /etc/hosts, often left over from a migration. I'd run getent hosts on the name, because that goes through the same path as the app and shows what it really sees. After that, caching. Many systems run systemd-resolved as a local stub, which is why resolv.conf may point at 127.0.0.53, and it can hold an answer until its TTL runs out. Some language runtimes also cache lookups inside the process, sometimes until a restart, so a long-running app can keep an old address even after everything else is fixed."

"I'd start with grep -rn on the folder to see every match with its file and line number, or -l if I just want the file list. I read that list before changing anything, because sometimes the old name appears somewhere it should stay, like a comment or a different service. Then I preview the sed replacement without -i, so it just prints the result. When I'm happy, I run sed -i with a backup suffix like .bak, which keeps the originals. Two details matter. In a regex a dot means any character, so I escape the dots in a hostname. And if the text contains slashes, I use a different delimiter, like a pipe, instead of escaping every slash. Afterwards I grep again, skipping the .bak files, to make sure nothing's left, run the service's config test, and only then remove the backups."

"In the usual access log format the client IP is the first field, so I have awk print field one. Then I sort, because uniq -c only counts identical lines that sit next to each other, so without sorting first the counts come out broken. uniq -c puts a count in front of each IP, sort -rn puts the biggest counts on top, comparing them as numbers, and head gives me ten. On a very large file the first sort is the slow part, so I'd let awk do the counting in a single pass with an associative array keyed by IP, and only sort the small result. And if the server sits behind a load balancer or proxy, I'd check which field really holds the client address, because the first field might just be the balancer."

"The cron line is 30, 2, star, star, 1 to 5, then the full path to the script. The five fields are minute, hour, day of month, month and day of week, and 1 to 5 means Monday to Friday. I add it with crontab -e as the user who should run it, or as a file in /etc/cron.d for system jobs, which also takes a user field. Scripts break under cron because it doesn't load your shell profile. The PATH is minimal, so commands your terminal finds aren't found, variables from your bashrc aren't there, the shell is plain sh unless the script says otherwise, and the working directory is the user's home folder, so relative paths break. So I use full paths, set PATH at the top of the script, and redirect stdout and stderr to a log. For new jobs I often use a systemd timer instead, because output lands in the journal and Persistent=true runs a job that was missed while the server was off."

"The first line after the shebang is set -euo pipefail. The e stops the script when a command fails, the u turns a mistyped variable name into an error instead of an empty string, and pipefail makes a pipeline fail if any stage fails, not just the last one. Then I quote every variable. The classic disaster is an rm on a path built from an empty variable, which ends up pointing much higher up the tree than intended. I check the argument before doing anything, and send errors to stderr with a non-zero exit so cron or CI sees the failure. I write the archive to a temp file from mktemp and only rename it when tar succeeds, with a trap on EXIT that removes the temp file if the script dies halfway. And I run shellcheck, which catches most quoting mistakes before they bite."

"In my previous role we set up new users on about twenty servers by hand, copying keys and adding groups, and someone was always missing a server or a group. I wrote a script that read a small list of users and their public keys, created the accounts, set the groups and installed the keys, and it could run again without changing anything that was already right. I gave it a dry-run flag that only printed what it would do, tried it on one test server, then two real ones, before the rest. It logged every change with the time and the server. Later the team moved the same logic into a configuration management tool, which was the right long-term home. Onboarding went from half a day of copy and paste to a few minutes, and our access reviews stopped finding strays."

"Both families have a low-level tool that installs a single package file, dpkg on Debian and Ubuntu and rpm on Red Hat, Rocky or Fedora, and a higher-level tool on top that talks to repositories and resolves dependencies: apt on one side, dnf on the other, with yum on older releases. A common mix-up with apt is update versus upgrade. update only refreshes the package lists, upgrade actually installs newer versions. To find which package owns a file, say a binary or a config file, I run dpkg -S with the path, or rpm -qf on the Red Hat side. That helps when a config gets overwritten, or when I need to know whose changelog to read. And if one upgrade would break something, I can hold that package, for example with apt-mark hold, and leave a note saying why so it isn't forgotten."

"The firmware, BIOS or more often UEFI now, runs its checks and finds a bootloader, usually GRUB. GRUB loads the chosen kernel along with an initramfs, a small temporary root filesystem holding the drivers and tools needed to reach the real root disk, like storage drivers, LVM or disk encryption. The kernel sets up the hardware and runs the initramfs, which finds and mounts the real root filesystem and switches over to it. Then PID 1 starts, which on most distros today is systemd. systemd then works through its units in dependency order, mounting what's in fstab, bringing up the network and starting services, until it reaches the default target, usually multi-user on a server. Knowing the stages tells me where to look. Stuck at a GRUB prompt means the bootloader or its config. A kernel panic about the root filesystem often points at the initramfs. Landing in emergency mode is usually a bad fstab line or a failed mount."

"First I'd read the advisory and work out how exposed we are. A remotely exploitable flaw on an internet-facing box is far more urgent than a local one on an internal server with few users. Then I'd agree the quietest window with the service owner and warn users ahead of time. Before patching I'd take a snapshot or check the backup, and confirm the previous kernel is still installed so GRUB can boot it if the new one misbehaves. In the window I install the update, reboot, check uname -r shows the new kernel, and test the service with a real request, not just a running process. If no window was possible soon, I'd look at interim steps like tighter firewall rules. Afterwards I'd raise the bigger issue: one server that can never go down is a risk in itself, and a second node would make the next patch routine."

"I start with uptime, because the three load averages tell me whether it's getting worse or calming down. Then dmesg with readable times, or journalctl -k, to catch the kernel complaining: OOM kills, disk errors, network drops. Next top: which process is busy, and the CPU line, where high wa means processes waiting on disk and high st on a VM means the host is short of CPU. free -h shows available memory, and vmstat 1 for a few seconds shows swapping in the si and so columns. For disk, df -h for space and iostat -x, if sysstat is installed, for a device that's nearly always busy. ss -s gives a quick count of connections. Then I ask what changed: a deploy, a new cron job, a traffic spike. Most slow servers come down to CPU, memory, disk or a slow dependency like the database, and this order finds which."

"At my last company our API started timing out every night around two. CPU looked fine in top, but the wa number was high, and iostat showed the data disk fully busy. iotop pointed at a backup job, a tar of the whole uploads folder, that someone had moved from Sunday to every night. It was hammering the same disk the database used. The quick fix was moving the job back to Sunday morning while we worked on a better one. The real fix was switching the backup to a snapshot of the volume instead of reading every file, which took that load off the disk. Afterwards I added an alert on disk wait time, and a line in our runbook: when a slowdown happens at the same time every day, check the scheduled jobs first."

"Early on I was sitting in a root shell on a staging server, fixing ownership for an app, and ran a recursive chown on the current folder, thinking I was in the app's directory. I'd actually changed into /etc a minute earlier. sudo stopped working for everyone, because it refuses to run when its config isn't owned by root, and a couple of services failed on their next restart. I told my lead straight away. I still had that root shell, and because it was a Red Hat family box, rpm's verify mode listed every packaged file whose owner had changed, so I put those back and compared the rest against a second staging server. It took a couple of hours and it was only staging, but it changed how I work. I don't sit in a root shell any more, I use full paths instead of a dot for anything recursive, and my prompt shows the hostname and folder."