Admin troubles

Various Tricky Situations and Admin Features

• du path shows that 70% is taken, while df -s path shows that there is 0% free. Why?
  • Option 1: Some process is writing to a deleted file (usually syslog). To find out who it is, you can run lsof | grep deleted or fuser path or guess it. The problem is solved by restarting the necessary service or sending a signal that forces the process to reopen the file.
  • Option 2: In some directory under path (usually /), something is mounted, e.g. /mnt. And everything that was in /mnt is now inaccessible, but the space is still occupied. A typical example is making backups on a network disk, at some point the disk is not mounted, but the backup is made (and eats everything). After a reboot, the disk is mounted, but what was on the disk in this point is now hidden.
  • Option 3: The inode count is exhausted, check df -i.
• Changed the rights using chmod, now it cannot be executed, what to do?
  • /lib/ld-linux.so[.2] /bin/chmod a+x /bin/chmod - the simplest option
  • cp /bin/ls /bin/ls.orig; cat /bin/chmod >/bin/ls; ls a+x /bin/chmod, and now return ls back
  • cp --attributes-only /bin/ls /bin/chmod
  • getfacl /bin/ls | setfacl --setfile =- /bin/chmod
• no fork is possible, but there is a console with bash. Need to reboot the server.
  • exec /sbin/reboot [-f]
  • echo 'b' > /proc/sysrq-trigger (echo - builtin command bash!)
• input ls -1, but get the same output as ls -l.
  • it’s alias ls='ls -l', you can make unalias, or run \ls -1
  • as an alternative - this is a function, instead of unalias, do unset -f
• how to make that instead of true is executed false?
  • rename, create a directory and a link in it with name true and link to false, add to PATH in the beginning
  • make alias
  • execute hash -p /usr/bin/false true. Remove - hash -d true
• I deleted file, is there a way to recover? In which cases?
  • if the file is open, you can copy it from the file descriptor in /proc/PID/fd/FD to /bin
  • turn off the computer (or turn off the file system) and find the file with command testdisk or similar
  • recover from backup

Useful Programs

• top / htop / iotop - who is consuming the most resources
• mpstat -P ALL 1 - CPU statistics
• pidstat 1 - process statistics, as simple as top but easier to analyze
• iostat -xz 1 - disk I/O statistics
• iftop - who is transferring traffic (and where from/where to)
• vmstat - almost everything in one place.
• sar - a lot of statistics on everything. Important to specify the interval, otherwise it will produce empty output. The most useful:
  • -n DEV = network device statistics
  • -n TCP,ETCP = similar to network device statistics by TCP/ETCP
  • -d = block devices
  • --dev... = list of devices, not all
  • -F = by file systems (only block devices! snap/proc and similar are ignored atomically)
  • -h = display in megabytes/gigabytes
  • -i file/-o file = read/write to binary file
  • interval and optionally number of samples - at the end. For example sar -w 1 4 = 4 samples of CPU with interval 1 sec.

Simple question for admin

• Where are configs stored? Logs?
• Why is sudo needed? Who can use it? How can it be limited?
• What are the options to write a string to a file?
• What is the package manager? How are packages structured?
• What is MAC and IP? What is the difference between TCP, UDP, and ICMP?
• How does traceroute work?
• What is NAT and Proxy?
• What is DNS? Is it needed in a local network (local zone)?
• What is DHCP?
• What is load average, why are there three?
• What is the difference between HTTP and HTTPS?
• How do you find the PID of a process on a server (for example, nginx)?
• What does Ctrl-C do in the terminal? How does it work?
• How can you prevent a user from logging in? What are the options, advantages, and disadvantages?
• What is an inode? How can you find the inode for a file?
• What is a link? What is the difference between a symbolic and hard link? What happens if you delete a file that a link points to?
• How do you search for files, check processes, and view services?
• How do you start/stop a service? How can you prevent a service from starting at boot? Can it be completely prevented?
• Why use the export command?
• What is an initrd image? Why is it needed?
• What is the file /etc/shadow?- How to force the user to change their password during registration?
• How to find out how the last command ended – was it successful or not?
• How to find out which kernel modules are loaded in the Linux OS?

sysrq - cheatsheet

How to disable unnecessary

auditctl -a never,task - disable audit of system calls nospectre_v1 nospectre_v2 pti=off mds=off tsx_async_abort=off performance=good mitigations=off - disable security mitigations docker run --security-opt seccomp=unconfined... - disable syscall filter modprobe -rv ip_tables - remove iptables

Disable IRQ balancing, then traffic will always be processed in the same queues

systemctl stop irqbalance.service

IRQS=($(grep eth0 /proc/interrupts | awk '{print $1}' | tr -d ':'))
for i in "${!IRQS[@]}"; do echo $i > /proc/irq/${IRQS[i]}/smp_affinity_list; done;

Miscelaneous

systemd-cgld - a decent replacement for ps when you need to see the cgroup breakdown. You can find suspicious processes.

Simple interview questions

• What happens when you try to curl to a website?
• What happens when you type ‘ls' in the console?
• What is the Load average?
• What is the interrupt in Linux? What types of interrupts do you know (sync/async = internal/external, masked/non-masked)?
• What is the /proc filesystem?
• What is the Linux inodes?