Monitoring system performance in Linux

Using sar, vmstat, nmon, to monitor system performance in Linux!

Table of contents

sar (System Activity Reporter)

If sar is not present in system, install the sysstat package. This will install sar & iostat

• SAR can collect historical data from the sytem, to enable this set ENABLED=”true” in /etc/default/sysstat

After config change, or after enabling sysstat you must restart it: service sysstat restart

• Customization for activity report i.e cron timers are located at /etc/cron.d/sysstat

Viewing sar metrics:

Sar command generates data / log files in /var/log/sa/ or /var/log/sysstat directory

Now that the log file is generated we can run the sar command to view the generated data file

sudo sar -u -f /var/log/sysstat/sa27
04:00:02 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
04:10:01 PM     all     64.38      0.02     28.57      0.03      0.00      7.00
04:20:01 PM     all     68.20      0.01     22.16      0.03      0.00      9.61
04:30:01 PM     all     68.98      0.01     25.12      0.02      0.00      5.87
04:40:01 PM     all     66.78      0.02     27.74      0.02      0.00      5.44
04:50:01 PM     all     62.13      0.02     30.68      0.02      0.00      7.15
05:00:01 PM     all     61.18      0.02     31.15      0.03      0.00      7.63
Average:        all     59.38      0.02     29.97      0.03      0.00     10.61

sar command options

Time delay & line count can be used in all commands: EgL sar -u 1 3. Here 1 second is the refresh interval & 3 lines will be printed.

• sar -u - Show the current usage of the server
• sar -r - Information for memory usage
• sar -S - Information about the swap usage
• sar -b - Show information about I/O information
• sar -d - Show disk statistics
• sar -q - Report system load average & queue length
• sar -p - Show CPU statistics, to check for each core use sar -p <n>

Additional configuration for sysstat can be found at /etc/sysstat/sysstat for Ubuntu & /etc/sysconfig/sysstat for RedHat

vmstat

The vmstat command reports statistics about kernel threads, virtual memory, disks, hypervisor pages, traps, and processor activity.

mhs@maheshrjl:~$ vmstat -a | column -t
procs -memory-  -swap- ---io-   -system-  ---cpu-
r      b         swpd   free      inact     active      si  so  bi  bo  in  cs  us  sy  id  wa  st
0      0         0      5654128   1194192   806940      0   0   2   16  31  71  0   0   99  0   0

Visit RedHat’s website or thegeekdiary to learn more about what these options mean.

vmstat options

• vmstat -s - Memory statistics
• vmstat -d - Disk statistics
• vmstat -f - Forks

iostat

iostat is used for monitoring system input/output statistics for devices and partitions.

mhs@maheshrjl:~$ iostat
Linux 5.10.16.3-microsoft-standard-WSL2 (maheshrjl)     03/27/22        _x86_64_        (8 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.27    0.00    0.41    0.01    0.00   99.31

Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
sda               0.18         0.00        85.43         0.00        225    4198428          0
sdb               1.11        11.67        33.93        16.01     573405    1667324     786564

Visit man7 or ibm’s doc to read more about these options

iostat command Options

• iostat -d - Shows disk statistics
• iostat -p - Shows statistics for all devices, to check for a specific device, use iostat -p sda1
• iostat -m - Shows memory stats in MB
• iostat -x - Display extended statistics.
• iostat -t - Shows time along with stats
• iostat -N - Display stats for LVM

additionally iotop -o can be used which is similar to htop

nmon

Displays local system statistics in interactive mode and records system statistics in recording mode.

Syntax: nmon -f -s13 -c 30

• The -f means you want the data saved to a file and not displayed on the screen.
• The -s13 means you want to capture data every 13 seconds.
• The -c 30 means you want thirty data points or snap shots.

nmon command Options

• m = Memory
• j = Filesystems
• d = Disks
• n = Network
• V = Virtual Memory
• r = Resource
• N = NFS
• k = kernel
• t = Top-processes
• . = only busy disks/procs

Additional documentation for nmon is available at IBM’s page