Sysstat is a package of performance monitoring tools for Linux operating systems. It provides a collection of utilities to collect, process and analyze system utilization data over time. The sysstat package includes tools like sar, iostat, mpstat, and pidstat, which provide various system performance metrics such as CPU, memory, disk and network usage, process statistics, and more. The data collected by sysstat can be used to identify performance bottlenecks, troubleshoot system issues, and make informed decisions about system resource usage.
This article shows how to install sysstat and to check the system.
Overview
The SYSSTAT software application is composed of several utilities. Each utility has a specific function:
- iostat reports CPU statistics and input/output statistics for devices, partitions and network filesystems.
- mpstat reports individual or combined processor related statistics.
- pidstat reports statistics for Linux tasks (processes) : I/O, CPU, memory, etc.
- sar collects, reports and saves system activity information (CPU, memory, disks, interrupts, network interfaces, TTY, kernel tables, NFS, sockets etc.)
- sadc is the system activity data collector, used as a backend for sar.
- sa1 collects and stores binary data in the system activity daily data file. It is a front end to sadc designed to be run from cron.
- sa2 writes a summarized daily activity report. It is a front end to sar designed to be run from cron.
- sadf displays data collected by sar in multiple formats (CSV, XML, etc.) This is useful to load performance data into a database, or import them in a spreadsheet to make graphs.
Let’s start by installing Sysstat
yum -y install sysstat
Enable Sysstat
systemctl enable sysstat
Check the cron job
cat /etc/cron.d/sysstat
Output
cat /etc/cron.d/sysstat
Output:
# The first element of the path is a directory where the debian-sa1
# script is located
PATH=/usr/lib/sysstat:/usr/sbin:/usr/sbin:/usr/bin:/sbin:/bin
# Activity reports every 10 minutes everyday
5-55/10 * * * * root command -v debian-sa1 > /dev/null && debian-sa1 1 1
# Additional run at 23:59 to rotate the statistics file
59 23 * * * root command -v debian-sa1 > /dev/null && debian-sa1 60 2
Enable SADC
nano /etc/default/sysstat
Output
Change ENABLED=”false” to ENABLED=”true”. Then start sysstat.
#
# Default settings for /etc/init.d/sysstat, /etc/cron.d/sysstat
# and /etc/cron.daily/sysstat files
#
# Should sadc collect system activity informations? Valid values
# are "true" and "false". Please do not put other values, they
# will be overwritten by debconf!
ENABLED="true"
systemctl restart sysstat
Usage Examples
NOTE: Sysstat takes time to gather data. If you see no real output from the below commands, give the system time to generate data.
View RAM usage
sar -r
Output
Linux 5.10.0-21-amd64 (dns.ipgw.io) 03/13/2023 _x86_64_ (1 CPU)
06:58:35 PM LINUX RESTART (1 CPU)
07:00:09 PM kbmemfree kbavail kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
07:10:09 PM 1277964 1627632 240072 11.82 23992 438200 1097844 36.24 161488 493468 132
07:20:01 PM 1277696 1627448 240244 11.83 24072 438204 1098864 36.28 161572 494000 156
07:30:09 PM 1277428 1627272 240424 11.84 24160 438204 1098016 36.25 161656 493696 176
Average: 1277696 1627451 240247 11.83 24075 438203 1098241 36.26 161572 493721 155
View Swap usage
sar -S
Output
Linux 5.10.0-21-amd64 (dns.ipgw.io) 03/13/2023 _x86_64_ (1 CPU)
06:58:35 PM LINUX RESTART (1 CPU)
07:00:09 PM kbswpfree kbswpused %swpused kbswpcad %swpcad
07:10:09 PM 998396 0 0.00 0 0.00
07:20:01 PM 998396 0 0.00 0 0.00
07:30:09 PM 998396 0 0.00 0 0.00
Average: 998396 0 0.00 0 0.00
It’s possible to output current or past status of system with “sar” command like follows. Output the past CPU usage from the log file.
Find the current file:
ls -l /var/log/sysstat/
total 8
-rw-r--r-- 1 root root 4792 Mar 13 19:10 sa13
The log file is “sa13.” Run the following:
sar -u -f /var/log/sysstat/sa13
Output:
Linux 5.10.0-21-amd64 (dns.ipgw.io) 03/13/2023 _x86_64_ (1 CPU)
06:58:35 PM LINUX RESTART (1 CPU)
07:00:09 PM CPU %user %nice %system %iowait %steal %idle
07:10:09 PM all 0.03 0.00 0.02 0.03 0.01 99.91
07:20:01 PM all 0.02 0.00 0.02 0.02 0.02 99.93
07:30:09 PM all 0.02 0.00 0.02 0.02 0.01 99.93
Average: all 0.02 0.00 0.02 0.02 0.01 99.92
Output the current CPU usage for 3 times every second.
sar -u 1 3
Output
Linux 2.6.32-358.6.2.el6.x86_64 (dlp.server.world) 06/24/2013 _x86_64_ (2 CPU)
05:29:12 PM CPU %user %nice %system %iowait %steal %idle
05:29:13 PM all 0.00 0.00 0.00 0.00 0.00 100.00
05:29:14 PM all 0.00 0.00 0.00 0.00 0.50 99.50
05:29:15 PM all 0.50 0.00 0.50 0.00 0.00 99.00
Average: all 0.17 0.00 0.17 0.00 0.17 99.50
Output the current Disk usage for 3 times every second.
sar -b 1 3
Output:
Linux 5.10.0-21-amd64 (dns.ipgw.io) 03/13/2023 _x86_64_ (1 CPU)
07:37:20 PM tps rtps wtps dtps bread/s bwrtn/s bdscd/s
07:37:21 PM 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:37:22 PM 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:37:23 PM 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Output the current Memory usage for 3 times every second.
sar -r 1 3
Output
Linux 5.10.0-21-amd64 (dns.ipgw.io) 03/13/2023 _x86_64_ (1 CPU)
07:37:45 PM kbmemfree kbavail kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
07:37:46 PM 1277404 1627348 240340 11.84 24224 438232 1098380 36.26 161744 493800 0
07:37:47 PM 1277404 1627348 240340 11.84 24224 438232 1098380 36.26 161744 493800 0
07:37:48 PM 1277404 1627348 240340 11.84 24224 438232 1098380 36.26 161744 493800 0
Average: 1277404 1627348 240340 11.84 24224 438232 1098380 36.26 161744 493800 0
Output the current send/receive packets for 3 times every second.
sar -n DEV 1 3
Output:
Linux 5.10.0-21-amd64 (dns.ipgw.io) 03/13/2023 _x86_64_ (1 CPU)
07:19:58 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
07:19:59 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:19:59 PM enp1s0 2.00 0.00 0.11 0.00 0.00 0.00 0.00 0.00
07:19:59 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
07:20:00 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:20:00 PM enp1s0 1.00 1.00 0.06 0.62 0.00 0.00 0.00 0.00
07:20:00 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
07:20:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:20:01 PM enp1s0 2.00 1.00 0.11 0.62 0.00 0.00 0.00 0.00
Average: IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
Average: lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: enp1s0 1.67 0.67 0.09 0.41 0.00 0.00 0.00 0.00
Output the current paging usage for 3 times every second.
sar -B 1 3
Output
07:22:09 PM pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff
07:22:10 PM 0.00 0.00 0.00 0.00 33.00 0.00 0.00 0.00 0.00
07:22:11 PM 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00
07:22:12 PM 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Average: 0.00 0.00 0.00 0.00 11.33 0.00 0.00 0.00 0.00
It’s possible to output every CPU cores’ usage with “mpstat” command like follows. Output the current all CPU cores’ usage for 3 times every second.
mpstat -P ALL 1 3
Output
Linux 5.10.0-21-amd64 (dns.ipgw.io) 03/13/2023 _x86_64_ (1 CPU)
07:24:09 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
07:24:10 PM all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
07:24:10 PM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
07:24:10 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
07:24:11 PM all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
07:24:11 PM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
07:24:11 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
07:24:12 PM all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
07:24:12 PM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
Average: all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
Average: 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
It’s possible to output disk I/O usage with “iostat” command like follows. Output the current I/O usage with Mega-bytes for 3 times every second.
iostat -mx 1 3
Output
Linux 5.10.0-21-amd64 (dns.ipgw.io) 03/13/2023 _x86_64_ (1 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
0.07 0.00 0.09 0.23 0.01 99.60
Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
loop0 0.00 0.00 0.00 0.00 55.08 21.62 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01
loop1 0.05 0.00 0.00 0.00 6.78 29.62 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.04
loop2 0.00 0.00 0.00 0.00 28.70 7.24 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01
loop3 0.00 0.00 0.00 0.00 31.74 8.05 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01
loop4 0.00 0.00 0.00 0.00 0.00 1.50 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sr0 0.00 0.00 0.00 0.00 0.11 0.17 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
vda 0.60 0.03 0.23 27.39 15.19 46.77 0.32 0.01 0.32 49.34 5.36 19.55 0.00 0.00 0.00 0.00 0.00 0.00 0.05 17.73 0.01 0.30
avg-cpu: %user %nice %system %iowait %steal %idle
1.00 0.00 1.00 0.00 0.00 98.00
Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
vda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.00 0.00 0.00 100.00
Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
vda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
It’s possible to output every CPU usage for processes with “pidstat” command like follows. Output the CPU usage for a process ID “1169” for 3 times every second.
pidstat -p 1169 1 3
Output
Linux 5.10.0-21-amd64 (dns.ipgw.io) 03/13/2023 _x86_64_ (1 CPU)
04:09:09 PM PID %usr %system %guest %CPU CPU Command
04:09:10 PM 1169 0.00 0.00 0.00 0.00 1 bash
04:09:11 PM 1169 0.00 0.00 0.00 0.00 1 bash
04:09:12 PM 1169 0.00 0.00 0.00 0.00 1 bash
Average: 1169 0.00 0.00 0.00 0.00 - bash
Viewing current processes
pidstat -d
Output
Linux 5.10.0-21-amd64 (dns.ipgw.io) 03/13/2023 _x86_64_ (1 CPU)
08:07:05 PM UID PID kB_rd/s kB_wr/s kB_ccwr/s iodelay Command
08:07:05 PM 0 1 6.38 0.01 0.00 375 systemd
08:07:05 PM 0 141 0.00 0.00 0.00 1 kworker/0:3-events
08:07:05 PM 0 173 0.00 0.52 0.00 1466 jbd2/vda1-8
08:07:05 PM 0 212 0.46 0.97 0.00 403 systemd-journal
08:07:05 PM 0 241 1.20 0.00 0.00 28 systemd-udevd
08:07:05 PM 0 246 0.09 0.00 0.00 162 loop0
08:07:05 PM 0 248 1.21 0.00 0.00 410 loop1
08:07:05 PM 0 249 0.04 0.00 0.00 141 loop2
08:07:05 PM 0 250 0.04 0.00 0.00 136 loop3
08:07:05 PM 104 446 0.05 0.00 0.00 28 systemd-timesyn
08:07:05 PM 0 449 0.01 0.00 0.00 18 cron
08:07:05 PM 103 450 0.06 0.00 0.00 92 dbus-daemon
08:07:05 PM 0 455 0.12 0.00 0.00 126 qemu-ga
08:07:05 PM 0 456 0.11 0.04 0.00 84 rsyslogd
08:07:05 PM 0 457 2.01 0.18 0.00 608 snapd
08:07:05 PM 0 459 0.02 0.00 0.00 21 systemd-logind
08:07:05 PM 108 462 1.74 0.01 0.00 401 zabbix_agent2
08:07:05 PM 0 463 0.02 0.00 0.00 33 agetty
08:07:05 PM 106 538 4.35 0.00 0.00 645 mariadbd
08:07:05 PM 0 565 0.66 0.00 0.00 0 sshd
08:07:05 PM 0 575 0.37 0.00 0.00 463 apache2
08:07:05 PM 0 1296 5.93 2.93 0.12 0 bash
08:07:05 PM 0 2255 0.00 0.00 0.00 1 kworker/u2:0-events_u