How To Use SAR To Monitor Linux System? SAR Examples And Usage

To get the overview of the servers machine with status of different important metrics at different points of time, you will need System Activity Reporter tool. It is very helpful tool for system administrator to get server overview. Suppose if you are facing an issue, like some customers are unable fetch some data from MySQL database. First thing that most of the Linux Admin do is to recall the same issue when it previously occurred, and if you remember the day of its previous accurrence then you can easily compare the statistics with the current statistics.

Check SAR Utility Package On System

First of all you will need check and confirm whether you have SAR utility installed on the server.

# rpm -qa | grep "sysstat"

Install SAR Utility

If above package not install than you can download and install easily it in your server using yum command.

For RHEL/CentOS/Fedora

# yum install sysstat -y

For Ubuntu/Debian

$ sudo apt-get install sysstat

Please make sure that you have EPEL, RPMFORMGE repository enabled on your server, otherwise your distribution DVD will be a nice place to look for the package.

SAR will provide you following information:

1. Memory used and available
2. Individual CPU statistics
3. Collective CPU usage
4. Swap space used and available
5. Overall I/O activities of the system
6. Individual device I/O activities
7. Context switch statistics
8. Run queue and load average data
9. Network statistics
10. Report sar data from a specific time

One main thing you will need to understand regarding SAR is that everything is done using a cron. You can /etc/cron.d/sysstat file in many Linux distribution.

SAR Crons

# cat /etc/cron.d/sysstat
# run system activity accounting tool every 10 minutes
*/10 * * * * root /usr/lib64/sa/sa1 1 1
# generate a daily summary of process accounting at 23:53
53 23 * * * root /usr/lib64/sa/sa2 -A

Above you can see “sa1” script located at “/usr/lib64/sa/” at every 10 minutes. You can also see the “sa2” script located at “/usr/lib64/sa/” at the end of the day around 23:53

First cron “/usr/lib64/sa/sa1” will run every 10 minutes which inturn will call the sadc utility to collect system stats and store it in a binary file ( one file for a day )

Second cron “/usr/lib64/sa/sa2” will run at the end of the day and dump all the contents of that binary file into another text file, and purges data older than a particular number of days, Normally 7 dyas by default. Which is mentioned in the following file.

# cat /etc/sysconfig/sysstat
# How long to keep log files (days), maximum is a month
HISTORY=7

You can modify the HISTORY as per requirement,

Enable Stat Collection

Once you have installed sysstat, it enables statistics collections by default. In case it is not enable you can enable it by editing below line:

# vim /etc/default/sysstat

ENABLE="false"

Change to

ENABLE="true"

Save and close the file and restart sysstat.

# service sysstat restart

Once it’s restarted it will start collecting data.

The simple sar command output is as shown below:

12:00:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
12:01:01 AM     all     73.28      0.00      1.25      0.00      0.00     25.47
12:02:01 AM     all      7.83      0.00      0.44      0.00      0.00     91.73
12:03:01 AM     all     61.65      0.00      0.70      0.00      0.00     37.66
12:04:01 AM     all     57.85      0.00      0.82      0.00      0.00     41.34
12:05:01 AM     all      4.25      0.00      0.41      0.00      0.00     95.34
12:06:01 AM     all      4.20      0.00      0.22      0.00      0.00     95.58
12:07:01 AM     all      5.05      0.00      0.33      0.00      0.00     94.63
12:08:01 AM     all      4.76      0.00      0.06      0.00      0.00     95.18
12:09:01 AM     all     37.57      0.00      0.37      0.00      0.00     62.05
12:10:01 AM     all     70.04      0.00      0.80      0.00      0.00     29.16
12:11:01 AM     all      5.03      0.00      0.12      0.00     

You can see the output that its reporting the output of the collected stats for every minute because I have set cron at 1 minute interval and you can show the details of the whole day.

SAR Command Output Explain:

• %user: This shows the total time that the processor is spending on different process YCX5UKN5ZKEJ
• %sys: this shows the percentage of time spend by the processor for operating system tasks(because the previous user shows the time spend for user end process)
• %iowait: the name iowait itself suggests that its the time spend by processor waiting for devices(input and output)
• %nice: Most of you guys must be knowing that a user can change the priority of a process in Linux by changing the nice value in Linux. This table shows the time spend by CPU for process whose nice value has been changed.
• %steal: This column shows the amount to time spend by a CPU (which is virtualized), for resources from the physical CPU
• %idle: This suggests the idle time spend by the processor.

By default sar stores all of its data under /var/log/sa/ and a days are named as shown below.

S01 s02 s03 s04

s01 means firt day of the month, s02 means second day of the month, s03,s04 and so on.

Check Sysstat Version

# sar -V
sysstat version 10.0.5
(C) Sebastien Godard (sysstat  orange.fr)

Check Some Historical Sysstat Files

# ls /var/log/sysstat/
sa03  sa04  sa05  sa17  sa28

Practical SAR Example and Usage

1. CPU Usage Of All CPUs (sar -u)

You can check the cumulative real time CPU usage of all CPUs. “1 3” reports for every 1 seconds a total of 3 times. You can focus on the last field “%idle” to see the cpu load.

# sar -u 1 3
Linux 3.14-kali1-amd64 (kali)     17/06/14     _x86_64_    (4 CPU)

21:57:24        CPU     %user     %nice   %system   %iowait    %steal     %idle
21:57:25        all      1.01      0.00      0.51      0.51      0.00     97.98
21:57:26        all      1.03      0.00      0.51      0.00      0.00     98.46
21:57:27        all      1.26      0.00      0.25      0.50      0.00     97.98
Average:        all      1.10      0.00      0.42      0.34      0.00     98.14

Some more examples:

• sar -u Displays CPU usage for the current day that was collected until that point.
• sar -u 1 3 Displays real time CPU usage every 1 second for 3 times.
• sar -u ALL Same as “sar -u” but displays additional fields.
• sar -u ALL 1 3 Same as “sar -u 1 3″ but displays additional fields.
• sar -u -f /var/log/sysstat/sa10 Displays CPU usage for the 10day of the month from the sa10 file.

2. CPU Usage Of Individual CPU or Core (sar -P)

You can the all cores activity if you have 4 cores on the server.

# sar -P ALL 1 1
Linux 3.14-kali1-amd64 (kali)     17/06/14     _x86_64_    (4 CPU)

22:05:24        CPU     %user     %nice   %system   %iowait    %steal     %idle
22:05:25        all      1.25      0.00      0.50      0.25      0.00     97.99
22:05:25          0      0.00      0.00      0.00      1.00      0.00     99.00
22:05:25          1      3.00      0.00      1.00      0.00      0.00     96.00
22:05:25          2      1.02      0.00      0.00      0.00      0.00     98.98
22:05:25          3      0.00      0.00      0.99      0.00      0.00     99.01

-P ALL :- Displays the statistics for ALL the individual Cores.

In the above example under “CPU” column 0,1,2 and 3 indicates the corresponding CPU core numbers.

“-P 1″ indicates that it should displays statistics only for the 2nd Core. (Note that Core number starts from 0).

Some more exmaples:

• sar -P ALL Displays CPU usage broken down by all cores for the current day.
• sar -P ALL 1 3 Displays real time CPU usage for ALL cores every 1 second for 3 times (broken down by all cores).
• sar -P 1 Displays CPU usage for core number 1 for the current day.
• sar -P 1 1 3 Displays real time CPU usage for core number 1, every 1 second for 3 times.
• sar -P ALL -f /var/log/sysstat/sa10 Displays CPU usage broken down by all cores for the 10day day of the month from sa10 file.

3. Memory Usage Free and Used ( sar -r )

sar -r display the memory statistics. “1 3″ reports for every 1 seconds a total of 3 times. Most likely you’ll focus on “kbmemfree” and “kbmemused” for free and used memory.

# sar -r 1 3
Linux 3.14-kali1-amd64 (kali)     17/06/14     _x86_64_    (4 CPU)

22:08:54    kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact
22:08:55      4644240   3550716     43.33    153180   2204384   2612376     27.44   1756372   1438360
22:08:56      4644240   3550716     43.33    153180   2204384   2612376     27.44   1756468   1438360
22:08:57      4643744   3551212     43.33    153188   2204384   2612376     27.44   1757084   1438360
Average:      4644075   3550881     43.33    153183   2204384   2612376     27.44   1756641   1438360

Some more examples:

• sar -r
• sar -r 1 3
• sar -r -f /var/log/sysstat/sa10

4. Swap Usage (sar -S)

sar -r display the swap statistics. “1 3″ reports for every 1 seconds a total of 3 times. If the “kbswpused” and “%swpused” are at 0, then your system is not swapping.

# sar -S 1 3
Linux 3.14-kali1-amd64 (kali)     17/06/14     _x86_64_    (4 CPU)

22:10:03    kbswpfree kbswpused  %swpused  kbswpcad   %swpcad
22:10:04      1325052         0      0.00         0      0.00
22:10:05      1325052         0      0.00         0      0.00
22:10:06      1325052         0      0.00         0      0.00
Average:      1325052         0      0.00         0      0.00

Some more examples:

• sar -S
• sar -S 1 3
• sar -S -f /var/log/sysstat/sa10

Please Note:

Use “sar -R” to identify number of memory pages freed, used, and cached per second by the system.
Use “sar -H” to identify the hugepages (in KB) that are used and available.
Use “sar -B” to generate paging statistics. i.e Number of KB paged in (and out) from disk per second.
Use “sar -W” to generate page swap statistics. i.e Page swap in (and out) per second.

5. I/O Activities (sar -b)

It show the I/O statistics. “1 3″ reports for every 1 seconds a total of 3 times.

Following fields are displays in the example below.

tps – Transactions per second (this includes both read and write)
rtps – Read transactions per second
wtps – Write transactions per second
bread/s – Bytes read per second
bwrtn/s – Bytes written per second

# sar -b 1 3
Linux 3.14-kali1-amd64 (kali)     17/06/14     _x86_64_    (4 CPU)

22:11:28          tps      rtps      wtps   bread/s   bwrtn/s
22:11:29         0.00      0.00      0.00      0.00      0.00
22:11:30         0.00      0.00      0.00      0.00      0.00
22:11:31         0.00      0.00      0.00      0.00      0.00
Average:         0.00      0.00      0.00      0.00      0.00

Some more examples:

• sar -b
• sar -b 1 3
• sar -b -f /var/log/sysstat/sa10

Note: Use “sar -v” to display number of inode handlers, file handlers, and pseudo-terminals used by the system.

6. Individual Block Device I/O Activities (sar -d)

sar -d is used to identify the activities by the individual block devices (i.e a specific mount point, or LUN, or partition).

# sar -d 1 1
Linux 3.14-kali1-amd64 (kali)     17/06/14     _x86_64_    (4 CPU)

22:12:08          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
22:12:09       dev8-0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
22:12:09      dev8-16      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

Average:          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
Average:       dev8-0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:      dev8-16      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

In the above example “DEV” indicates the specific block device.

For example: “dev53-1″ means a block device with 53 as major number, and 1 as minor number.

The device name (DEV column) can display the actual device name (for example: sda, sda1, sdb1 etc.,), if you use the -p option (pretty print) as shown below.

# sar -p -d 1 1
Linux 3.14-kali1-amd64 (kali)     17/06/14     _x86_64_    (4 CPU)

22:13:33          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
22:13:34          sda      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
22:13:34          sdb      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

Average:          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
Average:          sda      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:          sdb      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

Some more examples:

• sar -d
• sar -d 1 3
• sar -d -f /var/log/sysstat/sa10
• sar -p -d

7. Display context switch per second (sar -w)

sar -w display the total number of processes created per second and total number of context switches per second. “1 3″ reports for every 1 seconds a total of 3 times.

# sar -w 1 3
Linux 3.14-kali1-amd64 (kali)     17/06/14     _x86_64_    (4 CPU)

22:14:45       proc/s   cswch/s
22:14:46        14.00   2817.00
22:14:47        14.00   3355.00
22:14:48        14.00   2450.00
Average:        14.00   2874.00

Some more examples:

• sar -w
• sar -w 1 3
• sar -w -f /var/log/sysstat/sa10

8. Load average and run queue (sar -q)

sar -q shows the run queue size and load average of last 1 minute, 5 minutes, and 15 minutes. “1 3″ reports for every 1 seconds a total of 3 times.

# sar -q 1 3
Linux 3.14-kali1-amd64 (kali)     17/06/14     _x86_64_    (4 CPU)

22:15:18      runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
22:15:19            0       392      0.31      0.26      0.29         0
22:15:20            1       392      0.31      0.26      0.29         0
22:15:21            0       392      0.31      0.26      0.29         0
Average:            0       392      0.31      0.26      0.29         0

Please Note: The “blocked” column displays the number of tasks that are currently blocked and waiting for I/O operation to complete.

Some more examples:

• sar -q
• sar -q 1 3
• sar -q -f /var/log/sysstat/sa10

9. Network Statistics (sar -n)

sar -n displays the various network statistics. For example: number of packets received (transmitted) through the network card, statistics of packet failure etc.,. “1 3″ reports for every 1 seconds a total of 3 times.

sar -n KEYWORD

One of the following KEYWORD can be used.

DEV – Displays network devices vital statistics for eth0, eth1, etc.,
EDEV – Display network device failure statistics
NFS – Displays NFS client activities
NFSD – Displays NFS server activities
SOCK – Displays sockets in use for IPv4
IP – Displays IPv4 network traffic
EIP – Displays IPv4 network errors
ICMP – Displays ICMPv4 network traffic
EICMP – Displays ICMPv4 network errors
TCP – Displays TCPv4 network traffic
ETCP – Displays TCPv4 network errors
UDP – Displays UDPv4 network traffic
SOCK6, IP6, EIP6, ICMP6, UDP6 are for IPv6
ALL – This displays all of the above information. The output will be very long.

# sar -n DEV 1 1 
Linux 3.14-kali1-amd64 (kali)     17/06/14     _x86_64_    (4 CPU)

22:16:15        IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
22:16:16         eth0      4.00      4.00      0.92      0.26      0.00      0.00      0.00
22:16:16           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00

Average:        IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
Average:         eth0      4.00      4.00      0.92      0.26      0.00      0.00      0.00
Average:           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00

10. Report Sar Data Using Start Time (sar -s)

When you view historic sar data from the /var/log/sysstat/saXX file using “sar -f” option, it displays all the sar data for that specific day starting from 12:00 a.m for that day.

Using “-s hh:mi:ss” option, you can specify the start time. For example, if you specify “sar -s 10:00:00″, it will display the sar data starting from 10 a.m (instead of starting from midnight) as shown below.

You can combine -s option with other sar option.

For example, to report the load average on 26th of this month starting from 11 p.m in the morning, combine the -q and -s option as shown below.

# sar -q -f /var/log/sysstat/sa17 -s 23:00:01  | head
Linux 3.14-kali1-amd64 (kali)     17/06/14     _x86_64_    (4 CPU)

23:00:01      runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
23:02:01            2       381      0.19      0.36      0.37         0
23:04:01            0       381      0.19      0.28      0.33         0
23:06:01            0       382      0.07      0.20      0.30         0
23:08:01            1       382      0.17      0.18      0.28         0
23:10:01            1       373      0.06      0.14      0.25         0
23:12:01            0       372      0.10      0.13      0.23         0
23:14:01            0       372      0.36      0.19      0.24         0

There is no option to limit the end-time. You just have to get creative and use head command as shown below.

For example, starting from 2300hrs (11PM) if you want to see 7 entries, you have to pipe the above output to “head”

# sar -q -f /var/log/sysstat/sa17 -s 23:00:01  | head
Linux 3.14-kali1-amd64 (kali)     17/06/14     _x86_64_    (4 CPU)

23:00:01      runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
23:02:01            2       381      0.19      0.36      0.37         0
23:04:01            0       381      0.19      0.28      0.33         0
23:06:01            0       382      0.07      0.20      0.30         0
23:08:01            1       382      0.17      0.18      0.28         0
23:10:01            1       373      0.06      0.14      0.25         0
23:12:01            0       372      0.10      0.13      0.23         0
23:14:01            0       372      0.36      0.19      0.24         0

Thanks:)