Introduction
Linux is an extremely powerful operating system, but just like any other system, it is not immune to experiencing issues. However, it comes with a variety of effective tools that can be used to monitor the performance of the system. The iostat command is one of the most useful performance monitoring tools. This command provides insight about CPU utilization as well as input/output (I/O) statistics for devices, partitions, and network file systems. It is one of the most helpful tools for diagnosing system performance issues. The iostat is available on all the standard Linux distributions, including Red Hat, CentOS, and Ubuntu, among others.
In this article, we will examine some of the most common troubleshooting cases. Also, discuss how we can use the iostat command to diagnose and resolve performance issues.
Installation of iostat
The iostat command is part of Linux sysstat package, which offers a variety of utilities to collect system performance statistics. In particular, responsible for collecting and analyzing input/output statistics for the entire system, including devices, partitions, and network file systems. Using the appropriate package manager, we can install the sysstat package on different of Linux distributions.
For RHEL/CentOS:
$ sudo yum install sysstatFor Ubuntu:
$ sudo apt install sysstatIn debian based distros we will need to enable sysstat by setting the ‘ENABLED’ parameter true in sysstat configuration file. However, in RHEL/CentOS it is enabled by default. Use the editor to make the below changes in configuration file, after which you should save the file as show below.
$ sudo vim /etc/default/sysstatENABLED="true"Once configuration parameter set, we must start and enable sysstat service.
$ sudo systemctl enable sysstat
$ sudo systemctl start sysstat
The Basics of iostat
Let’s have a fundamental understanding of the iostat command under our belts before moving on to the other problem scenarios. Simply opening a terminal/shell in Linux system and entering the command iostat is all that is required to execute iostat with root or sudo privileges. You will be presented with statistics for both your CPU and your disk I/O. An overview of the system’s disk I/O data can be obtained by using the straightforward iostat command without any parameters.
$ sudo iostatLinux 5.14.0-325.el9.x86_64 (localhost.localdomain) 06/20/2023 _x86_64_ (1 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
5.20 0.72 9.45 0.15 0.00 84.49
Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd
dm-0 23.41 612.25 249.06 0.00 478299 194570 0
dm-1 0.52 2.91 1.53 0.00 2272 1196 0
sda 24.92 651.74 294.94 0.00 509150 230412 0
sr0 0.07 2.68 0.00 0.00 2096 0 0
sr1 0.07 2.66 0.00 0.00 2080 0 0
The output gives the average statistics collected by the system since the previous time it was restarted. You can add the ‘-x’ option to obtain additional information in greater depth. This gives enhanced statistics and displays more detailed information regarding disk I/O operations.
$ sudo iostat -xLinux 5.14.0-325.el9.x86_64 (localhost.localdomain) 06/20/2023 _x86_64_ (1 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
4.88 0.67 9.01 0.14 0.00 85.30
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
dm-0 18.66 571.99 0.00 0.00 0.55 30.65 3.27 233.11 0.00 0.00 1.61 71.34 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 2.43
dm-1 0.13 2.72 0.00 0.00 0.29 20.47 0.42 1.69 0.00 0.00 0.98 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02
sda 20.42 608.87 0.05 0.23 0.47 29.82 2.93 276.23 0.94 24.24 1.45 94.30 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 2.53
sr0 0.07 2.51 0.00 0.00 1.30 37.43 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01
sr1 0.06 2.49 0.00 0.00 1.48 38.52 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01
The ‘-m’ option is also quite helpful because it displays all sizes in megabytes, which makes the result much simpler to comprehend.
$ sudo iostat -xmLinux 5.14.0-325.el9.x86_64 (localhost.localdomain) 06/20/2023 _x86_64_ (1 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
4.48 0.61 8.46 0.13 0.00 86.32
Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
dm-0 17.08 0.51 0.00 0.00 0.55 30.65 3.02 0.21 0.00 0.00 1.61 70.85 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 2.23
dm-1 0.12 0.00 0.00 0.00 0.29 20.47 0.39 0.00 0.00 0.00 0.98 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02
sda 18.70 0.54 0.04 0.23 0.47 29.81 2.70 0.25 0.86 24.22 1.45 93.66 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 2.32
sr0 0.06 0.00 0.00 0.00 1.30 37.43 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01
sr1 0.06 0.00 0.00 0.00 1.48 38.52 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01
Troubleshooting with the iostat Command
The following are a few common scenarios in which the iostat command might be helpful for Linux system administrations in day-to-day operations, including situations involving troubleshooting.
High CPU Usage
When your computer’s performance starts to seem sluggish, one of the first things you should check is how much of the processor is being used. Start the process with the following command.
$ sudo iostat -cLinux 5.14.0-325.el9.x86_64 (localhost.localdomain) 06/20/2023 _x86_64_ (1 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
3.95 0.54 7.72 0.12 0.00 87.68
This will display statistics regarding the use of the CPU. If you see a high %user utilization as well as %system usage, it’s possible that your CPU is being overworked.
High I/O Wait
If CPU is spending a significant amount of time waiting for I/O operations ‘%iowait’ usage reflect the same, this may be a hint that there are problems with the performance of your disk. We can check the same with the command below.
$ sudo iostat -cLinux 5.14.0-325.el9.x86_64 (localhost.localdomain) 06/20/2023 _x86_64_ (1 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
3.95 0.54 7.72 0.12 0.00 87.68Disk Throughput
It is helpful to monitor the amount of data that is being read from or written to your disk to locate any bottlenecks in the system. This command displays statistics at the level of the device in megabytes. If the figures are high, it may mean that your disk is getting close to filling up.
$ sudo iostat -dmLinux 5.14.0-325.el9.x86_64 (localhost.localdomain) 06/20/2023 _x86_64_ (1 CPU)
Device tps MB_read/s MB_wrtn/s MB_dscd/s MB_read MB_wrtn MB_dscd
dm-0 14.86 0.38 0.15 0.00 472 191 0
dm-1 0.37 0.00 0.00 0.00 2 1 0
sda 15.81 0.40 0.18 0.00 503 226 0
sr0 0.04 0.00 0.00 0.00 2 0 0
sr1 0.04 0.00 0.00 0.00 2 0 0
Disk Utilization
Disk utilization is measured with ‘%util’, metric in iostat command output. If this value stays continuously high (over 90 percent), it’s possible that your disk is the source of performance issues.
$ sudo iostat -dxLinux 5.14.0-325.el9.x86_64 (localhost.localdomain) 06/20/2023 _x86_64_ (1 CPU)
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
dm-0 11.70 361.56 0.00 0.00 0.55 30.91 2.12 146.08 0.00 0.00 1.66 69.04 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 1.53
dm-1 0.08 1.70 0.00 0.00 0.29 20.47 0.26 1.06 0.00 0.00 0.98 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01
sda 12.80 384.61 0.03 0.23 0.47 30.04 1.89 173.05 0.61 24.24 1.48 91.32 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 1.59
sr0 0.04 1.56 0.00 0.00 1.30 37.43 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01
sr1 0.04 1.55 0.00 0.00 1.48 38.52 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01
Specific Device Monitoring
Simply appending the device name to the iostat command, we will be able to monitor the statistics for a particular device.
$ sudo iostat -dx /dev/sdaLinux 5.14.0-325.el9.x86_64 (localhost.localdomain) 06/20/2023 _x86_64_ (1 CPU)
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sda 11.24 337.80 0.03 0.23 0.47 30.04 1.68 152.10 0.54 24.20 1.48 90.54 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 1.40
Real-Time Monitoring
Utilize the interval parameter to carry out monitoring in real time. For instance, to refresh the statistics every five seconds specify time interval parameter to iostat command.
$ sudo iostat -c 5Linux 5.14.0-325.el9.x86_64 (localhost.localdomain) 06/20/2023 _x86_64_ (1 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
2.64 0.35 5.83 0.08 0.00 91.10
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 1.01 0.00 0.00 98.99
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.81 0.00 0.00 99.19
...Long-term Disk Performance
The iostat command have the ability to choose the frequency with which the data are displayed, which is useful for doing analysis over a longer period of time for disk performance historically and real real time monitoring. Below command run after every five second with frequency of ten times.
$ sudo iostat -dx 5 10Identifying Heavy Disk Read Operations
The ‘r/s’ field, which stands for ‘read transactions per second’, can be used to assist in identifying read operations that are intensive.
$ sudo iostat -dxLinux 5.14.0-325.el9.x86_64 (localhost.localdomain) 06/20/2023 _x86_64_ (1 CPU)
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
dm-0 8.20 256.17 0.00 0.00 0.55 31.24 1.52 100.25 0.00 0.00 1.65 65.85 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 1.07
dm-1 0.06 1.16 0.00 0.00 0.29 20.47 0.39 1.57 0.00 0.00 0.86 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01
sda 8.96 271.91 0.02 0.23 0.47 30.36 1.40 119.51 0.60 29.97 1.45 85.46 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 1.12
sr0 0.03 1.07 0.00 0.00 1.30 37.43 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01
sr1 0.03 1.06 0.00 0.00 1.48 38.52 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01
If the value for ‘r/s’ is high, this indicates that the system is now undergoing a significant amount of read-heavy operations.
Identifying Heavy Disk Write Operations
The ‘w/s’ (write transactions per second) is a metric that can be utilized to detect write operations that are intensive.
$ sudo iostat -dxLinux 5.14.0-325.el9.x86_64 (localhost.localdomain) 06/20/2023 _x86_64_ (1 CPU)
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
dm-0 8.20 256.17 0.00 0.00 0.55 31.24 1.52 100.25 0.00 0.00 1.65 65.85 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 1.07
dm-1 0.06 1.16 0.00 0.00 0.29 20.47 0.39 1.57 0.00 0.00 0.86 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01
sda 8.96 271.91 0.02 0.23 0.47 30.36 1.40 119.51 0.60 29.97 1.45 85.46 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 1.12
sr0 0.03 1.07 0.00 0.00 1.30 37.43 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01
sr1 0.03 1.06 0.00 0.00 1.48 38.52 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01
A high ‘w/s’ number indicates a significant amount of write activity.
Identifying Average Queue Length
The ‘aqu-sz’ parameter might help identify if I/O requests are queuing up. Make use of the command that is below.
$ sudo iostat -dxLinux 5.14.0-325.el9.x86_64 (localhost.localdomain) 06/20/2023 _x86_64_ (1 CPU)
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
dm-0 8.20 256.17 0.00 0.00 0.55 31.24 1.52 100.25 0.00 0.00 1.65 65.85 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 1.07
dm-1 0.06 1.16 0.00 0.00 0.29 20.47 0.39 1.57 0.00 0.00 0.86 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01
sda 8.96 271.91 0.02 0.23 0.47 30.36 1.40 119.51 0.60 29.97 1.45 85.46 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 1.12
sr0 0.03 1.07 0.00 0.00 1.30 37.43 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01
sr1 0.03 1.06 0.00 0.00 1.48 38.52 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01
A large value for ‘aqu-sz’ may suggest that there is a problem with the performance of the system because requests are currently being held in the queue.
Checking Directory Read/Write
We can specify directory for iostat to read device information from if you want to. The option -f instructs iostat to read device statistics using only the files that are located in the alternative directory. Whereas, the option +f instructs it to read device statistics using both the standard kernel files and the files in the alternative directory.
$ sudo iostat -f /usrLinux 5.14.0-325.el9.x86_64 (localhost.localdomain) 06/20/2023 _x86_64_ (1 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
1.56 0.22 4.24 0.05 0.00 93.93
Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd
$ sudo iostat +f /usrLinux 5.14.0-325.el9.x86_64 (localhost.localdomain) 06/20/2023 _x86_64_ (1 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
1.51 0.21 4.15 0.05 0.00 94.08
Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd
dm-0 6.51 170.31 66.77 0.00 504367 197732 0
dm-1 0.30 0.77 1.04 0.00 2272 3076 0
sda 6.92 180.73 79.53 0.00 535239 235515 0
sr0 0.02 0.71 0.00 0.00 2096 0 0
sr1 0.02 0.70 0.00 0.00 2080 0 0
Conclusion
Monitoring and troubleshooting the performance of a Linux system requires iostat command, which is an essential tool. Gaining a knowledge of its output will assist in efficiency diagnosing performance issues and maintaining the optimal functioning of system. If you continue to hone your skills with it, you’ll soon be able to see possible issues with relative ease. The iostat command is a strong instrument which offers insights into the operation of the system to identify potential bottlenecks. We will be able to proactively manage system resources and troubleshoot problems as they occur if you monitor critical metrics.
Keep in mind that every system is different, and normal and abnormal performance might shift depending on the context. Make use of right tools like iostat over time build an understanding of what’s normal for system and when to investigate problems.
Continue to look for new information, educate yourself, and solve problems!