DAG: Dstat: Versatile resource statistics tool

» Dag Wieers » Home-made » Dstat: Versatile resource statistics tool

Dstat: Versatile resource statistics tool

Dstat is a versatile replacement for vmstat, iostat, netstat, nfsstat and ifstat. Dstat overcomes some of their limitations and adds some extra features, more counters and flexibility. Dstat is handy for monitoring systems during performance tuning tests, benchmarks or troubleshooting.

Dstat allows you to view all of your system resources instantly, you can eg. compare disk usage in combination with interrupts from your IDE controller, or compare the network bandwidth numbers directly with the disk throughput (in the same interval).

Dstat gives you detailed selective information in columns and clearly indicates in what magnitude and unit the output is displayed. Less confusion, less mistakes.

Dstat is unique in letting you aggregate block device throughput for a certain diskset or networkset, ie. you can see the throughput for all the block devices that make up a single filesystem or storage system.

You can write your own dstat plugins to monitor whatever you like in just a few minutes based on provided examples and a little bit of Python knowledge.

Dstat's output by default is designed for being interpreted by humans in real-time, however the new CSV output allows you to store CSV output in detail to a file to be imported later into Gnumeric or Excel to generate graphs.

Features

Combines vmstat, iostat, ifstat, netstat information and more
Shows stats in exactly the same timeframe
Enable/order counters as they make most sense during analysis/troubleshooting
Modular design
Written in python so easily extendable for the task at hand
Easy to extend, add your own counters (please contribute those)
Includes about 10 external plugins to show how easy it is to add counters
Can summarize grouped block/network devices and give total numbers
Can show interrupts per device
Very accurate timeframes, no timeshifts when system is stressed
Shows exact units and limits conversion mistakes
Indicate different units with different colors
Show intermediate results when delay > 1
Allows to export CSV output, which can be imported in Gnumeric and Excel to make graphs

External plugins

Here are the existing plugins, send me your own plugins.

dstat_app - the most expensive process on the system
dstat_battery - the percentage of battery charge (needs ACPI)
dstat_cpufreq - the CPU frequency (needs ACPI)
dstat_dbus - the number of dbus connections (needs python-dbus)
dstat_freespace - see the disk usage per partition
dstat_gpfs - the GPFS read/write IO
dstat_gpfsop - the GPFS filesystem operations
dstat_nfs3 - the NFS v3 client operations
dstat_nfs3op - the extended NFS v3 client operations
dstat_nfsd3 - the NFS v3 server operations
dstat_nfsd3op - the extended NFS v3 server operations
dstat_postfix - counters of the differnt queues (needs postfix)
dstat_rpc - RPC client calls
dstat_rpcd - RPC server calls
dstat_sendmail - counters of the queue (needs sendmail)
dstat_thermal - CPU temperature
dstat_utmp - number of utmp sessions (needs python-utmp)
dstat_wifi - wireless link quality and signal/noise ratio (needs python-wifi)

Future

Add /etc/dstat.conf configuration file to customize
Create a complete counter/object model structure
Interface directly with rrdtool for real-time graphing
Create client-server application model for remote graphing
See TODO for more details

Requirements

Dstat exists currently in 2 versions. The normal dstat script works with a python 2.2 or newer. The backported dstat15 script works with a python 1.5 or newer. At this time they both are full-featured.

It has been tested with different sets of 2.4 and 2.6 kernels.

Screenshot 1

[dag@emyn ~]$ dstat --help
Usage: dstat [-afv] [options..] [delay [count]]
Versatile tool for generating system resource statistics

Dstat options:
  -c, --cpu              enable cpu stats
     -C 0,3,total           include cpu0, cpu3 and total
  -d, --disk             enable disk stats
     -D total,hda           include hda and total
  -g, --page             enable page stats
  -i, --int              enable interrupt stats
     -I 5,eth2              include int5 and interrupt used by eth2
  -l, --load             enable load stats
  -m, --mem              enable memory stats
  -n, --net              enable network stats
     -N eth1,total          include eth1 and total
  -p, --proc             enable process stats
  -s, --swap             enable swap stats
  -t, --time             enable time counter
  -y, --sys              enable system stats
  --ipc                  enable ipc stats
  --lock                 enable lock stats
  --raw                  enable raw stats
  --tcp                  enable tcp stats
  --udp                  enable udp stats
  --unix                 enable unix stats

  -M stat1,stat2         enable external stats
     --mods stat1,stat2

  -a, --all              equals -cdngy (default)
  -f, --full             expand -D, -I and -N discovery lists
  -v, --vmstat           equals -pmgdsc -D total

  --integer              show integer values
  --nocolor              disable colors (implies --noupdate)
  --noheaders            disable repetitive headers
  --noupdate             disable intermediate updates
  --output file          write CSV output to file

  delay is the delay in seconds between each update
  count is the number of updates to display before exiting
  The default delay is 1 and count is unspecified (unlimited)

Screenshot 2

Only in black and white :)

[dag@emyn ~]$ dstat --vmstat
---procs--- ------memory-usage----- ---paging-- -disk/total ---system-- ----total-cpu-usage----
run blk new|_used _free _buff _cach|__in_ _out_|_read write|_int_ _csw_|usr sys idl wai hiq siq
  0   0   0| 324M 11.0M 40.5M  128M|   0     0 |   0     0 |   0     0 | 12   5  81   2   0   0
  0   0   0| 324M 10.9M 40.5M  128M|   0     0 |   0     0 |1193  2098 | 10   6  83   0   1   0
  7   0  26| 325M 9.84M 40.5M  128M|   0     0 |   0     0 |1184  2851 | 26  11  63   0   0   0
  0   0   0| 326M 9568k 40.5M  128M|   0     0 |   0     0 |1187  2692 | 15   9  76   0   0   0
  0   0   0| 326M 9568k 40.5M  128M|   0     0 |   0     0 |1195  1755 |  5   5  90   0   0   0
  0   0   0| 326M 9568k 40.5M  128M|   0     0 |   0     0 |1207  1903 |  7   5  88   0   0   0
  0   0   1| 324M 10.8M 40.5M  128M|   0     0 |   0     0 |1227  2184 | 13   7  79   0   1   0
  0   0   0| 324M 10.8M 40.5M  128M|   0     0 |   0    48k|1214  1800 | 11   6  83   0   0   0
  0   0   0| 324M 10.8M 40.5M  128M|   0     0 |   0  28.5k|1150  1778 |  5   4  90   0   1   0
  0   0   0| 324M 10.8M 40.5M  128M|   0     0 |   0     0 |1254  1912 |  6   6  88   0   0   0
  1   0   0| 324M 10.8M 40.5M  128M|   0     0 |   0     0 |1128  1807 |  5   4  91   0   0   0

Screenshot 3

[dag@emyn ~]$ dstat -cdnsil -D hda -N eth1
----total-cpu-usage---- --disk/hda- --net/eth1- ----swap--- ----interrupts--- ---load-avg---
usr sys idl wai hiq siq|_read write|_recv _send|_used _free|__12_ __14_ __15_|_1m_ _5m_ 15m_
 12   5  81   2   0   0|   0     0 |   0     0 | 162M  362M|   0     0     0 |0.10 0.37 0.39
 20  11  69   0   0   0|   0   512B|1708B 1674B| 162M  362M| 105     1    27 |0.10 0.37 0.39
 32   8  60   0   0   0|   0     0 |9280B 2042B| 162M  362M| 126     0    24 |0.10 0.37 0.39
 80  12   7   0   1   0|   0     0 |87.9k 15.6k| 162M  362M|   0     0    27 |0.10 0.37 0.39
 26   9  65   0   0   0|   0     0 |23.5k 3576B| 162M  362M|  12     0    24 |0.10 0.37 0.39
 13   7  80   0   0   0|   0     0 |36.8k 4885B| 162M  362M|  21     0    27 |0.10 0.37 0.39
  7   5  88   0   0   0|   0     0 |3808B  210B| 162M  362M|   0     0    24 |0.10 0.37 0.39
  3   6  89   1   1   0|   0   456k|   0     0 | 162M  362M|   0    32    27 |0.10 0.37 0.39
  4   3  93   0   0   0|   0   109k|   0     0 | 162M  362M|   0    27    24 |0.10 0.37 0.39
  7   5  88   0   0   0|   0     0 |   0     0 | 162M  362M|  60     0    27 |0.10 0.37 0.39

Real screenshot

Here are 2 screenshots of older dstats in action.

Dstat 0.4 on a Power5 system that is being stress tested.

Dsta 0.3 (first release) on 5 RHEL3 nodes in a cluster from a Windows terminal.

Bug reports

If you've found a bug, please check the TODO file for known problems and send me updates if you have more information to provide.

Please also copy&paste the output of the problem, with a description, the version of the kernel and if appropriate the involved /proc entries.

Dstat has a --debug option to profile plugins and show what plugins and /proc entries are affected. Dstat also shows some more information with the --version option that might be useful.

Download

The following packages (in order of appearance) are available. Thanks to all the people involved in packaging ! I owe you one.

or grab the latest 0.6.7 tarball at: http://dag.wieers.com/home-made/dstat/dstat-0.6.7.tar.bz2

Mailinglist

There's a mailinglist about dstat and some other related tools at: http://lists.rpmforge.net/mailman/listinfo/tools

Sourcecode

You can have access to the latest changes via subversion from:

http://svn.rpmforge.net/svn/trunk/tools/dstat/

or see the latest commits via viewvc:

http://svn.rpmforge.net/viewvc/trunk/tools/dstat/

Documentation

The current asciidoc manual page is found here:

dstat.1.html

Subversion also holds the current documentation and example config-files, so please look there for more information. If you have improvements, found a bug or have a great idea, please mail me so we can look at how to integrate it.

What users have to say

I found some comments about dstat on blogs, here's what they say:

Last modified on: Fri 14 March 2008