Sysdig – A Powerful System Monitoring and Troubleshooting Tool for Linux

[‘

n

Sysdig is an open-source, cross-platform, powerful and flexible system monitoring and troubleshooting tool for Linux; it also works on Windows and Mac OSX but with limited functionality and can be used for system analysis, inspection and debugging.

n

Normally, you would employ a mix of various Linux performance monitoring and troubleshooting tools including these ones listed below to perform the Linux monitoring and debugging tasks:

n

    n

  1. strace – discover system calls and signals to a process.
  2. n

  3. tcpdump – raw network traffic monitoring.
  4. n

  5. netstat – network connections monitoring.
  6. n

  7. htop – real time process monitoring.
  8. n

  9. iftop – real time network bandwidth monitoring.
  10. n

  11. lsof – view which files are opened by which process.
  12. n

n

However, sysdig integrates what all the above tools and many more, offer in a single and simple program, more so with amazing container support. It enables you to capture, save, filter and examine the real behavior (stream of events) of Linux systems as well as containers.

n

It comes with a command line interface and a powerful interactive UI (csysdig) which allow you to watch system activity in real time, or perform a trace dump and save for later analysis. You can watch how csysdig works from the below video.

n

n

Sysdig Features:

n

    n

  • It is fast, stable and easy-to-use with comprehensively well documented.
  • n

  • Comes with native support for container technologies, including Docker, LXC.
  • n

  • It is scriptable in Lua; offers chisels (lightweight Lua scripts) for processing captured system events.
  • n

  • Supports useful filtering of output.
  • n

  • Supports system and application tracing.
  • n

  • It can be integrated with Ansible, Puppet and Logstash.
  • n

  • Enable sample advanced log analysis.
  • n

  • It also offers Linux server attack (forensics) analysis features for ethical hackers and lot’s more.
  • n

n

In this article, we will show how to install sysdig on a Linux system, and use it with basic examples of system analysis, monitoring and troubleshooting.

n

How To Install Sysdig in Linux

n

Installing sysdig package is as easy as running the command below, which will check all the requirements; if every thing is in place, it will download and install the package from the Draios APT/YUM repository.

n

# curl -s https://s3.amazonaws.com/download.draios.com/stable/install-sysdig | bash rnORrn$ curl -s https://s3.amazonaws.com/download.draios.com/stable/install-sysdig | sudo bashrn

n

After installing it, you need to run sysdig as root because it requires access to critical areas such as /proc file system, /dev/sysdig* devices and needs to auto-load the sysdig-probe kernel module (in case it is not); otherwise use the sudo command.

n

The most basic example is running it without any arguments, this will enable you to view your Linux system stream of events updated in real-time:

n

$ sudo sysdigrn

n

Watch Linux System Events
Watch Linux System Events

n

The above output (raw data) does not perhaps make a lot of sense to you, for a more useful output run csysdig:

n

$ sudo csysdig rn

n

Monitor Linux System Events
Monitor Linux System Events

n

Note: To get the real feel of this tool, you need to use sysdig which produces raw data as we saw before, from a running Linux system: this calls for you to understand how to use filters and chisels.

n

But if you need a painless means of using sysdig – continue with csysdig.

n

Understanding Sysdig Chisels and Filters

n

Sysdig chisels are minimal Lua scripts for examining the sysdig event stream to carry out useful system troubleshooting actions and more. The command below will help you view all available chisels:

n

$ sudo sysdig -clrn

n

The screen shot shows a sample list of chisels under different categories.

n

View Sysdig Chisels
View Sysdig Chisels

n

If you want to find out more information about a particular chisel, use the -i flag:

n

$ sudo sysdig -i topprocs_cpurn

n

View Sysdig Chisel Info
View Sysdig Chisel Info

n

Sysdig filters add more power to the kind of output you can obtain from event streams, they allow you to customize the output. You should specify them at the end of a command line.

n

A straightforward and commonest filter is a basic “class.field=value” check, you can also combine chisels with filters for even more powerful customizations.

n

To view a list of available field classes, fields and their descriptions, type:

n

$ sudo sysdig -lrn

n

View Sysdig Field Classes
View Sysdig Field Classes

n

Creating Linux System Trace File

n

To dump sysdig output in a file for later analysis, use the -w flag like this.

n

You can read the trace dump file using the -r flag:

n

$ sudo sysdig -r trace.scaprn

n

The -s option is used to specify the amount of bytes of data to be captured for each system event. In this example, we are filtering events for the mongod process.

n

$ sudo sysdig -s 3000 -w trace.scaprn$ sudo sysdig -r trace.scap proc.name=mongodrn

n

Create MongoDB Trace File
Create MongoDB Trace File

n

Monitoring Linux Processes

n

To list system processes, type:

n

$ sudo sysdig -c psrn

n

Monitor Linux Processes
Monitor Linux Processes

n

Monitor Processes by CPU Usage

n

To watch top processes by CPU usage percentage, run this command:

n

$ sudo sysdig -c topprocs_cpurn

n

Monitor Processes by CPU Usage
Monitor Processes by CPU Usage

n

Monitoring Network Connections and I/O

n

To view system network connections, run:

n

$ sudo sysdig -c netstatrn

n

Monitor Network Connections
Monitor Network Connections

n

The following command will help you list top network connections by total bytes:

n

$ sudo sysdig -c topconnsrn

n

Next, you can also list top processes by network I/O as follows:

n

$ sudo sysdig -c topprocs_net    rn

n

Monitoring System File I/O

n

You can output the data read and written by processes on the system as below:

n

$ sudo sysdig -c echo_fdsrn

n

Monitor System IO
Monitor System IO

n

To list top processes by (read + write) disk bytes, use:

n

$ sudo sysdig -c topprocs_file   rn

n

Troubleshooting a Linux System Performance

n

To keep an eye on system bottlenecks (slow system calls), execute this command:

n

$ sudo sysdig -c bottlenecksrn

n

Troubleshoot Linux Performance
Troubleshoot Linux Performance

n

Track Execution Time of a Process

n

To track the execution time of a process, you can run this command and dump the trace in a file:

n

$ sudo sysdig -w extime.scap -c proc_exec_time rn

n

Track Process Execution Time
Track Process Execution Time

n

Then use a filter to zero down on details of a particular process (postgres in this example) as follows:

n

$ sudo sysdig -r extime.scap proc.name=postgresrn

n

Discover Slow Network I/0

n

This simple command will help you detect slow network I/0:

n

$ sudo sysdig -c netlower     rn

n

Watching Log File Entries

n

The command below helps you display every message written to syslog, if you are interested in log entries for a specific process, create a trace dump and filter it out accordingly as shown before:

n

$ sudo sysdig -c spy_syslog      rn

n

You can print any data written by any process to a log file as follows:

n

$ sudo sysdig -c spy_logs   rn

n

Monitoring HTTP Server Requests

n

If you have a HTTP server such as Apache or Nginx running on our system, look through the server’s requests log with this command:

n

$ sudo sysdig -c httplog    rn$ sudo sysdig -c httptop   [Print Top HTTP Requests] rn

n

Monitor HTTP Requests
Monitor HTTP Requests

n

Display Login Shells and Interactive User Activity

n

The command below will enable you view all the login shell IDs:

n

$ sudo sysdig -c list_login_shellsrn

n

Last but not least, you can show interactive activity of system users like so:

n

$ sudo sysdig -c spy_usersrn

n

Monitor User Activity
Monitor User Activity

n

For more usage information and examples, read the sysdig and csysdig man pages:

n

$ man sysdig rn$ man csysdigrn

n

Reference: https://www.sysdig.org/

n

Also check these useful Linux performance monitoring tools:

n

    n

  1. BCC – Dynamic Tracing Tools for Linux Performance Monitoring, Networking and More
  2. n

  3. pyDash – A Web Based Linux Performance Monitoring Tool
  4. n

  5. Perf- A Performance Monitoring and Analysis Tool for Linux
  6. n

  7. Collectl: An Advanced All-in-One Performance Monitoring Tool for Linux
  8. n

  9. Netdata – A Real-Time Performance Monitoring Tool for Linux Systems
  10. n

n

Conclusion

n

Sysdig brings together functionalities from numerous command line tools into one remarkable interface, thus allowing you to dig deep into your Linux system events to gather data, save for later analysis and it offers incredible container support.

n

To ask any questions or share any thoughts about this tool, use the feedback form below.

n

‘]