UNIX: commands that can be used for performance-related issues

Here are summed up commands that can help identify performance related issues in UNIX systems:

AIX
The %CPU is the percentage of CPU time that has been allocated to that process since the process was started.
#ps aux

Display CPU usage by process and thread id. The TID column shows the threadID:
#ps -efmo THREAD
#tprof -skex sleep 60

Display processes with the highest CPU utilization:
#ps -eo pid,pcpu,args | sort +1n

To find the threadid (tid) of a known process that is using CPU:
#ps -mp -o THREAD

Display processes with the highest memory usage:
#ps -eo pid,vsz,args | sort +1n

To see your server utilization in IBM AIX including top processes, CPU usage, memory, virtual memory, paging space, I/O and load:

# topas

Type for help
Type to quit

To list the top ten users of paging space in IBM AIX:
#svmon -Pgt 10 (paging)

To list the top ten users of realmem in IBM AIX:
#svmon -Put 10

To find memory usage:
#svmon -u | more
#svmon -P | more


LINUX

Show a breakdown of utilization by an individual processor. The command shows the kernel level, user CPU, system CPU, nice time, idle time, wait time and interrupts per second. Similar data can be obtained with the sar command.
#mpstat

Display five reports of statistics for all processors at two second intervals, enter:
# mpstat -P ALL 2 5

List processes by % CPU usage:
#ps -e -o pcpu,cpu,nice,state,cputime,args

Displays the top ten CPU users on the Linux system:
#ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10
OR
#ps -eo pcpu,pid,user,args | sort -r -k1 | less

List all threads for a particular process:
#ps -C -m -o pid,tid,pcpu,state

List processes by memory usage:
#ps -e -orss=,args= | sort -b -k1,1n | pr -TW$COLUMNS

Show the amount of (remaining) RAM (-m displays in MB):
#free -m

An interactive tool that allows a system administrator to view the process table in order of CPU or memory usage, by user, and at varying refresh rates in real-time.

#top

Solaris:

Show all top threads to determine what could be using the most CPU:
#prstat -Lmc -p
#prstat -L -p 1 1

To show the PID, user, state, and thread id:
#ps -Le -o pid,user,s,lwp,pcpu,args | awk '$3 != "S" { print }'

Reports paging activity details for applications (executables), data (anonymous) and filesystem activity:
#vmstat -p

Prints out details of memory use by a process:
#pmap -x

To find how much disk space is used by users in kilobytes in Solaris:
#quot -af


Commands that can be used on most UNIX platforms

Running iostat provides much information, but the values of concern are %user and %sys. If (%user + %sys) > 80 percent over a period of time, then it is likely the bottleneck is CPU. In particular, it is necessary to watch for average CPU being greater than 70 percent with peaks above 90 percent.

#iostat

To report system-wide process use, swapping, memory use, disk I/O and CPU use:
#vmstat

To provides statistics on the average length of the run queue, the percentage of time the run queue is occupied, the average length of the swap queue and the percentage of time the swap queue is occupied.

#sar

Breaks the time into user, system, time waiting for blocked I/O (i.e., NFS, disk, etc.) and idle time.
#sar -u
#sar -q
#sar -k

One advantage to using sar is that you can write the data to a file and then post-process it when the system is not busy. The file is created using the -a and -o flags. An example of creating a file of 30 snapshots, each two seconds apart, would look like:

#sar a o sardata.out 2 30

This file would then be processed using:
#sar -u -f sardata.out

NMON for AIX and Linux


For additional information on the above commands (and more), refer to the appropriate links below:

For AIX:
Performance management: AIX 5L Version 5.3

For Linux:
How do I find out Linux CPU utilization?

For Solaris:
Why should I monitor Processes?

Related information

Identifying possible performance bottlenecks on UNIX wh