Performance related Commands to monitor system performance:

1) Find the process that uses most CPU
ps -eo pcpu,pid,user,args | sort -k 1 -r | head -2
(eg) output of the above command when websphere server process is utilizing the most of the CPU

%CPU PID USER COMMAND
71.7 31237 root /cust/IBM/WebSphere/AppServer/java/bin/java
-Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere

/AppServer/java/jre/lib/ext/ibmext.jar
-Dwas.status.socket=54859 -classpath

/cust/IBM/WebSphere/AppServer/profiles/AppSrv02/properties:/cust/IBM/WebSphere/AppSer

2) Find the last 10 process that use the most CPU
ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10

%CPU PID USER COMMAND
2.1 6375 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=2132 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppServ
0.3 13886 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=25042 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppSer
0.2 17271 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=9606 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppServ
0.2 17871 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=9606 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppServ
0.2 17713 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=9140 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppServ
0.1 18499 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=11309 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppSer
0.1 17504 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=8969 -classpath /cust/IBM/WebSphere/AppServer/profiles/Dmgr01/properties:/cust/IBM/WebSphere/AppServer

3)Find cpu usage
mpstat -P ALL or mpstat

Linux 2.4.21-27.ELsmp (machine1.www.com) 02/02/2008
11:55:00 AM CPU %user %nice %system %iowait %irq %soft %idle intr/s
11:55:00 AM all 1.51 0.00 0.41 0.04 0.01 0.09 97.94 184.92
11:55:00 AM 0 1.50 0.00 0.41 0.04 0.01 0.11 97.93 123.76
11:55:00 AM 1 1.51 0.00 0.42 0.04 0.00 0.07 97.95 61.16

4)Find CPU usage with the process listing in runtime and much more statistics in terms of memory, virtual memory, etc.
top

12:00:20 up 118 days, 15:16, 2 users, load average: 0.07, 0.02, 0.00
572 processes: 570 sleeping, 2 running, 0 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 2.7% 0.0% 1.1% 0.0% 0.0% 0.0% 95.9%
cpu00 1.9% 0.0% 0.3% 0.0% 0.0% 0.0% 97.6%
cpu01 3.5% 0.0% 1.9% 0.1% 0.0% 0.0% 94.2%
Mem: 11819592k av, 5088092k used, 6731500k free, 0k shrd, 295304k buff
3998132k active, 682312k inactive
Swap: 2048276k av, 0k used, 2048276k free 1258592k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
17719 root 16 0 306M 306M 3992 S 0.7 2.6 22:14 1 java
19358 root 16 0 2532 2532 896 R 0.5 0.0 0:00 1 top
17771 root 15 0 306M 306M 3992 S 0.4 2.6 3:48 0 java
18058 root 16 0 458M 458M 4032 S 0.1 3.9 12:55 0 java
1 root 15 0 512 512 452 S 0.0 0.0 1:19 0 init
2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0

5)Continuous CPU usage monitoring and find the average CPU used.
a) sar -u 2 5
b) sar -o output.file 12 8 >/dev/null 2>&1 &
c) nohup sar -o output.file 12 8 >/dev/null 2>&1 &

Linux 2.4.21-27.ELsmp (machine1.web.com) 02/02/2008

12:04:34 PM CPU %user %nice %system %iowait %idle
12:04:36 PM all 0.25 0.00 0.00 0.00 99.75
12:04:38 PM all 0.00 0.00 0.50 0.25 99.25
12:04:40 PM all 7.00 0.00 7.25 0.00 85.75
12:04:42 PM all 21.05 0.00 1.50 0.00 77.44
12:04:44 PM all 3.00 0.00 0.50 0.00 96.50
Average: all 6.26 0.00 1.95 0.05 91.74