The command I normally have been using to check the server performance is the good-old ‘top’ command. Gives you a very good snapshot of the things going on in your server.

However, today when I issued the top command I found that the CPU load was around 16.23 but none of the processes were consuming as much of the CPU. Moreover, all the CPUs showed that they were 0% idle. This is strange, idle time is 0%, CPU average load is 16.23 whereas none of the processes were really consuming as much of the CPU. So where’s the problem and who’s causing it?

While I was searching on several forums for help, I found a nice blog post here and here that gives you a list of very good commands to check a variety of stats. You can read up full details in there, however, I would just list those commands in here for future reference. The commands of interest were:

• mpstat
• sar
• ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10
• iostat
• vmstat

If you understand the purpose of each of the above command, then it would give you a very good understanding of what your server is undergoing at that very precise moment.

Oh and another command very useful was ‘watch -n1’. If you add ‘watch -n1’ to any of the above listed commands then you would be able to monitor the update to the output of those commands as ‘watch’ runs the command at specified time delay of 1 second.

By the way, at the end of doing the above exercise, I wasn’t able to find the processes or the reason why the CPU load was 16.23 average while each of the CPU was 0% idle. The server was responding fine with the requests and hence I left it alone. Didn’t find the root cause of the problem. I think that some how one of the variables on which top depends was some how messed up. Also, I was pretty sure that this problem will go away once I reboot the server, but I don’t want to reboot server without really having a need to.

Hope, this post would give you some insight in finding more details on performance and stats for your server. If it does or even if it doesn’t, please leave a comment below 🙂