How to troubleshoot high memory usage in linux

How to find what is consuming memory on a Linux server, and the kernel overcommit settings that control it.

Identifying the “Out of Memory” scenario.

The pressure may be over by the time you look, so check the kernel log for OOM kills first: Linux kernel manages the server memory by killing tasks/processes based on some process criteria and release the memory footprint occupied by the killed process. 

A log entry will be generated for every such process terminated by the kernel and the logs are usually available in /var/log/ location. 

You can use the below grep command to search in all log files in /var/log/ location for an out of memory error.

grep -i -r 'out of memory' /var/log/
Mar 1 01:24:05 srv kernel: [3966348.114233] Out of memory: Kill process 48305 (php-cgi) score 21 or sacrifice child
Mar 1 01:24:08 srv kernel: [3966350.061954] Out of memory: Kill process 48318 (php-cgi) score 21 or sacrifice child

Current usage:

root@srv:~# free -m 
              total        used        free      shared  buff/cache   available
Mem:     1981         720         319         138         940         916
Swap:        0           0           0

The history of memory usage for the day can be found by using the “sar” command.

root@server [~]# sar -r
Linux (server.example.com)       03/05/2019      _x86_64_        (2 CPU)
12:00:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit
12:10:01 AM   1672764   1407160     45.69    100356    749140   1618364     44.90
12:20:01 AM   1289208   1790716     58.14    106580   1130096   1599588     44.38
12:30:01 AM   1248100   1831824     59.48    109184   1144680   1621332     44.98
12:40:01 AM   1267972   1811952     58.83    111460   1155828   1604104     44.51
12:50:01 AM   1254556   1825368     59.27    113888   1159632   1599480     44.38
01:00:01 AM   1092296   1987628     64.53    116020   1164540   1802228     50.00
01:10:01 AM   1212168   1867756     60.64    118204   1169516   1633940     45.33
...
Average:    kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit
Average:      1465222   1614702        52.43          179213    834889      1655342     45.93

A RAM upgrade is necessary if the server shows consistent high memory usage or the average usage for the day is more than 90% as such a high usage can deplete the available free memory at times for a busy server. On a virtual machine, sustained memory pressure is solved with more RAM or by fixing the application, never with swap.

Another handy tool to identify the Memory consuming processes is the “top” command, which will give you the option to sort the running processes based on its resource usages.

root@server [~]# top -c
top - 18:41:45 up 109 days, 18:03,  5 users,  load average: 1.30, 1.24, 1.24
Tasks: 544 total,   2 running, 541 sleeping,   0 stopped,   1 zombie
Cpu(s):  8.3%us,  1.4%sy,  0.2%ni, 89.8%id,  0.3%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:  12296096k total, 11531800k used,   764296k free,   586732k buffers
Swap: 16777212k total,   209160k used, 16568052k free,  2471072k cached
    PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                    
2376001 root      20   0  230m 101m 4588 R 31.4  0.8   1:22.06 spamd child                                                                                                                 
2448741 root      30  10 19472 1056  792 S  2.6  0.0   0:00.08 /usr/sbin/lveps -c 1 -p -d -n -o id:10,ep:10,pno:10,pid:15,tno:5,tid:15,cpu:7,mem:15,com:256                                
 953572 mysql     20   0 17.5g 5.3g 5336 S  1.7 44.9 102:38.10 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=server
 844549 root      30  10  424m  59m 4532 S  1.3  0.5  52:13.20 /opt/alt/python27/bin/python2.7 /usr/share/lve-stats/lvestats-server.py start --pidfile /var/run/lvestats.pid               
1351883 root      20   0 1127m 638m 1768 S  1.3  5.3  37:37.24 /usr/local/cpanel/3rdparty/bin/clamd                                                                                        
 844526 root      30  10  462m  35m 1204 S  1.0  0.3  30:54.20 /opt/alt/python27/bin/python2.7 /usr/share/lve-stats/lvestats-server.py start --pidfile /var/run/lvestats.pid               
   3109 nscd      20   0 2725m 3956 2260 S  0.7  0.0 188:08.43 /usr/sbin/nscd                                                                                                              
2243573 nobody    20   0  189m  78m 1140 S  0.7  0.7   4:22.20 litespeed (lshttpd - #01)                                                                                                   
2448384 mailnull  20   0 78756 7736 3348 S  0.7  0.1   0:00.06 /usr/sbin/exim -bd -q1h -oP /var/spool/exim/exim-daemon.pid

You can follow the below key patterns to sort the processes based on its memory usage.

Enter the command top Press SHIFT+o to get the top command options. Press N and enter

Identifying Memory Leaks 

You can find the server time at which this ‘out of memory’ was reported, here it is “Mar 1 01:24:05”. Use grep command to search this timestamp in the log files for your application servers like Apache, MySQL etc.

For a cPanel server, you can use the grep command to search the website access logs to see any suspicious/abusive access to the website to cause this resource exhaustion.

grep  -ir “01/Mar/2019:01:2”  /usr/local/apache/domlogs/

Some suspicious activities identifiable from the access logs are 

1. High access from specific IP addresses. 
2. High access to unavailable resources/files etc. 
3. High number of HTTP POST requests. 
4. High number of failed access attempts like login.

Use the command mysqladmin proc stat to identify any MySQL queries hanging for a long time to cause a high memory usage for them.

root@server [~]# mysqladmin proc stat
+--------+------+-----------+----+---------+------+-------+------------------+----------+
| Id     | User | Host      | db | Command | Time | State | Info             | Progress |
+--------+------+-----------+----+---------+------+-------+------------------+----------+
| 137324 | root | localhost |    | Query   | 0    | init  | show processlist | 0.000    |
+--------+------+-----------+----+---------+------+-------+------------------+----------+
Uptime: 370827  Threads: 1  Questions: 20484133  Slow queries: 0  Opens: 1456  Flush tables: 1  Open tables: 747  Queries per second avg: 55.239

Check the columns Time, db, state , find the queries with a high “Time” entry and check them with your website developer. 

Enable “Slow Query Logging” in MySQL and fix the long queries with your website developer.

Memory Overcommit 

Usually, the Linux server will allow more memory to be reserved for a process than its actual requirement, this is based on the assumption that no process will use all the memory allowed for it which can be used for other processes. It is possible to configure the Linux system how it handles the memory overcommit and the configuration can be applied using the sysctl utility.

All sysctl control parameters can be listed using the command sysctl -a and the parameter of our interest is vm.overcommit_memory.

[root@srv ~]# sysctl -a | grep vm.overcommit_memory
vm.overcommit_memory = 0

vm.overcommit_memory can have 3 values 0,1 and 2.

• 0 Allow overcommit based on the estimate “if we have enough RAM” 
• 1 Always allow overcommit 
• 2 Deny overcommit if the system doesn’t have the memory.

Another sysctl parameter to consider is the overcommit_ratio, which defines what percentage of physical memory in addition to the swap space can be considered when allowing processes to overcommit. 

Take the case if we set vm.overcommit_memory to 2, the kernel will not allow overcommit exceeding the swap space plus vm.overcommit_ratio of the total RAM space. Our virtual machines run without swap by design, so on a VM the swap term is zero: vm.overcommit_memory=2 vm.overcommit_ratio=50% RAM=8GB SWAP=0

With the above-mentioned configuration, processes can commit only 50% of 8GB RAM, which is 4GB, and allocations beyond that fail even while RAM is still free. On a VM, keep vm.overcommit_memory at 0 unless your application needs strict accounting; if it does, set overcommit_ratio to 100 so the whole RAM can be committed. A workload that needs more memory than that needs more RAM, a larger plan or a tuned application.

sysctl -w vm.overcommit_memory=2
sysctl -w vm.overcommit_ratio=100

sysctl -w does not survive a reboot; put the same two lines in /etc/sysctl.conf.