I’m running something on my Raspberry Pi server that’s got a memory leak. I think it’s related to my Software Defined Radio (more on that another day), but I’m too lazy to actually track it down and fix it, so I’ve implemented the below hack to just restart my server when memory gets too low.
#!/bin/bash
FREE_SWAP=$(free | grep Swap | awk '{print $4/$2 * 100.0}')
FREE_MEM=$(free | grep Mem | awk '{print $7/$2 * 100.0}')
if [ -t 1 ]; then
echo "Free mem: ${FREE_MEM}%, Free swap: ${FREE_SWAP}%"
fi
if [ 1 -gt "${FREE_SWAP%.*}" ]; then
echo "[$(date)] SWAP IS TOO LOW! AT $FREE_SWAP, CHECKING MEM"
if [ 5 -gt "${FREE_MEM%.*}" ]; then
echo "[$(date)] SWAP IS TOO LOW! AT $FREE_SWAP, MEM IS TOO LOW, AT $FREE_MEM -- RESTARTING"
reboot
fi
fi
if [ 1 -gt "${FREE_MEM%.*}" ]; then
echo "[$(date)] MEM IS TOO LOW! AT $FREE_MEM -- RESTARTING"
reboot
fiCode language: PHP (php)If the free swap memory gets below 1%, we check physical memory. If physical memory is below 5%, we reboot.
We also do a general check on physical memory, and if it ever gets below 1%, we reboot.
So far, this has been working out well. I’ve thrown it in a cron to run every five minutes:
# Check memory every five minutes and reboot if too low
*/5 * * * * /usr/local/bin/memcheck >> /var/log/memcheck.logCode language: PHP (php)You can see, checking my log, the number of times it’s saved my butt from having to fix a frozen machine:
$ cat /var/log/memcheck.log | ack RESTARTING
[Mon Feb 7 03:20:02 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 4.44943 -- RESTARTING
[Tue Feb 15 03:15:33 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 3.79613 -- RESTARTING
[Mon Feb 21 14:30:01 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 2.31316 -- RESTARTING
[Sun Feb 27 01:40:01 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 4.02053 -- RESTARTING
[Mon Mar 7 00:05:01 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 4.95096 -- RESTARTING
[Wed Mar 9 00:55:01 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 4.81816 -- RESTARTING
[Wed Mar 9 17:50:02 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 4.72946 -- RESTARTING
[Tue Mar 15 04:10:01 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 4.79014 -- RESTARTING
[Sun Mar 20 16:20:01 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 4.95519 -- RESTARTING
[Fri Mar 25 14:10:02 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 4.98074 -- RESTARTING
[Tue Mar 29 15:05:01 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 1.63566 -- RESTARTING
[Tue Apr 5 16:55:01 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 4.44242 -- RESTARTING
[Fri Apr 8 01:40:55 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 1.23014 -- RESTARTING
[Fri Apr 8 01:56:41 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 1.00451 -- RESTARTING
[Mon Apr 11 03:15:02 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 3.76512 -- RESTARTING
[Thu Apr 14 17:45:03 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 2.02634 -- RESTARTING
[Wed Apr 27 17:15:02 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 4.55957 -- RESTARTING
[Fri Apr 29 14:55:02 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 3.96232 -- RESTARTING
[Mon May 2 11:00:02 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 2.62884 -- RESTARTING
[Sat May 7 11:50:01 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 4.97909 -- RESTARTING
[Wed May 11 08:00:02 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 4.83372 -- RESTARTING
[Tue May 31 15:00:02 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 3.5021 -- RESTARTING
[Thu Jun 16 00:20:02 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 2.98098 -- RESTARTING
[Tue Jul 5 16:10:08 UTC 2022] SWAP IS TOO LOW! AT 0.0030518, MEM IS TOO LOW, AT 4.10944 -- RESTARTING
[Tue Aug 9 04:10:20 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 3.11883 -- RESTARTING
[Wed Aug 17 14:45:03 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 4.20299 -- RESTARTINGCode language: JavaScript (javascript)
Leave a Reply