Tag: monitoring

  • Sending Prowl Alerts via Bash

    Sending Prowl Alerts via Bash

    I’m working on some server scripting and I wanted to find a way to make sure I get proper alerts. I tried hard getting Twilio working cheaply, but that failed. So I remembered that I use Prowl for other things on my phone–so why not just send more alerts that way?

    So, again with the help of ChatGPT I have made a terrible monstrosity:

    #!/bin/bash
    
    # Exit if any command fails, if an unset variable is used, or if a command in a pipeline fails
    set -euo pipefail
    
    # Default values
    application="Shell Notification"
    priority=0
    event="Event"
    message=""
    logfile="/var/log/send-prowl-alert.log"
    url=""
    
    # Usage information
    usage() {
        echo "Usage: $0 --message=<message> [--application=<application>] [--event=<event>] [--description=<description>] [--priority=<priority>] [--url=<url>]"
        echo "Required:"
        echo "  --message=<message>       A description of the event, generally terse"
        echo "                            Maximum of 10000 bytes"
        echo "Optional:"
        echo "  --application=<application>  The name of the application (default: 'Shell Notification')"
        echo "                               Maximum of 256 bytes"
        echo "  --event=<event>              The name of the event or subject of the notification (default: 'Event')"
        echo "                               Maximum of 1024 bytes"
        echo "  --priority=<priority>        The priority of the alert (default: 0)"
        echo "                               An integer value ranging [-2, 2] representing:"
        echo "                                   -2 Very Low"
        echo "                                   -1 Moderate"
        echo "                                    0 Normal"
        echo "                                    1 High"
        echo "                                    2 Emergency"
        echo "                               Emergency priority messages may bypass quiet hours according to the user's settings."
        echo "  --url=<url>                  The URL which should be attached to the notification"
        echo "                               Maximum of 512 bytes"
        echo "  --help                       Displays this help message"
    }
    
    # URL Encodes a string
    urlencode() {
      local string="$1"
      local length="${#string}"
      local i
    
      for ((i = 0; i < length; i++)); do
        local c="${string:i:1}"
        case $c in
          [a-zA-Z0-9.~_-])
            printf '%s' "$c"
            ;;
          *)
            printf '%%%02X' "'$c"
            ;;
        esac
      done
    }
    
    # Check if the script is being run manually or via cron
    if [[ -t 1 ]]; then
        # Output to console if being run manually
        log_message() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*"; }
    else
        # Output to log file if being run via cron
        log_message() {
            echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" >> "${logfile}"
            # Truncate the log file to 1 megabyte
            if [[ $(wc -c <"${logfile}") -gt 1000000 ]]; then
                tail -c 1000000 "${logfile}" > "${logfile}.tmp" && mv "${logfile}.tmp" "${logfile}"
            fi
        }
    fi
    
    # Parse arguments
    for arg in "$@"; do
        case "${arg}" in
            --message=*)
                message=${arg#*=}
                ;;
            --application=*)
                application=${arg#*=}
                ;;
            --event=*)
                event=${arg#*=}
                ;;
            --priority=*)
                priority=${arg#*=}
                ;;
            --url=*)
                url=${arg#*=}
                ;;
            --help)
                usage
                exit 0
                ;;
            *)
                echo "Error: Unsupported argument ${arg}"
                usage
                exit 1
                ;;
        esac
    done
    
    
    # Create a variable to store error messages
    errors=""
    
    # Check if curl is installed
    if ! command -v curl &> /dev/null; then
        errors+="Error: curl is not installed. Please install curl and try again.\n"
    fi
    
    # Check that a message was provided
    if [[ -z "${message}" ]]; then
        errors+="Error: No message provided."
    fi
    
    # Check if the URL is valid
    if [[ -n $url && ! $url =~ ^(https?://)[^\s/$.?#].[^\s]*$ ]]; then
        errors+="Error: Invalid URL.\n"
    fi
    
    # Check byte size of the parameters and append error if they exceed the max limit
    if [[ "${#url}" -gt 512 ]]; then
        errors+="Error: URL exceeds maximum byte limit (512 bytes).\n"
    fi
    
    if [[ "${#application}" -gt 256 ]]; then
        errors+="Error: Application name exceeds maximum byte limit (256 bytes).\n"
    fi
    
    if [[ "${#event}" -gt 1024 ]]; then
        errors+="Error: Event name exceeds maximum byte limit (1024 bytes).\n"
    fi
    
    if [[ "${#message}" -gt 10000 ]]; then
        errors+="Error: Message exceeds maximum length (10000 bytes).\n"
    fi
    
    if [[ ! "${priority}" =~ ^-?[0-2]$ ]]; then
        errors+="Error: Invalid priority. Must be an integer between -2 and 2.\n"
    fi
    
    # Check if the API key file exists and is not empty
    if [[ ! -s ~/.prowl_api_key ]]; then
        errors+="Error: The Prowl API key file does not exist or is empty. Please create a file at ~/.prowl_api_key and add your API key to it.\nExample:\n  echo 'your_api_key' > ~/.prowl_api_key\n  chmod 600 ~/.prowl_api_key\n"
    fi
    
    # Check the permissions on the API key file, if it exists
    # We have to do it in a convoluted cross platform way because macos does not support `stat -c`
    if [[ -f ~/.prowl_api_key ]]; then
        # Retrieve the file permissions and remove any trailing '@' symbol (indicating extended attributes)
        permissions=$(ls -l -d ~/.prowl_api_key | awk '{print $1}' | tr -d '@')
    
        # Check if the permissions are not set to '-rw-------'
        if [[ "$permissions" != "-rw-------" ]]; then
            # Add an error message to the 'errors' variable
            errors+="Error: The permissions on the API key file are not set correctly. Please run 'chmod 600 ~/.prowl_api_key' and try again.\n"
        fi
    fi
    
    # If any errors were detected, print the error messages and exit
    if [[ -n $errors ]]; then
        echo -e "${errors}"  # -e enables interpretation of backslash escapes like \n
        exit 1
    fi
    
    # Get the Prowl API key
    prowl_api_key=$(cat ~/.prowl_api_key)
    
    # Save the alert data for logging before we encode it
    alert_log_string="$application : $event : $priority : $url : $message"
    
    # URL-encode the variables
    prowl_api_key=$(urlencode "$prowl_api_key")
    application=$(urlencode "$application")
    event=$(urlencode "$event")
    message=$(urlencode "$message")
    priority=$(urlencode "$priority")
    url=$(urlencode "$url")
    
    # Send the alert
    response=$(curl -s "https://api.prowlapp.com/publicapi/add?apikey=${prowl_api_key}&application=${application}&event=${event}&description=${message}&priority=${priority}&url=${url}")
    
    # Check if the alert was sent successfully
    if [[ "${response}" != *"success code=\"200\""* ]]; then
        log_message "Error sending alert: ${alert_log_string}"
        log_message "Response: ${response}"
        exit 1
    fi
    
    log_message "Alert sent successfully: $alert_log_string"
    exit 0
    Code language: PHP (php)

    Good luck!

  • Bad Hack: Restart Linux Server when memory is low 😬

    Bad Hack: Restart Linux Server when memory is low 😬

    I’m running something on my Raspberry Pi server that’s got a memory leak. I think it’s related to my Software Defined Radio (more on that another day), but I’m too lazy to actually track it down and fix it, so I’ve implemented the below hack to just restart my server when memory gets too low.

    #!/bin/bash
    FREE_SWAP=$(free | grep Swap | awk '{print $4/$2 * 100.0}')
    FREE_MEM=$(free | grep Mem | awk '{print $7/$2 * 100.0}')
    
    if [ -t 1 ]; then
            echo "Free mem: ${FREE_MEM}%, Free swap: ${FREE_SWAP}%"
    fi
    
    if [ 1 -gt "${FREE_SWAP%.*}" ]; then
            echo "[$(date)] SWAP IS TOO LOW! AT $FREE_SWAP, CHECKING MEM"
            if [ 5 -gt "${FREE_MEM%.*}" ]; then
                    echo "[$(date)] SWAP IS TOO LOW! AT $FREE_SWAP, MEM IS TOO LOW, AT $FREE_MEM -- RESTARTING"
                    reboot
            fi
    fi
    
    if [ 1 -gt "${FREE_MEM%.*}" ]; then
            echo "[$(date)] MEM IS TOO LOW! AT $FREE_MEM -- RESTARTING"
            reboot
    fiCode language: PHP (php)

    If the free swap memory gets below 1%, we check physical memory. If physical memory is below 5%, we reboot.

    We also do a general check on physical memory, and if it ever gets below 1%, we reboot.

    So far, this has been working out well. I’ve thrown it in a cron to run every five minutes:

    # Check memory every five minutes and reboot if too low
    */5 * * * * /usr/local/bin/memcheck >> /var/log/memcheck.logCode language: PHP (php)

    You can see, checking my log, the number of times it’s saved my butt from having to fix a frozen machine:

    $ cat /var/log/memcheck.log | ack RESTARTING
    [Mon Feb  7 03:20:02 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 4.44943 -- RESTARTING
    [Tue Feb 15 03:15:33 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 3.79613 -- RESTARTING
    [Mon Feb 21 14:30:01 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 2.31316 -- RESTARTING
    [Sun Feb 27 01:40:01 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 4.02053 -- RESTARTING
    [Mon Mar  7 00:05:01 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 4.95096 -- RESTARTING
    [Wed Mar  9 00:55:01 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 4.81816 -- RESTARTING
    [Wed Mar  9 17:50:02 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 4.72946 -- RESTARTING
    [Tue Mar 15 04:10:01 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 4.79014 -- RESTARTING
    [Sun Mar 20 16:20:01 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 4.95519 -- RESTARTING
    [Fri Mar 25 14:10:02 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 4.98074 -- RESTARTING
    [Tue Mar 29 15:05:01 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 1.63566 -- RESTARTING
    [Tue Apr  5 16:55:01 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 4.44242 -- RESTARTING
    [Fri Apr  8 01:40:55 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 1.23014 -- RESTARTING
    [Fri Apr  8 01:56:41 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 1.00451 -- RESTARTING
    [Mon Apr 11 03:15:02 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 3.76512 -- RESTARTING
    [Thu Apr 14 17:45:03 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 2.02634 -- RESTARTING
    [Wed Apr 27 17:15:02 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 4.55957 -- RESTARTING
    [Fri Apr 29 14:55:02 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 3.96232 -- RESTARTING
    [Mon May  2 11:00:02 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 2.62884 -- RESTARTING
    [Sat May  7 11:50:01 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 4.97909 -- RESTARTING
    [Wed May 11 08:00:02 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 4.83372 -- RESTARTING
    [Tue May 31 15:00:02 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 3.5021 -- RESTARTING
    [Thu Jun 16 00:20:02 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 2.98098 -- RESTARTING
    [Tue Jul  5 16:10:08 UTC 2022] SWAP IS TOO LOW! AT 0.0030518, MEM IS TOO LOW, AT 4.10944 -- RESTARTING
    [Tue Aug  9 04:10:20 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 3.11883 -- RESTARTING
    [Wed Aug 17 14:45:03 UTC 2022] SWAP IS TOO LOW! AT 0, MEM IS TOO LOW, AT 4.20299 -- RESTARTINGCode language: JavaScript (javascript)
  • Quick Tip: Looping a command in bash

    Quick Tip: Looping a command in bash

    I recently came across the need to watch my disk space while running a slow program to make sure I didn’t run out. If I did, the command would fail silently and I’d have to start over again.

    This can easily be done with this short snippet:

    watch -n10 bash -c "df -h | ack disk1s5"Code language: JavaScript (javascript)
    Every 10.0s: bash -c df -h | ack disk1s5                                                                                  mbp.local: Mon Jul 20 15:09:51 2020
    
    /dev/disk1s5           488245288  10940872 144033352     8%   488237 4881964643    0%   /

    The important part here is disk1s5, which is the device file for the partition I wanted to watch. If you need to find this, it can be done simply by running the df as a whole:

    $ df
    Filesystem             1K-blocks      Used Available Capacity  iused      ifree %iused  Mounted on
    /dev/disk1s5           488245288  10940872 144035124     8%   488237 4881964643    0%   /
    devfs                        191       191         0   100%      662          0  100%   /dev
    /dev/disk1s1           488245288 331456068 144035124    70%  1379027 4881073853    0%   /System/Volumes/Data
    /dev/disk1s4           488245288   1048596 144035124     1%        1 4882452879    0%   /private/var/vm
    map auto_home                  0         0         0   100%        0          0  100%   /System/Volumes/Data/home
    /dev/disk1s3           488245288    516448 144035124     1%       48 4882452832    0%   /Volumes/RecoveryCode language: PHP (php)

    That is all.

    Photo by Wendy Wei from Pexels

  • Page Generation Graph for WordPress

    Page Generation Graph for WordPress

    At work, one of the more interesting customizations we have on WordPress.com for our VIP clients is a dashboard that contains custom widgets.  One of them is a page generation graph that shows the average page generation time for their site compared to all others.  That way they can judge their code performance against a good baseline.

    (more…)