Tag: performance

  • But Can It Run E1 Doom?

    But Can It Run E1 Doom?

    I have a Reolink security camera called the E1 Zoom.

    E1 Zoom… E1 Doom?

    You see the problem.

    Doom’s first episode is E1, the first level is E1M1, and the camera is an E1, so the jokes wrote themselves. And I guess so did the code.


    Confidently wrong AI garbo, as usual.

    I had four requirements:

    1. Video
    2. Controls
    3. Audio
    4. DEMONS

    I also made up a few rules to keep myself honest, because otherwise I’d have cheated immediately. Doom had to show up in the normal Reolink app as the camera’s normal video stream, which means no running Doom on my machine and pointing the camera at it, and no sneaking in a second video server. The camera had to use its own video and audio encoders, and when I was done it had to go back to being a camera without me pulling the power cable.

    Easy peasy pinky squeezy

    I say “I built” throughout this post pretty generously. I spent a lot of time working with my AI slop buddies, and they did most of the typing. My main contributions were coming up with bad ideas, testing them on the camera, and repeatedly saying things like “no, not like that” and “okay, but can we cut out the over-engineering?”

    It was a healthy working relationship.

    Doom automap of E1M1: Hangar

    E1M1: Hangar

    The camera is a Reolink E1 Zoom E340, and it has two little ARM cores, around 130 MB of RAM, a pile of video hardware, and an SD card (a whole computer, basically). That’s a lot more computer than Doom originally needed, but the original Doom computer also wasn’t busy being a security camera at the same time.

    Doom needed a 386 with 4 MB of RAM (but anyone who was there knew that would have sucked), we have 32x the RAM. The camera processor is 24x faster: 6.65 BogoMips on a 386 compared to our 24 the processor tells us we have (2,091 BogoMips adjusted for dumbflation)

    I’m not covering how I got root on the camera. This story starts after that part already works, and if you want to repeat it, figuring that out is on you.

    I used Woof 15.3.0 as the Doom source port, because it already does widescreen and modern resolutions, and gutting a desktop port turned out to be easier than building a tiny embedded one up. Out went everything that assumes you have a monitor, replaced with a small camera-shaped layer instead.

    For the actual game I used the official Doom v1.9 shareware WAD, so it’s real E1M1, real sounds, real music, real demons. Accept no substitutes.

    Doom automap of E1M2: Nuclear Plant

    E1M2: Nuclear Plant

    The camera already runs several video encoders, and the Reolink app’s Fluent mode uses a 640×360 stream, so the least bad way to do this was to shove Doom into that pipeline before it reaches the encoder.

    Woof renders its frame into shared memory, and a small worker loaded inside Reolink’s own camera process picks that frame up, converts it into the overlay format the camera already understands, and slaps it over the whole video frame.

    Then the camera just does camera stuff. Beep boop. Its own hardware encodes the frame, its own buffers carry it, and whatever Reolink normally does to get video to your phone happens exactly as usual.

    There’s no special Doom video URL anywhere. I open the Kitchen camera in the Reolink app and my kitchen has been replaced by Phobos.

    Perfect.

    Doom automap of E1M3: Toxin Refinery

    E1M3: Toxin Refinery

    My first working video was technically Doom, but it was a 4:3 image sitting in the middle of the camera feed with black bars around it.

    No no, this will not do. The whole camera needed to be Doom.

    So the final setup renders a real 426×200 widescreen view and scales it over the entire camera frame. It isn’t a stretched 320×200 image either, so you actually get the wider field of view and the status bar sits where it should.

    Then I got this:

    The Doom menu and E1M1 sheared diagonally across the camera frame, each row offset further sideways than the one above it.

    Every row was shifted a little farther sideways than the one above it, and it looked like the camera was falling into a very low-budget dimensional portal.

    The image was 426 pixels wide, but each row was aligned to four bytes, so the actual row pitch was 428 bytes. I was advancing 426, which meant every row started two bytes too early.

    Changed 426 to 428. Fixed.

    The demons were fine. The stride was wrong.

    The same Doom menu rendering correctly over E1M1, full width, status bar in place.
    Doom automap of E1M4: Command Control

    E1M4: Command Control

    The E1 Zoom has a pan/tilt/zoom control pad in the app, which is obviously a Doom controller. Up, Down, Left, Right, Zoom to fire, maybe Focus to open doors. This was going to be great!

    Except the Zoom button doesn’t send anything useful, it just opens another zoom menu, and Focus and presets are buried behind more menus too.

    So I had… four buttons.

    I hooked the camera’s own command handler, so when Doom is running, direction presses get intercepted before they reach the motors and turned into key presses instead. The motors are, I think, grateful.

    The extremely good control scheme is:

    • Up/Down: move forward and backward
    • Left/Right: turn
    • Double-tap Up: Fire + Use + Enter, all at once
    • Hold Down for three seconds: return to the camera

    Yes, double-tap Up may fire your gun while opening a door, but nobody is speedrunning this over a security camera stream (please don’t try).

    To start Doom from the app, I also added a secret code:

    Up, Up, Down, Down, Left, Right, Left, Right

    Why not go for the full nerd points.

    There was one extremely Reolink complication. A normal short tap in the iPhone app doesn’t send the direction you tapped, it sends a directionless stop, and only held buttons send real directions. So the code still works for playing, but the listener watching for my secret code is really just seeing eight timed stop events. On screen I enter Up, Up, Down, Down, Left, Right, Left, Right, and inside the camera it’s technically “tap eight arrows.” I have decided this still counts.

    Doom automap of E1M5: Phobos Lab

    E1M5: Phobos Lab

    Four buttons proved the point, but playing Doom that way is terrible, and I still needed menus, weapons, run, fire, use, Escape, and everything else.

    So I built a tiny web server with a keyboard controller on it. On port 666, obviously.

    The page doesn’t carry any video, since I keep the Reolink app open for that. It just captures real key presses and forwards them to Doom, so you get the normal Doom controls, plus buttons to start Doom, return to the camera, and release any stuck keys. I slopped the whole page up to look like a little 1995 deathmatch setup utility.

    The port 666 web controller, styled like a 1995 setup utility, with MOVE, ACTION and MENU key bindings and a RELEASE ALL button.

    Here are the mockups that I didn’t ship because they were just too much.

    Browsers love to lose a key-up event when you switch tabs, so the page releases everything you’re holding the moment it loses focus, and there’s a heartbeat running underneath that. If the heartbeat stops, Doom lets go of everything instead of running face-first into a wall forever.

    This is an unauthenticated toy server running as root on my LAN. It is not an enterprise deployment of Doom port 666. Please be serious.

    Doom automap of E1M6: Central Processing

    E1M6: Central Processing

    Video and controls worked, so now I needed sound.

    The camera microphone already hands off little blocks of raw audio to the camera’s audio encoder, so I added another small hook right before that handoff. When Doom isn’t running the hook does nothing at all and the microphone works normally, and when Doom is running it swaps the microphone audio for Doom audio. The camera’s own encoder and audio stream do the rest.

    For sound effects I wrote (“wrote”) a 16-voice mixer for Doom’s original sound lumps, handling volume, pitch, and resampling. Music runs on a stripped-down emulator of the sound chip Doom’s music was written for in the first place, borrowed from the rp2040-doom project, which uses a lot less CPU than the fancier version I started with.

    A multitrack editor timeline of the camera's audio. The top lane is the mixed 16 kHz mono waveform the camera streamed. Below it, the E1M1 score drawn as a piano roll, melody notes in gold and percussion in gray. Below that, seven lanes of Doom sound effect voices, each clip drawn as the real sample waveform with the sound's name above it, and a final collapsed row for the nine voices the demo never used.
    Now accepting preorders for my mixtape.

    If Doom audio ever runs dry I output silence, because the alternative is accidentally cutting back to the kitchen microphone, and nobody needs hot-mic jump scares in E1M1.

    I spent a while convinced none of this worked, because I still couldn’t hear anything in the app. Turns out the Reolink Home Hub mutes camera audio until you enter listening mode, and opening the control pad can mute it again, all while the camera is happily sending perfectly good Doom audio the entire time.

    I tapped the speaker button. Audio worked.

    Good debugging session.

    Doom automap of E1M7: Computer Station

    E1M7: Computer Station

    At this point the camera could run Doom, so naturally I decided it was using too much CPU. Why? No idea. It ran fine. But the camera still had to do its actual job at the same time, and I wanted Doom under 10% of one core with nothing dropped.

    The biggest fix was embarrassingly obvious. Doom was rendering around 28 frames a second, but the camera stream was only configured for 10, so most of those frames were being thrown away. I had been very carefully optimizing frames that nobody would ever see! Now the runtime reads the camera’s configured frame rate and Doom only renders what the stream can actually use.

    I kept asking whether fancy vector instructions would make it faster, but palette conversion is mostly random lookups and this chip doesn’t have a useful gather instruction, so boring regular ARM instructions won. Slightly disappointing, but… faster is faster.

    The fastest build I got hit 9.87% of one core, or under 5% of the whole camera, so technically I hit the goal (technically). Then I turned the real sound-effect channels back on and raised the music quality, because it sounded better, and the build I actually kept sits at 11.23% instead. Slightly worse benchmark. Noticeably better Doom. Worth it!

    Horizontal bar chart of CPU cost per build, as a percentage of one core. The 640x300 zero-copy baseline sits at 23.1 percent. Successive builds fall to 9.9 percent for the stride-eight build, under the 10 percent goal. One rejected envelope experiment came out worse than the baseline. The build finally kept, stride-four with music and real sound effects, sits at 11.2 percent.
    The last bar is slower on purpose. That’s why it’s yellow, like a school bus.

    I could probably keep optimizing this forever, but I had already spent an afternoon profiling a once-per-second “is Doom still changing?” check on a security camera running Doom, and at some point you have to stop and think about choices you’ve made in life.

    Doom automap of E1M8: Phobos Anomaly

    E1M8: Phobos Anomaly

    Then the camera froze and rebooted. Sigh.

    It happened while I had the controller page open, so keyboard traffic was the obvious suspect. I hammered it for almost two minutes, opening menus, moving, firing, using things, and cycling keys far longer than the original failure had taken, and everything was completely fine (of course it was).

    Well then…

    The useful clue came from a log on the SD card, where some of the camera’s important threads were showing up stopped during cleanup. I had a tool that double-checked I’d restored the camera’s original instructions correctly, and although it only ever read memory, it did that by freezing all twenty-odd threads in the camera process, peeking, and starting them again. Cleanup ran it several times.

    Meanwhile, Reolink’s own supervisor process owns a 60-second hardware watchdog, and it only feeds that watchdog while the camera’s internals look healthy. So: freeze the video and audio threads at exactly the wrong moment, miss a heartbeat, wait 60 seconds… reboot.

     Annotated ARM disassembly of the camera's watchdog feed callback. A health check branches either to the keepalive call, or falls through to an early return that exits while the hardware timer keeps counting down.
    lol, whoops, surprise lockup and reboot

    The timing matched. So I changed the read-only check to peek at memory without freezing anything, and now only the two operations that genuinely rewrite instructions pause the camera at all. The stopped threads disappeared from the logs.

    The original reboot wiped its own logs, so I can’t prove the exact moment it missed that first heartbeat. But the timing lines up, the stopped threads are right there in the log, and it hasn’t happened again since I stopped freezing things.

    The final boss was cleanup code. Sounds about right.

    Doom automap of E1M9: Military Base

    E1M9: Military Base

    I originally packaged everything into a tidy read-only image on the SD card, which was lovely right up until every single test meant uploading the whole thing over my spotty Wi-Fi to change one file. So now it’s just a folder.

    A tiny boot service starts the controller and the secret-code listener, and that’s all it does. It does not start Doom (that’s the point). The camera sits there armed, looking completely normal, until I press Start or enter the code on the control pad, and the game, the renderer, the music, and everything else genuinely don’t exist until I ask for demons. There’s no reason to make a security camera run Doom all day just because it can!

    A firmware update also moved every address I’d carefully found, because apparently I didn’t have enough problems. Hard-coded offsets would have made this a one-firmware trick, so now the runtime finds everything it needs by pattern-matching the camera’s own code at startup, and if anything looks missing, duplicated, or generally wrong, it gives up before touching the camera. This does not magically preserve root through a firmware update, mind you. That’s a separate problem, and I’m still not covering it.

    The service gets exactly one attempt at startup. If a firmware update broke something, it gives up and the camera stays a camera. No heroic restart loop.

    Quitting is the best part. The music cuts out, the original Doom death sound plays, and Doom melts away in columns while my actual kitchen appears underneath it.

    A frame captured mid-melt: Doom dissolving downward in vertical columns, the status bar breaking apart, with the camera view of the ceiling showing through the gaps.

    Then the microphone comes back, the game exits, and the control pad goes back to moving the camera like nothing happened. The controller page stays up, waiting for another bad idea.

    So, can the E1 Zoom run E1 Doom?

    Hell yeah it can! 🤘

    It runs the real shareware Doom in the normal Reolink app, using the camera’s own video and audio encoders. It has music, sound effects, a control-pad Doom controller, a full keyboard controller on port 666, and a proper Doom melt back to my kitchen.

    Video. Controls. Audio. DEMONS!

    The patches are on GitHub: emrikol/e1-doom.

    I have no idea what practical purpose any of this serves. Sigh.


    I had to use my phone for the video, so I recorded this on a potato.
  • WordPress Performance: Caching Navigation Menus

    WordPress Performance: Caching Navigation Menus

    Background

    In the before time, WordPress developers used to build themes (and plugins?) using PHP. When we wanted to add a user generated navigation menu to sites, we had to use a function called wp_nav_menu(). Of course, now that we’re in the future, we don’t need to worry about such things.

    But if you’re still using PHP to build WordPress sites, and still using wp_nav_menu() you might not know that these menus can be performance killers!

    Problem!

    Under the hood, nav menus are stored as terms in a nav_menu taxonomy. For large sites with lots of terms and taxonomies, and complex menus (custom walkers, yay!) you can really start to see menus struggle. Sure, this might be only 50-100ms, but that really adds up after millions of pageviews.

    Caching Solution

    One of the solutions you can do us to wrap the wp_nav_menu() calls inside a caching function, like what the Cache Nav Menus plugin does. Of course, this can complicate things, and even require you to fork third party plugins and themes to maintain caching compatability.

    Another option you have is to cheat. By hooking into pre_wp_nav_menu you can cache the nav menus in place. Below is an example I’ve given customers before that shows how you can cache menus in place with a simple (mu) plugin:

    /**
     * Filters and caches the output of a WordPress navigation menu.
     *
     * This function is hooked to the 'pre_wp_nav_menu' action, it generates a unique cache key
     * for every individual menu based on the menu arguments and the last time the menu was modified.
     * This key is then used to store and retrieve cached versions of the menu.
     *
     * @param string|null $output Nav menu output to short-circuit with.
     * @param stdClas     $args   An object containing wp_nav_menu() arguments.
     *
     * @return string|null        Nav menu output.
     */
    function wpvip_pre_cache_nav_menu( string|null $output, $args ): string|null {
    	/**
    	 * Filters whether to enable caching for a specific menu.
    	 *
    	 * This filter can be used to selectively disable caching for specific WordPress navigation menus.
    	 * By default, all menus are cached for an hour.
    	 *
    	 * @param bool   $enable_cache Whether to enable caching for the menu. Default true.
    	 * @param string $args         An object containing wp_nav_menu() arguments.
    	 */
    	$enable_cache = apply_filters( 'wpvip_pre_cache_nav_menu', true, $args );
    
    	if ( ! $enable_cache ) {
    		return $output;
    	}
    
    	// Define a unique cache key.
    	$cache_key   = $args->menu . ':' . md5( serialize( $args ) ); // phpcs:ignore WordPress.PHP.DiscouragedPHPFunctions.serialize_serialize
    	$cache_group = 'wpvip_pre_cache_nav_menu';
    
    	// Try to get the cached menu.
    	$output = wp_cache_get( $cache_key, $cache_group );
    
    	// If the menu isn't cached, generate and cache it.
    	if ( false === $output ) {
    		$args->echo = false;
    		remove_action( 'pre_wp_nav_menu', 'wpvip_pre_cache_nav_menu', 10, 2 );
    		$output = wp_nav_menu( $args );
    		add_action( 'pre_wp_nav_menu', 'wpvip_pre_cache_nav_menu', 10, 2 );
    		wp_cache_set( $cache_key, $output, $cache_group, HOUR_IN_SECONDS );
    	}
    
    	return $output;
    }
    add_action( 'pre_wp_nav_menu', 'wpvip_pre_cache_nav_menu', 10, 2 );

    Now, due to the weird way that you can call menus via wp_nav_menu() this might require some tweaking depending on your needs (are you using an int, string, or WP_Query to query the desired menu? Why so many choices?)

    This is defaulting to caching for one hour, but you can likely cache for MUCH longer since menus rarely change. You could even cache forever and use the wp_update_nav_menu hook to purge and/or prime the cache.

    Now go away and break some stuff.

  • Matrix Reimagined: Crafting Digital Rain with Bash and ChatGPT

    Matrix Reimagined: Crafting Digital Rain with Bash and ChatGPT

    Just for fun, and I have no idea why I thought about it, I decided to work with ChatGPT (4) to build a simple bash-based version of the Matrix Digital Rain. I know there’s already better versions, like cmatrix, but we do not do things because they are easy. We do them because we are bored.

    I’ve asked ChatGPT to heavily comment the code for us so that we can see exactly what’s going on:

    #!/bin/bash

    # This script creates a Matrix-style falling text effect in the terminal.

    # Define strings for extra characters (Japanese Katakana) and extended ASCII characters
    extra_chars="カキクケコサシスセソタチツテトナニヌネノハヒフヘホマミムメモヤユヨラリルレロワン"
    extended_ascii="│┤┐└┴┬├─┼┘┌≡"

    # Define arrays of color codes for a fading green color effect, and a static color
    fade_colors=('\033[38;2;0;255;0m' '\033[38;2;0;192;0m' '\033[38;2;0;128;0m' '\033[38;2;0;64;0m' '\033[38;2;0;32;0m' '\033[38;2;0;32;0m' '\033[38;2;0;32;0m' '\033[38;2;0;32;0m' '\033[38;2;0;32;0m' '\033[38;2;0;32;0m' '\033[38;2;0;32;0m' '\033[38;2;0;32;0m' '\033[38;2;0;32;0m' '\033[38;2;0;32;0m' '\033[38;2;0;32;0m' '\033[38;2;0;32;0m' '\033[38;2;0;32;0m' '\033[38;2;0;32;0m' '\033[38;2;0;32;0m' '\033[38;2;0;32;0m' '\033[38;2;0;16;0m' '\033[38;2;0;8;0m') # Fading green colors
    static_color='\033[38;2;0;0;0m' # Static dark green color
    white_bold='\033[1;37m' # White and bold for the primary character

    # Get terminal dimensions
    COLUMNS=$(tput cols) # Number of columns in the terminal
    ROWS=$(tput lines) # Number of rows in the terminal


    # Hide the cursor for a cleaner effect and clear the screen
    echo -ne '\033[?25l'
    clear

    # Function to generate a random character from the set of extra characters and extended ASCII
    random_char() {
    local chars="abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789${extra_chars}${extended_ascii}"
    echo -n "${chars:RANDOM%${#chars}:1}"
    }

    # Generate a list of 1000 random characters
    random_chars=""
    for (( i=0; i<1000; i++ )); do
    random_chars+=$(random_char) # Add a random character to the end of the string
    done

    # Initialize a counter for cycling through the random characters
    char_counter=0 # Counter for cycling through the random characters

    # Initialize arrays to keep track of the position and trail characters of each column
    positions=() # Array to store the current position in each column
    trail_chars=() # Array to store the trail characters in each column
    for (( c=1; c<=COLUMNS; c++ )); do
    positions[$c]=$((RANDOM % ROWS)) # Random starting position for each column
    trail_chars[$c]="" # Start with an empty trail for each column
    done

    # Function to update the display with the falling text effect
    update_line() {
    local last_pos=0 # Track the last position to optimize cursor movement

    for (( c=1; c<=COLUMNS; c++ )); do
    # Randomly skip updating some columns to create a dynamic effect
    if [ $((RANDOM % 4)) -ne 0 ]; then
    continue
    fi

    local new_char=${random_chars:$char_counter:1} # Select the next character from the random string
    char_counter=$(( (char_counter + 1) % 1000 )) # Update the counter, cycling back after 1000

    local pos=${positions[$c]} # Current position in this column
    local trail=${trail_chars[$c]} # Current trail of characters in this column

    trail_chars[$c]="${new_char}${trail:0:$((ROWS - 1))}" # Update the trail by adding new character at the top

    # Render the trail of characters
    for (( i=0; i<${#trail}; i++ )); do
    local trail_pos=$((pos - i)) # Calculate the position for each character in the trail
    if [ $trail_pos -ge 0 ] && [ $trail_pos -lt $ROWS ]; then
    local color=${fade_colors[i]:-$static_color} # Choose color from the fade array or static color if beyond the array
    if [ $i -eq 0 ]; then
    color=$white_bold # First character in the trail is white and bold
    fi
    if [ $last_pos -ne $trail_pos ]; then
    printf "%b" "\033[${trail_pos};${c}H" # Move cursor to the right position
    last_pos=$trail_pos
    fi
    printf "%b" "${color}${trail:$i:1}\033[0m" # Print the character with color
    fi
    done

    positions[$c]=$((pos + 1)) # Update the position for the next cycle
    if [ $pos -ge $((ROWS + ${#fade_colors[@]})) ]; then
    positions[$c]=0 # Reset position if it moves off screen
    trail_chars[$c]=""
    fi
    done
    }

    # Main loop for continuous execution of the update_line function
    while true; do
    update_line
    done

    # Reset terminal settings on exit (show cursor, clear screen, reset text format)
    echo -ne '\033[?25h' # Show cursor
    clear
    tput sgr0 # Reset text format

    Challenges Faced

    Developing the Matrix Digital Rain script presented specific challenges, especially in terms of performance. The initial use of tput for cursor manipulation proved inefficient for the dynamic text display. This issue was resolved by switching to printf and ANSI escape sequences, which significantly enhanced the rendering performance.

    Another problem arose with the use of fullwidth Katakana characters, which were incompatible with monospaced fonts, disrupting the visual flow. The solution involved adopting halfwidth Katakana, ensuring better compatibility and preserving the uniformity essential for the Matrix-style effect.

    Another notable challenge emerged in development: the inefficiency of generating random characters on the fly. Due to Bash’s slower handling of string functions, this method significantly hindered performance. To tackle this, a strategic shift was made from real-time character generation to utilizing a predefined lookup table.

    This approach involved generating a large set of pseudorandom characters before entering the main loop of the program. By doing so, I could rapidly access this table during runtime, boosting performance. This change played a crucial role in maintaining fluidity and responsiveness. It also preserved the illusion of randomness, essential for the authentic Matrix effect, thus striking a balance between performance efficiency and visual fidelity.

    If you’d like to see what it looks like, here’s an example:

    So yeah. That’s it I guess? Enjoy!

  • Bash Script: Calculate before/after averages

    Bash Script: Calculate before/after averages

    I’ve been doing some performance testing, and wanted a quick way to test how well or poorly changes affect a site. Normally I’d whip out the ol’ calculator app and do this manually. That got tiring after a while, so instead with the help of ChatGPT, I made this little bash script that will do the work for you:

    #!/bin/bash

    # Function to calculate the average of a list of numbers
    average() {
    local sum=0
    local count=0

    for num in "$@"; do
    sum=$(echo "$sum + $num" | bc -l)
    count=$((count+1))
    done

    echo "$sum / $count" | bc -l
    }

    # Parse arguments
    for i in "$@"; do
    case $i in
    --before=*)
    BEFORE="${i#*=}"
    shift
    ;;
    --after=*)
    AFTER="${i#*=}"
    shift
    ;;
    *)
    # unknown option
    ;;
    esac
    done

    # Check if both BEFORE and AFTER parameters are provided
    if [ -z "$BEFORE" ] || [ -z "$AFTER" ]; then
    echo "Error: Missing required parameters."
    echo "Usage: $0 --before=<comma-separated-values> --after=<comma-separated-values>"
    exit 1
    fi

    IFS=',' read -ra BEFORE_LIST <<< "$BEFORE"
    IFS=',' read -ra AFTER_LIST <<< "$AFTER"

    # Calculate average for before and after lists
    BEFORE_AVG=$(printf "%.2f\n" $(average "${BEFORE_LIST[@]}"))
    AFTER_AVG=$(printf "%.2f\n" $(average "${AFTER_LIST[@]}"))

    echo "Before average: $BEFORE_AVG"
    echo "After average: $AFTER_AVG"

    # Calculate average percent increase, decrease or no change for the list
    if [ "$BEFORE_AVG" != "0.00" ]; then
    PERCENT_CHANGE=$(echo "(($AFTER_AVG - $BEFORE_AVG) / $BEFORE_AVG) * 100" | bc -l)
    if [ "$(echo "$PERCENT_CHANGE > 0" | bc -l)" -eq 1 ]; then
    printf "Average percent increased: %.2f%%\n" "$PERCENT_CHANGE"
    elif [ "$(echo "$PERCENT_CHANGE < 0" | bc -l)" -eq 1 ]; then
    printf "Average percent decreased: %.2f%%\n" "$PERCENT_CHANGE" | tr -d '-'
    else
    echo "No change in average."
    fi
    else
    echo "Percent change from before to after: undefined (division by zero)"
    fi

    It runs like this:

    $ average.sh --before=13.07,9.75,16.14,7.71,10.32 --after=1.22,1.28,1.13,1.19,1.26
    Before average: 11.40
    After average: 1.22
    Average percent decreased: 89.30%
    

    In this instance, it was calculating seconds–but you need to remember that it only goes to two decimal places, so if you need something finer you’ll need to adjust the code or your inputs.

    Happy Slacking!

  • Gathering database performance with WP-CLI

    Gathering database performance with WP-CLI

    Recently at work, my team was asked to help gather data about database server performance before and after an upgrade. To help with this, we collected a number of heavy database pages on some WordPress sites, dumped every query running to generate the page, and grabbed them to profile.

    I whipped up this quick and dirty WP-CLI command that will run a list of SQL queries 1,000 times and give you the minimum time, maximum time, average time, and standard deviation for each query.

    With this data, you can re-run the same queries again after a server change to determine if there has been any major SQL performance differences:

    /**
     * Profiles DB performance by running SQL queries and returning timing statistics.
     */
    public function db_profile( $args, $assoc_args ) {
    	$format = WP_CLI\Utils\get_flag_value( $assoc_args, 'format', 'table' );
    
    // Put all of your queries here!
    $site_sql_lines = <<<END
    SHOW TABLES LIKE 'wp_posts';
    END;
    
    	$lines = explode( PHP_EOL, $site_sql_lines );
    
    	global $wpdb;
    
    	$stats         = array();
    	$total_time_us = 0;
    	$runs          = 100;
    	$progress      = \WP_CLI\Utils\make_progress_bar( sprintf( 'Running %s Queries', number_format( count( $lines ) * $runs ) ), count( $lines ) * $runs );
    
    	for ( $run = 1; $run <= $runs; $run++ ) {
    		foreach ( $lines as $index => $line ) {
    			if ( ! isset( $stats[ $index ] ) || ! is_array( $stats[ $index ] ) ) {
    				$stats[ $index ] = array(
    					'query' => $line,
    					'runs'  => array(),
    				);
    			}
    
    			$start_time = hrtime( true );
    			$results    = count( $wpdb->get_results( $line, ARRAY_N ) ); // phpcs:ignore WordPress.DB.DirectDatabaseQuery, WordPress.DB.PreparedSQL.NotPrepared
    			$end_time   = hrtime( true );
    			$time_us    = ( $end_time - $start_time ) / 1000;
    
    			$total_time_us += $time_us;
    
    			$stats[ $index ]['runs'][ $run ] = $time_us;
    			$progress->tick();
    		}
    	}
    
    	$data = array();
    
    	foreach ( $stats as $index => $stat ) {
    		$stats[ $index ]['stats'] = array(
    			'min' => min( $stats[ $index ]['runs'] ),
    			'max' => max( $stats[ $index ]['runs'] ),
    			'avg' => array_sum( $stats[ $index ]['runs'] ) / count( $stats[ $index ]['runs'] ),
    		);
    
    		$data[ $index ] = array(
    			'Query'  => $stat['query'],
    			'Min'    => number_format( (float) $stats[ $index ]['stats']['min'] / 1000, 3 ),
    			'Max'    => number_format( (float) $stats[ $index ]['stats']['max'] / 1000, 3 ),
    			'Avg'    => number_format( (float) $stats[ $index ]['stats']['avg'] / 1000, 3 ),
    			'StdDev' => number_format( (float) $this->stats_standard_deviation( $stats[ $index ]['runs'] ) / 1000, 3 ),
    		);
    
    		if ( 'csv' === $format ) {
    			$data[ $index ]['Runs'] = wp_json_encode( $stats[ $index ]['runs'] );
    		}
    	}
    
    	$progress->finish();
    
    	// Output data.
    	WP_CLI\Utils\format_items( $format, $data, array_keys( $data[0] ) );
    
    	// Output totals if we're not piping somewhere.
    	if ( ! WP_CLI\Utils\isPiped() ) {
    		WP_CLI::success(
    			sprintf(
    				'Total queries ran: %s, DB Time Taken: %s',
    				WP_CLI::colorize( '%g' . number_format( count( $lines ) * $runs, 0 ) . '%n' ),
    				WP_CLI::colorize( '%g' . $this->convert_to_human_readable( $total_time_us ) . '%n' ),
    			)
    		);
    	}
    }
    

    You can then run it and gather the data as a table, or as a CSV file (--format=csv) which is much more likely to be useful.

  • Debugging WordPress Hooks: Speed

    Debugging WordPress Hooks: Speed

    If you google debugging WordPress hooks you’ll find a lot of information.

    About 1,180,000 results

    Let’s add another one.

    WordPress hooks are powerful, but also complex under the hood. There’s plenty of topics I could talk about here, but right now I’m only going to talk about speed. How long does it take for a hook to fire and return?

    Some cool work in this area has already been done thanks to Debug Bar Slow Actions and Query Monitor, but outside of something like Xdebug or New Relic, you’ll have a hard time figuring out how long each individual hook callback takes without modifying either WordPress core or each hook call.

    … or maybe not …

    While doing some client debugging for my job at WordPress VIP (did I mention we’re hiring?) I came across the need to do this exact thing. I’ve just finished the code that will make this happen, and I’m releasing it into the wild for everyone to benefit.

    What’s the problem we’re trying to solve? Well, this client has a sporadic issue where posts saving in the admin time out and sometimes fail. We’ve ruled out some potential issues, and are looking at a save_post hook going haywire.

    Now I can capture every single save_post action, what the callback was, and how long it took. Here’s an example for this exact post:

    START: 10:save_post, (Callback: `delete_get_calendar_cache`)
    STOP: 10:save_post, Taken 132.084μs (Callback: `delete_get_calendar_cache`)
    START: 10:save_post, (Callback: `sharing_meta_box_save`)
    STOP: 10:save_post, Taken 15.974μs (Callback: `sharing_meta_box_save`)
    START: 10:save_post, (Callback: `Jetpack_Likes_Settings::meta_box_save`)
    STOP: 10:save_post, Taken 19.073μs (Callback: `Jetpack_Likes_Settings::meta_box_save`)
    START: 10:save_post, (Callback: `SyntaxHighlighter::mark_as_encoded`)
    STOP: 10:save_post, Taken 19.073μs (Callback: `SyntaxHighlighter::mark_as_encoded`)
    START: 10:save_post, (Callback: `AMP_Post_Meta_Box::save_amp_status`)
    STOP: 10:save_post, Taken 16.928μs (Callback: `AMP_Post_Meta_Box::save_amp_status`)
    START: 20:save_post, (Callback: `Publicize::save_meta`)
    STOP: 20:save_post, Taken 428.915μs (Callback: `Publicize::save_meta`)
    START: 9000:save_post, (Callback: `Debug_Bar_Slow_Actions::time_stop`)
    STOP: 9000:save_post, Taken 11.921μs (Callback: `Debug_Bar_Slow_Actions::time_stop`)
    START: 1:save_post, (Callback: `The_SEO_Framework\Load::_update_post_meta`)
    STOP: 1:save_post, Taken 6.016ms (Callback: `The_SEO_Framework\Load::_update_post_meta`)
    START: 1:save_post, (Callback: `The_SEO_Framework\Load::_save_inpost_primary_term`)
    STOP: 1:save_post, Taken 1.535ms (Callback: `The_SEO_Framework\Load::_save_inpost_primary_term`)
    START: 10:save_post, (Callback: `delete_get_calendar_cache`)
    STOP: 10:save_post, Taken 179.052μs (Callback: `delete_get_calendar_cache`)
    START: 10:save_post, (Callback: `sharing_meta_box_save`)
    STOP: 10:save_post, Taken 247.002μs (Callback: `sharing_meta_box_save`)
    START: 10:save_post, (Callback: `Jetpack_Likes_Settings::meta_box_save`)
    STOP: 10:save_post, Taken 25.988μs (Callback: `Jetpack_Likes_Settings::meta_box_save`)
    START: 10:save_post, (Callback: `The_SEO_Framework\Load::delete_excluded_ids_cache`)
    STOP: 10:save_post, Taken 185.966μs (Callback: `The_SEO_Framework\Load::delete_excluded_ids_cache`)
    START: 10:save_post, (Callback: `SyntaxHighlighter::mark_as_encoded`)
    STOP: 10:save_post, Taken 15.020μs (Callback: `SyntaxHighlighter::mark_as_encoded`)
    START: 10:save_post, (Callback: `AMP_Post_Meta_Box::save_amp_status`)
    STOP: 10:save_post, Taken 12.875μs (Callback: `AMP_Post_Meta_Box::save_amp_status`)
    START: 20:save_post, (Callback: `Publicize::save_meta`)
    STOP: 20:save_post, Taken 377.893μs (Callback: `Publicize::save_meta`)
    START: 9000:save_post, (Callback: `Debug_Bar_Slow_Actions::time_stop`)
    STOP: 9000:save_post, Taken 14.067μs (Callback: `Debug_Bar_Slow_Actions::time_stop`)

    So how does it work?

    1. We add an all action that will fire for every other action.
    2. In the all callback, we make sure we’re looking for the correct hook.
    3. We then build an array to store some data, use the $wp_filter global to fill out information such as the priority and the callback, and store the start time.
    4. Next we have to add a new action to run for our hook right before the callback we want to time. We use the fact that, even though add_action() is supposed to use an int for the priority, it will also accept a string. We add new hooks, and re-prioritze all of the existing hooks with floats that are stringified.
    5. This allows us to capture the start time and end time of each individual callback, instead of the priority group as a whole.

    Of course, this does add a tiny bit of overhead, and could cause some problems if any other plugins use stringified hook priorities, or other odd issues–so be careful 🙂

    Finally, here’s the code:

    class VIP_Hook_Timeline {
    	public $hook;
    	public $callbacks     = [];
    	public $callback_mod  = 0.0001;
    	public $callback_mods = [];
    
    	public function __construct( $hook ) {
    		$this->hook = $hook;
    		add_action( 'all', array( $this, 'start' ) );
    	}
    
    	public function start() {
    		// We only want to get a timeline for one hook.
    		if ( $this->hook !== current_filter() ) {
    			return;
    		}
    
    		global $wp_filter;
    
    		// Iterate over each priority level and set up array.
    		foreach( $wp_filter[ $this->hook ] as $priority => $callback ) {
    			// Make the mod counter if not exists.
    			if ( ! isset( $this->callback_mods[ $priority ] ) ) {
    				$this->callback_mods[ $priority ] = $priority - $this->callback_mod;
    			}
    
    			// Make the array if not exists.
    			if ( ! is_array( $this->callbacks[ $priority ] ) ) {
    				$this->callbacks[ $priority ] = [];
    			}
    
    			// Iterate over each callback and set up array.
    			foreach( array_keys( $callback ) as $callback_func ) {
    				if ( ! is_array( $this->callbacks[ $priority ][ $callback_func ] ) ) {
    					$this->callbacks[ $priority ][ $callback_func ] = [ 'start' => 0, 'stop' => 0 ];
    				}
    			}
    		}
    
    		foreach( $this->callbacks as $priority => $callback ) {
    			foreach ( array_keys( $callback ) as $callback_func ) {
    
    				// Get data befmore we move things around.
    				$human_callback = $this->get_human_callback( $wp_filter[ $this->hook ][ $priority ][$callback_func] );
    
    				// Modify the priorities.
    				$pre_callback_priority = $this->callback_mods[ $priority ];
    				$this->callback_mods[ $priority ] = $this->callback_mods[ $priority ] + $this->callback_mod;
    
    				$new_callback_priority = $this->callback_mods[ $priority ];
    				$this->callback_mods[ $priority ] = $this->callback_mods[ $priority ] + $this->callback_mod;
    
    				$post_callback_priority = $this->callback_mods[ $priority ];
    				$this->callback_mods[ $priority ] = $this->callback_mods[ $priority ] + $this->callback_mod;
    
    				// Move the callback to our "new" priority.
    				if ( $new_callback_priority != $priority ) {
    					$wp_filter[ $this->hook ]->callbacks[ strval( $new_callback_priority ) ][ $callback_func ] = $wp_filter[ $this->hook ]->callbacks[ $priority ][ $callback_func ];
    					unset( $wp_filter[ $this->hook ]->callbacks[ $priority ][ $callback_func ] );
    					if ( empty( $wp_filter[ $this->hook ]->callbacks[ $priority ] ) ) {
    						unset( $wp_filter[ $this->hook ]->callbacks[ $priority ] );
    					}
    				}
    
    				// Add a new action right before the one we want to debug to capture start time.
    				add_action( $this->hook, function( $value = null ) use ( $callback_func, $priority, $human_callback ) {
    					$this->callbacks[ $priority ][ $callback_func ]['start'] = microtime( true );
    
    					// Uncomment this if you just want to dump data to the PHP error log, otherwise add your own logic.
    					//$message = 'START: %d:%s, (Callback: `%s`)';
    					// phpcs:ignore WordPress.PHP.DevelopmentFunctions.error_log_error_log
    					//error_log( sprintf( $message,
    					//	$priority,
    					//	$this->hook,
    					//	$human_callback,
    					//) );
    
    					// Just in case it's a filter, return.
    					return $value;
    				}, strval( $pre_callback_priority ) );
    
    				// Add a new action right after the one we want to debug to capture end time.
    				add_action( $this->hook, function( $value = null ) use ( $callback_func, $priority, $human_callback ) {
    					$this->callbacks[ $priority ][ $callback_func ]['stop'] = microtime( true );
    
    					// Uncomment this if you just want to dump data to the PHP error log, otherwise add your own logic.
    					//$message = 'STOP: %d:%s, Taken %s (Callback: `%s`)';
    					// phpcs:ignore WordPress.PHP.DevelopmentFunctions.error_log_error_log
    					//error_log( sprintf( $message,
    					//	$priority,
    					//	$this->hook,
    					//	$this->get_human_diff( $priority, $callback_func ),
    					//	$human_callback,
    					//) );
    
    					// Just in case it's a filter, return.
    					return $value;
    				}, strval( $post_callback_priority ) );
    				
    			}
    		}
    	}
    
    	public function get_human_callback( $callback ) {
    		$human_callback = '[UNKNOWN HOOK]';
    		if ( is_array( $callback['function'] ) && count( $callback['function'] ) == 2 ) {
    			list( $object_or_class, $method ) = $callback['function'];
    			if ( is_object( $object_or_class ) ) {
    				$object_or_class = get_class( $object_or_class );
    			}
    			$human_callback =  sprintf( '%s::%s', $object_or_class, $method );
    		} elseif ( is_object( $callback['function'] ) ) {
    			// Probably an anonymous function.
    			$human_callback =  get_class( $callback['function'] );
    		} else {
    			$human_callback =  $callback['function'];
    		}
    		return $human_callback;
    	}
    
    	public function get_start( $priority, $callback_func ) {
    		return (float) $this->callbacks[ $priority ][ $callback_func ]['start'];
    	}
    
    	public function get_stop( $priority, $callback_func ) {
    		return (float) $this->callbacks[ $priority ][ $callback_func ]['stop'];
    	}
    
    	public function get_diff( $priority, $callback_func ) {
    		return (float) ( $this->get_stop( $priority, $callback_func ) - $this->get_start( $priority, $callback_func ) );
    	}
    
    	public function get_human_diff( $priority, $callback_func ) {
    		$seconds = $this->get_diff( $priority, $callback_func );
    
    		// Seconds.
    		if ( $seconds >= 1 || $seconds == 0 ) {
    			return number_format( $seconds, 3 ) . 's';
    		}
    
    		// Milliseconds.
    		if ( $seconds >= .001 ) {
    			return number_format( $seconds * 1000, 3 ) . 'ms';
    		}
    
    		// Microseconds.
    		if ( $seconds >= .000001 ) {
    			return number_format( $seconds * 1000000, 3 ) . 'μs';
    		}
    
    		// Nanoseconds.
    		if ( $seconds >= .000000001 ) {
    			// WOW THAT'S FAST!
    			return number_format( $seconds * 1000000000, 3 ) . 'ns';
    		}
    
    		return $seconds . 's?';
    	}
    }
    new VIP_Hook_Timeline( 'save_post' );

  • Disabling WordPress Faux Cron

    Disabling WordPress Faux Cron

    The WordPress WP-Cron system is a decently okay faux cron system, but it has its problems, such as running on frontend requests and not running if no requests are coming through.

    WP-Cron works by: on every page load, a list of scheduled tasks is checked to see what needs to be run. Any tasks scheduled to be run will be run during that page load. WP-Cron does not run constantly as the system cron does; it is only triggered on page load. Scheduling errors could occur if you schedule a task for 2:00PM and no page loads occur until 5:00PM.

    From the WordPress Plugin Handbook

    These are problems because:

    • A heavy cron event can cause severe slowdown on random frontend requests, hurting page speeds.
    • Not running without requests can be bad for sites that are infrequently updated and heavily cached.

    The solution to this is to disable the built-in cron firing that’s done with pageviews, and use a system cron (or other service) to poll for cron events.

    Disabling the cron firing is done by adding this to the wp-config.php file:

    define( 'DISABLE_WP_CRON', true );

    For this site specifically, I use the “Cron Jobs” system of DreamHost to run this WP-CLI command every 10 minutes:

    wp cron event run --due-now --path=/path/to/derrick.blog/ --url=https://derrick.blog/

    This forces the cron to run and check for ready jobs every 10 minutes.  It’s possible that some cron events might run later than they “should” but in practice, I’ve seen this running more cron jobs than if I relied on page loads.

  • Query Caching (and a little extra)

    Query Caching (and a little extra)

    By default, WordPress does not cache WP_Query queries.  Doing so can greatly improve performance.  The way I do this is via the Advanced Post Cache plugin:

    By running this plugin (hopefully as an mu-plugin) with a persistent object cache, WP_Query calls, along with get_post() calls (only if suppress_filters is false) will be cached.

    Bonus!

    Now that we’re caching queries, here’s how I do a little extra caching to squeeze out a tiny bit more performance:

    <?php
    // By default Jetpack does not cache responses from Instagram oembeds.
    add_filter( 'instagram_cache_oembed_api_response_body', '__return_true' );
    
    // Cache WP Dashboard Recent Posts Query
    add_filter( 'dashboard_recent_posts_query_args', 'cache_dashboard_recent_posts_query_args', 10, 1 );
    function cache_dashboard_recent_posts_query_args( $query_args ) {
    	$query_args['cache_results'] = true;
    	$query_args['suppress_filters'] = false;
    	return $query_args;
    }
    
    // Cache WP Dashboard Recent Drafts Query
    add_filter( 'dashboard_recent_drafts_query_args', 'cache_dashboard_recent_drafts_query_args', 10, 1 );
    function cache_dashboard_recent_drafts_query_args( $query_args ) {
    	$query_args['suppress_filters'] = false;
    	return $query_args;
    }
    
    // Cache comment counts, https://github.com/Automattic/vip-code-performance/blob/master/core-fix-comment-counts-caching.php
    add_filter( 'wp_count_comments', 'wpcom_vip_cache_full_comment_counts', 10, 2 );
    function wpcom_vip_cache_full_comment_counts( $counts = false , $post_id = 0 ){
    	//We are only caching the global comment counts for now since those are often in the millions while the per page one is usually more reasonable.
    	if ( 0 !== $post_id ) {
    		return $counts;
    	}
    
    	$cache_key = "vip-comments-{$post_id}";
    	$stats_object = wp_cache_get( $cache_key );
    
    	//retrieve comments in the same way wp_count_comments() does
    	if ( false === $stats_object ) {
    		$stats = get_comment_count( $post_id );
    		$stats['moderated'] = $stats['awaiting_moderation'];
    		unset( $stats['awaiting_moderation'] );
    		$stats_object = (object) $stats;
    
    		wp_cache_set( $cache_key, $stats_object, 'default', 30 * MINUTE_IN_SECONDS );
    	}
    
    	return $stats_object;
    }
    
    // Cache monthly media array.
    add_filter( 'media_library_months_with_files', 'wpcom_vip_media_library_months_with_files' );
    function wpcom_vip_media_library_months_with_files() {
    	$months = wp_cache_get( 'media_library_months_with_files', 'extra-caching' );
    
    	if ( false === $months ) {
    		global $wpdb;
    		$months = $wpdb->get_results( $wpdb->prepare( "
    			SELECT DISTINCT YEAR( post_date ) AS year, MONTH( post_date ) AS month
    			FROM $wpdb->posts
    			WHERE post_type = %s
    			ORDER BY post_date DESC
    			", 'attachment' )
    		);
    		wp_cache_set( 'media_library_months_with_files', $months, 'extra-caching' );
    	}
    
    	return $months;
    }
    
    add_action( 'add_attachment', 'media_library_months_with_files_bust_cache' );
    function media_library_months_with_files_bust_cache( $post_id ) {
    	if ( defined( 'WP_IMPORTING' ) && WP_IMPORTING ) {
    		return;
    	}
    
    	// What month/year is the most recent attachment?
    	global $wpdb;
    	$months = $wpdb->get_results( $wpdb->prepare( "
    			SELECT DISTINCT YEAR( post_date ) AS year, MONTH( post_date ) AS month
    			FROM $wpdb->posts
    			WHERE post_type = %s
    			ORDER BY post_date DESC
    			LIMIT 1
    		", 'attachment' )
    	);
    
    	// Simplify by assigning the object to $months
    	$months = array_shift( array_values( $months ) );
    
    	// Compare the dates of the new, and most recent, attachment
    	if (
    		! $months->year == get_the_time( 'Y', $post_id ) &&
    		! $months->month == get_the_time( 'm', $post_id )
    	) {
    		// the new attachment is not in the same month/year as the
    		// most recent attachment, so we need to refresh the transient
    		wp_cache_delete( 'media_library_months_with_files', 'extra-caching' );
    	}
    }

  • iPhone – Dawn of The First Day

    iPhone – Dawn of The First Day

    The first thing that I’ve learned during my transition to iOS is that the mobile keyboard experience is very subpar compared to Android. The stock iOS keyboard is bad, but even using a third party keyboard like Gboard doesn’t even come close to the experience on Android. The layout requires so many more taps to do simple things like numbers and punctuation.

    Secondly, the down firing speakers of the iPhone 6s+ are absolutely terrible for consuming media–at least compared to the Nexus 6 and it’s stereo front firing speakers. Trying to watch a video on YouTube, I found myself having to hold the iPhone in an uncomfortable position to get even mediocre sound.

    Now, that’s not to say that it’s all bad. The speed of this old iPhone, compared to my Nexus 6 is blazingly fast. All of the OS transitions are buttery smooth, there is absolutely no lag to be seen, and the camera can take pictures faster than I can tap!

    The iOS experience doesn’t seem to be created for one-handed use, since I constantly have to stretch far to the top of the screen or exit lazy mode and get my second hand out to tap something at the top. Android’s bottom navigation bar is definitely a better user experience.