Tag: queries

  • Syntax Highlighting SQL in Terminal

    Syntax Highlighting SQL in Terminal

    Do you ever find yourself doing some debugging with error_log() or its friends? Does that debugging ever involve SQL queries? Are you tired of staring at grey queries all the time? I have just the product for you!

    Introducing Syntax Highlighting SQL in Terminal! Brought to you by our friends at Large Language Models, Incorporated, this new PHP function will use ANSI escape sequences to colorize your SQL queries for easier viewing and debugging!

    <?php
    
    /**
     * Highlights SQL queries.
     *
     * This method takes a SQL query as input and returns the query with syntax highlighting applied.
     *
     * @param string $sql The SQL query to highlight.
     *
     * @return string The highlighted SQL query.
     */
    function highlight_sql( string $sql ): string {
    	$keywords = array(
    		'SELECT',
    		'FROM',
    		'WHERE',
    		'AND',
    		'OR',
    		'INSERT',
    		'INTO',
    		'VALUES',
    		'UPDATE',
    		'SET',
    		'DELETE',
    		'CREATE',
    		'TABLE',
    		'ALTER',
    		'DROP',
    		'JOIN',
    		'INNER',
    		'LEFT',
    		'RIGHT',
    		'ON',
    		'AS',
    		'DISTINCT',
    		'GROUP',
    		'BY',
    		'ORDER',
    		'HAVING',
    		'LIMIT',
    		'OFFSET',
    		'UNION',
    		'ALL',
    		'COUNT',
    		'SUM',
    		'AVG',
    		'MIN',
    		'MAX',
    		'LIKE',
    		'IN',
    		'BETWEEN',
    		'IS',
    		'NULL',
    		'NOT',
    		'PRIMARY',
    		'KEY',
    		'FOREIGN',
    		'REFERENCES',
    		'DEFAULT',
    		'AUTO_INCREMENT',
    	);
    
    	$functions = array(
    		'COUNT',
    		'SUM',
    		'AVG',
    		'MIN',
    		'MAX',
    		'NOW',
    		'CURDATE',
    		'CURTIME',
    		'DATE',
    		'TIME',
    		'YEAR',
    		'MONTH',
    		'DAY',
    		'HOUR',
    		'MINUTE',
    		'SECOND',
    		'TIMESTAMP',
    	);
    
    	$colors = array(
    		'keyword'  => "\033[1;34m", // Blue.
    		'function' => "\033[1;32m", // Green.
    		'string'   => "\033[1;33m", // Yellow.
    		'comment'  => "\033[1;90m", // Bright Black (Gray).
    		'reset'    => "\033[0m",
    	);
    
    	// Highlight comments.
    	$sql = preg_replace( '/\/\*.*?\*\//s', $colors['comment'] . '$0' . $colors['reset'], $sql );
    
    	// Highlight single-quoted strings.
    	$sql = preg_replace( '/\'[^\']*\'/', $colors['string'] . '$0' . $colors['reset'], $sql );
    
    	// Highlight double-quoted strings.
    	$sql = preg_replace( '/"[^"]*"/', $colors['string'] . '$0' . $colors['reset'], $sql );
    
    	// Highlight functions.
    	foreach ( $functions as $function ) {
    		$sql = preg_replace( '/\b' . preg_quote($function, '/') . '\b/i', $colors['function'] . '$0' . $colors['reset'], $sql );
    	}
    
    	// Highlight keywords.
    	foreach ( $keywords as $keyword ) {
    		$sql = preg_replace( '/\b' . preg_quote($keyword, '/') . '\b/i', $colors['keyword'] . '$0' . $colors['reset'], $sql );
    	}
    
    	return $sql;
    }
    
    echo highlight_sql( 'SELECT COUNT(*) AS total FROM users WHERE id = 1 AND user_id = "example"; /* YEAH! Comment! */' ) . PHP_EOL;Code language: PHP (php)

    Don’t blame me if this breaks anything, use at your own debugging risk.

  • Gathering database performance with WP-CLI

    Gathering database performance with WP-CLI

    Recently at work, my team was asked to help gather data about database server performance before and after an upgrade. To help with this, we collected a number of heavy database pages on some WordPress sites, dumped every query running to generate the page, and grabbed them to profile.

    I whipped up this quick and dirty WP-CLI command that will run a list of SQL queries 1,000 times and give you the minimum time, maximum time, average time, and standard deviation for each query.

    With this data, you can re-run the same queries again after a server change to determine if there has been any major SQL performance differences:

    /**
     * Profiles DB performance by running SQL queries and returning timing statistics.
     */
    public function db_profile( $args, $assoc_args ) {
    	$format = WP_CLI\Utils\get_flag_value( $assoc_args, 'format', 'table' );
    
    // Put all of your queries here!
    $site_sql_lines = <<<END
    SHOW TABLES LIKE 'wp_posts';
    END;
    
    	$lines = explode( PHP_EOL, $site_sql_lines );
    
    	global $wpdb;
    
    	$stats         = array();
    	$total_time_us = 0;
    	$runs          = 100;
    	$progress      = \WP_CLI\Utils\make_progress_bar( sprintf( 'Running %s Queries', number_format( count( $lines ) * $runs ) ), count( $lines ) * $runs );
    
    	for ( $run = 1; $run <= $runs; $run++ ) {
    		foreach ( $lines as $index => $line ) {
    			if ( ! isset( $stats[ $index ] ) || ! is_array( $stats[ $index ] ) ) {
    				$stats[ $index ] = array(
    					'query' => $line,
    					'runs'  => array(),
    				);
    			}
    
    			$start_time = hrtime( true );
    			$results    = count( $wpdb->get_results( $line, ARRAY_N ) ); // phpcs:ignore WordPress.DB.DirectDatabaseQuery, WordPress.DB.PreparedSQL.NotPrepared
    			$end_time   = hrtime( true );
    			$time_us    = ( $end_time - $start_time ) / 1000;
    
    			$total_time_us += $time_us;
    
    			$stats[ $index ]['runs'][ $run ] = $time_us;
    			$progress->tick();
    		}
    	}
    
    	$data = array();
    
    	foreach ( $stats as $index => $stat ) {
    		$stats[ $index ]['stats'] = array(
    			'min' => min( $stats[ $index ]['runs'] ),
    			'max' => max( $stats[ $index ]['runs'] ),
    			'avg' => array_sum( $stats[ $index ]['runs'] ) / count( $stats[ $index ]['runs'] ),
    		);
    
    		$data[ $index ] = array(
    			'Query'  => $stat['query'],
    			'Min'    => number_format( (float) $stats[ $index ]['stats']['min'] / 1000, 3 ),
    			'Max'    => number_format( (float) $stats[ $index ]['stats']['max'] / 1000, 3 ),
    			'Avg'    => number_format( (float) $stats[ $index ]['stats']['avg'] / 1000, 3 ),
    			'StdDev' => number_format( (float) $this->stats_standard_deviation( $stats[ $index ]['runs'] ) / 1000, 3 ),
    		);
    
    		if ( 'csv' === $format ) {
    			$data[ $index ]['Runs'] = wp_json_encode( $stats[ $index ]['runs'] );
    		}
    	}
    
    	$progress->finish();
    
    	// Output data.
    	WP_CLI\Utils\format_items( $format, $data, array_keys( $data[0] ) );
    
    	// Output totals if we're not piping somewhere.
    	if ( ! WP_CLI\Utils\isPiped() ) {
    		WP_CLI::success(
    			sprintf(
    				'Total queries ran: %s, DB Time Taken: %s',
    				WP_CLI::colorize( '%g' . number_format( count( $lines ) * $runs, 0 ) . '%n' ),
    				WP_CLI::colorize( '%g' . $this->convert_to_human_readable( $total_time_us ) . '%n' ),
    			)
    		);
    	}
    }
    
    Code language: PHP (php)

    You can then run it and gather the data as a table, or as a CSV file (--format=csv) which is much more likely to be useful.

  • Quick Tip: Get Size of Revisions in WordPress

    Quick Tip: Get Size of Revisions in WordPress

    One thing that you might not think of when watching the size of a large WordPress site grow, is unnecessary data in the database. With the introduction of the block editor years ago, there has been a large increase in the number of revisions a post makes when being edited.

    This can create a lot of revisions in the database if you’re not setting a limit.

    If you’d like to do a quick and dirty audit of your revision data, you can use a very ugly SQL query like this:

    SELECT COUNT( ID ) as revision_count, ( SUM( CHAR_LENGTH( ID ) ) + SUM( CHAR_LENGTH( post_author ) ) + SUM( CHAR_LENGTH( post_date ) ) + SUM( CHAR_LENGTH( post_date_gmt ) ) + SUM( CHAR_LENGTH( post_content ) ) + SUM( CHAR_LENGTH( post_title ) ) + SUM( CHAR_LENGTH( post_excerpt ) ) + SUM( CHAR_LENGTH( post_status ) ) + SUM( CHAR_LENGTH( comment_status ) ) + SUM( CHAR_LENGTH( ping_status ) ) + SUM( CHAR_LENGTH( post_password ) ) + SUM( CHAR_LENGTH( post_name ) ) + SUM( CHAR_LENGTH( to_ping ) ) + SUM( CHAR_LENGTH( pinged ) ) + SUM( CHAR_LENGTH( post_modified ) ) + SUM( CHAR_LENGTH( post_modified_gmt ) ) + SUM( CHAR_LENGTH( post_content_filtered ) ) + SUM( CHAR_LENGTH( post_parent ) ) + SUM( CHAR_LENGTH( guid ) ) + SUM( CHAR_LENGTH( menu_order ) ) + SUM( CHAR_LENGTH( post_type ) ) + SUM( CHAR_LENGTH( post_mime_type ) ) + SUM( CHAR_LENGTH( comment_count ) ) ) as post_size FROM wp_posts WHERE post_type='revision' GROUP BY post_type ORDER BY revision_count DESC;Code language: JavaScript (javascript)

    This will give you the number of revisions you have, and the approximate amount of data its using in the database:

    revision_count post_size
    441419 2842450412

    You can see now a very large WordPress site can amass a lot of unnecessary data in its database over a number of years. 2.8 Gigabytes of revisions is a lot of stuff if you’re never going to use them again.

  • Getting WordPress Database Size via WP-CLI

    Getting WordPress Database Size via WP-CLI

    One WP-CLI command that I’ve found handy is this db-size command. It allows you to output a site’s registered database tables along with the data and index size in any format that WP-CLI natively supports, with multiple sort options:

    /**
     * Gets size of database tables for the current site.
     *
     * ## OPTIONS
     *
     * [--raw]
     * : Outputs full size in bytes instead of human readable sizes.
     *
     * [--order_by=<Total><total>]
     * : Allows custom ordering of the data.
     * ---
     * default: Total
     * options:
     *   - Table
     *   - Data Size
     *   - Index Size
     *   - Total
     * ---
     *
     * [--order=<asc><asc>]
     * : Allows custom ordering direction of the data.
     * ---
     * default: asc
     * options:
     *   - asc
     *   - desc
     * ---
     *
     * [--format=<format><format>]
     * : Render output in a particular format.
     * ---
     * default: table
     * options:
     *   - table
     *   - csv
     *   - json
     *   - count
     *   - yaml
     * ---
     *
     * @subcommand db-size
     */
    public function db_size( $args, $assoc_args ) {
    	global $wpdb;
    
    	$output   = array();
    	$db_size  = array();
    	$order_by = WP_CLI\Utils\get_flag_value( $assoc_args, 'order_by', 'Total' );
    	$order    = WP_CLI\Utils\get_flag_value( $assoc_args, 'order', 'asc' );
    	$format   = WP_CLI\Utils\get_flag_value( $assoc_args, 'format', 'table' );
    	$raw      = (bool) WP_CLI\Utils\get_flag_value( $assoc_args, 'raw', false );
    
    	// Fetch list of tables from database.
    	$tables = array_map(
    		function( $val ) {
    			return $val[0];
    		},
    		$wpdb->get_results( 'SHOW TABLES;', ARRAY_N ) // phpcs:ignore WordPress.DB.DirectDatabaseQuery
    	);
    
    	// Fetch table information from database.
    	$report = array_map(
    		function( $table ) use ( $wpdb ) { // phpcs:ignore WordPress.DB.DirectDatabaseQuery
    			return $wpdb->get_row( "SHOW TABLE STATUS LIKE '$table'" ); // phpcs:ignore WordPress.DB.PreparedSQL.InterpolatedNotPrepared
    		},
    		$tables
    	);
    
    	foreach ( $report as $table ) {
    		// Keep a running total of sizes.
    		$db_size['data']  += $table->Data_length; // phpcs:ignore WordPress.NamingConventions.ValidVariableName.UsedPropertyNotSnakeCase
    		$db_size['index'] += $table->Index_length; // phpcs:ignore WordPress.NamingConventions.ValidVariableName.UsedPropertyNotSnakeCase
    
    		// Format output for WP-CLI's format_items function.
    		$output[] = array(
    			'Table'      => $table->Name, // phpcs:ignore WordPress.NamingConventions.ValidVariableName.UsedPropertyNotSnakeCase
    			'Data Size'  => $table->Data_length, // phpcs:ignore WordPress.NamingConventions.ValidVariableName.UsedPropertyNotSnakeCase
    			'Index Size' => $table->Index_length, // phpcs:ignore WordPress.NamingConventions.ValidVariableName.UsedPropertyNotSnakeCase
    			'Total'      => $table->Data_length + $table->Index_length, // phpcs:ignore WordPress.NamingConventions.ValidVariableName.UsedPropertyNotSnakeCase
    		);
    	}
    
    	// Sort table data.
    	self::sort_table_by( $order_by, $output, $order );
    
    	if ( ! $raw ) {
    		// Make data human readable.
    		foreach ( array_keys( $output ) as $key ) {
    			$output[ $key ]['Data Size']  = size_format( $output[ $key ]['Data Size'] );
    			$output[ $key ]['Index Size'] = size_format( $output[ $key ]['Index Size'] );
    			$output[ $key ]['Total']      = size_format( $output[ $key ]['Total'] );
    		}
    	}
    
    	// Output data.
    	WP_CLI\Utils\format_items( $format, $output, array( 'Table', 'Data Size', 'Index Size', 'Total' ) );
    
    	// Output totals if we're not piping somewhere.
    	if ( ! WP_CLI\Utils\isPiped() ) {
    		WP_CLI::success(
    			sprintf(
    				'Total size of the database for %s is %s. Data: %s; Index: %s',
    				home_url(),
    				WP_CLI::colorize( '%g' . size_format( $db_size['data'] + $db_size['index'] ) . '%n' ),
    				WP_CLI::colorize( '%g' . size_format( $db_size['data'] ) . '%n' ),
    				WP_CLI::colorize( '%g' . size_format( $db_size['index'] ) . '%n' )
    			)
    		);
    	}
    }
    
    /**
     * Sorts a table by a specific field and direction.
     *
     * @param string $field The field to order by.
     * @param array  &$array The table array to sort.
     * @param string $direction The direction to sort. Ascending ('asc') or descending ('desc').
     */
    private function sort_table_by( $field, &$array, $direction ) {
    	// Taken from https://joshtronic.com/2013/09/23/sorting-associative-array-specific-key/ Thanks!
    	usort(
    		$array,
    		function ( $a, $b ) use ( $field, $direction ) {
    			$a = $a[ $field ];
    			$b = $b[ $field ];
    
    			if ( $a === $b ) {
    				return 0;
    			}
    
    			switch ( $direction ) {
    				case 'asc':
    					return ( $a < $b ) ? -1 : 1;
    				case 'desc':
    					return ( $a > $b ) ? -1 : 1;
    				default:
    					return 0;
    			}
    		}
    	);
    	return true;
    }</format></asc></total>Code language: PHP (php)

    Here’s some example output from one of my test sites:

    $ wp test db-size
     +----------------------------+-----------+------------+-------+
     | Table                      | Data Size | Index Size | Total |
     +----------------------------+-----------+------------+-------+
     | wp_mlp_relationships       | 16KB      | 0B         | 16KB  |
     | wp_redacted_table          | 16KB      | 0B         | 16KB  |
     | wp_redacted_table          | 16KB      | 0B         | 16KB  |
     | wp_3_links                 | 16KB      | 16KB       | 32KB  |
     | wp_term_relationships      | 16KB      | 16KB       | 32KB  |
     | wp_site                    | 16KB      | 16KB       | 32KB  |
     | wp_registration_log        | 16KB      | 16KB       | 32KB  |
     | wp_mlp_site_relations      | 16KB      | 16KB       | 32KB  |
     | wp_mlp_languages           | 16KB      | 16KB       | 32KB  |
     | wp_mlp_content_relations   | 16KB      | 16KB       | 32KB  |
     | wp_links                   | 16KB      | 16KB       | 32KB  |
     | wp_redacted_table          | 16KB      | 16KB       | 32KB  |
     | wp_3_term_relationships    | 16KB      | 16KB       | 32KB  |
     | wp_blog_versions           | 16KB      | 16KB       | 32KB  |
     | wp_2_links                 | 16KB      | 16KB       | 32KB  |
     | wp_2_term_relationships    | 16KB      | 16KB       | 32KB  |
     | wp_2_term_taxonomy         | 16KB      | 32KB       | 48KB  |
     | wp_2_commentmeta           | 16KB      | 32KB       | 48KB  |
     | wp_2_postmeta              | 16KB      | 32KB       | 48KB  |
     | wp_term_taxonomy           | 16KB      | 32KB       | 48KB  |
     | wp_3_commentmeta           | 16KB      | 32KB       | 48KB  |
     | wp_2_termmeta              | 16KB      | 32KB       | 48KB  |
     | wp_2_terms                 | 16KB      | 32KB       | 48KB  |
     | wp_termmeta                | 16KB      | 32KB       | 48KB  |
     | wp_commentmeta             | 16KB      | 32KB       | 48KB  |
     | wp_blogmeta                | 16KB      | 32KB       | 48KB  |
     | wp_blogs                   | 16KB      | 32KB       | 48KB  |
     | wp_2_a8c_cron_control_jobs | 16KB      | 32KB       | 48KB  |
     | wp_redacted_table          | 16KB      | 32KB       | 48KB  |
     | wp_3_terms                 | 16KB      | 32KB       | 48KB  |
     | wp_3_termmeta              | 16KB      | 32KB       | 48KB  |
     | wp_3_term_taxonomy         | 16KB      | 32KB       | 48KB  |
     | wp_redacted_table          | 16KB      | 32KB       | 48KB  |
     | wp_terms                   | 16KB      | 32KB       | 48KB  |
     | wp_3_postmeta              | 16KB      | 32KB       | 48KB  |
     | wp_usermeta                | 16KB      | 32KB       | 48KB  |
     | wp_sitemeta                | 16KB      | 32KB       | 48KB  |
     | wp_users                   | 16KB      | 48KB       | 64KB  |
     | wp_postmeta                | 16KB      | 48KB       | 64KB  |
     | wp_posts                   | 16KB      | 64KB       | 80KB  |
     | wp_signups                 | 16KB      | 64KB       | 80KB  |
     | wp_2_posts                 | 16KB      | 64KB       | 80KB  |
     | wp_3_posts                 | 16KB      | 64KB       | 80KB  |
     | wp_2_comments              | 16KB      | 80KB       | 96KB  |
     | wp_comments                | 16KB      | 80KB       | 96KB  |
     | wp_3_comments              | 16KB      | 80KB       | 96KB  |
     | wp_2_options               | 80KB      | 32KB       | 112KB |
     | wp_3_options               | 80KB      | 32KB       | 112KB |
     | wp_options                 | 176KB     | 32KB       | 208KB |
     +----------------------------+-----------+------------+-------+
     Success: Total size of the database for https://www.example.com is 2.6MB. Data: 1MB; Index: 1.5MBCode language: JavaScript (javascript)

    Enjoy!

  • Query Caching (and a little extra)

    Query Caching (and a little extra)

    By default, WordPress does not cache WP_Query queries.  Doing so can greatly improve performance.  The way I do this is via the Advanced Post Cache plugin:

    By running this plugin (hopefully as an mu-plugin) with a persistent object cache, WP_Query calls, along with get_post() calls (only if suppress_filters is false) will be cached.

    Bonus!

    Now that we’re caching queries, here’s how I do a little extra caching to squeeze out a tiny bit more performance:

    <?php
    // By default Jetpack does not cache responses from Instagram oembeds.
    add_filter( 'instagram_cache_oembed_api_response_body', '__return_true' );
    
    // Cache WP Dashboard Recent Posts Query
    add_filter( 'dashboard_recent_posts_query_args', 'cache_dashboard_recent_posts_query_args', 10, 1 );
    function cache_dashboard_recent_posts_query_args( $query_args ) {
    	$query_args['cache_results'] = true;
    	$query_args['suppress_filters'] = false;
    	return $query_args;
    }
    
    // Cache WP Dashboard Recent Drafts Query
    add_filter( 'dashboard_recent_drafts_query_args', 'cache_dashboard_recent_drafts_query_args', 10, 1 );
    function cache_dashboard_recent_drafts_query_args( $query_args ) {
    	$query_args['suppress_filters'] = false;
    	return $query_args;
    }
    
    // Cache comment counts, https://github.com/Automattic/vip-code-performance/blob/master/core-fix-comment-counts-caching.php
    add_filter( 'wp_count_comments', 'wpcom_vip_cache_full_comment_counts', 10, 2 );
    function wpcom_vip_cache_full_comment_counts( $counts = false , $post_id = 0 ){
    	//We are only caching the global comment counts for now since those are often in the millions while the per page one is usually more reasonable.
    	if ( 0 !== $post_id ) {
    		return $counts;
    	}
    
    	$cache_key = "vip-comments-{$post_id}";
    	$stats_object = wp_cache_get( $cache_key );
    
    	//retrieve comments in the same way wp_count_comments() does
    	if ( false === $stats_object ) {
    		$stats = get_comment_count( $post_id );
    		$stats['moderated'] = $stats['awaiting_moderation'];
    		unset( $stats['awaiting_moderation'] );
    		$stats_object = (object) $stats;
    
    		wp_cache_set( $cache_key, $stats_object, 'default', 30 * MINUTE_IN_SECONDS );
    	}
    
    	return $stats_object;
    }
    
    // Cache monthly media array.
    add_filter( 'media_library_months_with_files', 'wpcom_vip_media_library_months_with_files' );
    function wpcom_vip_media_library_months_with_files() {
    	$months = wp_cache_get( 'media_library_months_with_files', 'extra-caching' );
    
    	if ( false === $months ) {
    		global $wpdb;
    		$months = $wpdb->get_results( $wpdb->prepare( "
    			SELECT DISTINCT YEAR( post_date ) AS year, MONTH( post_date ) AS month
    			FROM $wpdb->posts
    			WHERE post_type = %s
    			ORDER BY post_date DESC
    			", 'attachment' )
    		);
    		wp_cache_set( 'media_library_months_with_files', $months, 'extra-caching' );
    	}
    
    	return $months;
    }
    
    add_action( 'add_attachment', 'media_library_months_with_files_bust_cache' );
    function media_library_months_with_files_bust_cache( $post_id ) {
    	if ( defined( 'WP_IMPORTING' ) && WP_IMPORTING ) {
    		return;
    	}
    
    	// What month/year is the most recent attachment?
    	global $wpdb;
    	$months = $wpdb->get_results( $wpdb->prepare( "
    			SELECT DISTINCT YEAR( post_date ) AS year, MONTH( post_date ) AS month
    			FROM $wpdb->posts
    			WHERE post_type = %s
    			ORDER BY post_date DESC
    			LIMIT 1
    		", 'attachment' )
    	);
    
    	// Simplify by assigning the object to $months
    	$months = array_shift( array_values( $months ) );
    
    	// Compare the dates of the new, and most recent, attachment
    	if (
    		! $months->year == get_the_time( 'Y', $post_id ) &&
    		! $months->month == get_the_time( 'm', $post_id )
    	) {
    		// the new attachment is not in the same month/year as the
    		// most recent attachment, so we need to refresh the transient
    		wp_cache_delete( 'media_library_months_with_files', 'extra-caching' );
    	}
    }Code language: HTML, XML (xml)