Tuning CPU performance on Linux server

If you have an older desktop CPU for the server, like me (AMD Phenom II), you web applications may be suffering from slow page loads. Some (older) CPUs switch to the faster frequencies from power-saving mode very slowly. And this causes slow page rendering.

There are two ways to improve CPU performance on Linux sever.

One is by holding frequencies at maximum all the time. That increases power consumption.
#!/bin/bash
for core in $(seq 0 3); do # Core from 0 to 3 (4 core CPU)
  if [ "`cpufreq-info -c $core | grep \\\"performance\\\"`" == "" ]; then
    cpufreq-set -c $core -g performance
  fi
done
The second one is to instruct the OS to scale up the frequencies on 20% load:
#!/bin/bash
file=/sys/devices/system/cpu/cpufreq/ondemand/up_threshold
if [ -f $file ]; then
  if [ "`cat $file`" != "20" ]; then
    echo 20 > $file
  fi
fi

Performance isn’t much slower and fan noise is much more ear pleasant.

Cpu-freq must be installed. Use “sudo apt-get install cpufrequtils”. Save the code into a file and run it every few minutes with the help of “cron”.

In my case the time to render a WordPress site decreased from around 800 ms to 500 ms.

qTranslate qtrans_use recursion optimization

This is another post (first post) about qTranslate performance optimization. This plugin really slows down the blog I’m administrating, so it keeps taunting me to optimize it’s execution. The already optimized (translation of options disabled) plugin adds about 150ms to rendering time. This enhancement saves about 60ms.

Webgrind analysis, before I made the changes, looks like this:
qTranslate + Webgring = before optimization

The changed “function qtrans_use” has some extra code added. The idea is, not to call recursions, when it’s not needed:

function qtrans_use($lang, $text, $show_available=false) {
    $split_regex = "#(|\[:[a-z]{2}\])#ism";
    if (is_string($text)) {
        if (!preg_match($split_regex, $text)) {
            return $text;
        }
    }

    global $q_config;
    // return full string if language is not enabled
    if(!qtrans_isEnabled($lang)) return $text;
    if(is_array($text)) {
        // handle arrays recursively
        foreach($text as $key => $t) {
            $transNeeded = true;
            if (is_string($t)) {
                if (!preg_match($split_regex, $t)) {
                    $transNeeded = false;
                }
            }
            if ($transNeeded) {
                $text[$key] = qtrans_use($lang,$t,$show_available);
            }
        }
        return $text;
    }

    if(is_object($text)||@get_class($text) == '__PHP_Incomplete_Class') {
        foreach(get_object_vars($text) as $key => $t) {
            $transNeeded = true;
            if (is_string($t)) {
                if (!preg_match($split_regex, $t)) {
                    $transNeeded = false;
                }
            }
            if ($transNeeded) {
                $text->$key = qtrans_use($lang,$t,$show_available);
            }
        }
        return $text;
    }
    ...

The Webgrind analysis looks much nicer now:
qTranslate + Webgring = after optimization

Performance improvements in PHP 5.4.0

One day I looked at a presentation about the upcoming PHP 5.4.0. I just stared at the slide 3 when I saw the numbers for performance and memory usage improvements. I said to myself: I have to make a test. I downloaded PHP 5.4.0alpha2, run “.configure” with mysql options and “make” and started the php cli command. And a mysql error popped up.

Today I looked at qa.php.net and a beta version was available. I ran the commands again and this time it worked. This are the results I got testing a rather complex (lots of posts and plugins) WordPress blog:

Run 5 times PHP 5.3.5 PHP 5.4.0beta1
Exec. time from 884 to 1004 ms from 641 to 777 ms
Mem. usage 26.00 MiB 14.75 MiB

For me it’s quite impressive. Script was executed in 25% less time and PHP process used 43% less memory. I can’t wait to install this on my production server, because waiting a half a second to generate a page (for my current configuration) is a little excessive.

Setting client caching for WordPress media files

Read this as you were a TV commercial anchor: Does your WordPress blog have images attached? Do your users suffers form slow page load times? Fear no more. The client image caching tutorial is here. 🙂

I used Google Page Speed and WebPagetest to analyze the performance of “my” blog. The things I found out are:
WP Minify, which joins different CSS and JavaScript files is working as expected;
– page requests do not send cookies only to HTML/PHP pages but to images also (not good);
– loaded images do not get cached by the browser and they are reloaded every time the user refreshes the page (very bad);
– ETags should be left alone – Google wants them, WebPagetest doesn’t.

Sending cookies to images is not a big problem, because the only thing that suffers is a little slower sending of requests to the server. This can be fixed by setting a new domain name (let’s say content.domain.com) pointing to the same server, and setting a “Full URL path to files” in Admin panel, Settings, Media. It should look like “http://content.domain.com/wp-content/uploads”.

The really bad thing is, especially for a blog with lots of photos, that the images the WordPress blog server sends to the user, aren’t cached by his browser. In my case that means almost a megabyte of extra content sent to the client with every page refresh. If you want to get around this, you must create a “.htaccess” file directly in the root folder of your WordPress installation. And then add the following content:

# Add expire headers
<FilesMatch "\.(txt|js|css)$">
Header set Cache-Control "max-age=3600"
</FilesMatch>
<FilesMatch "\.(gif|jpg|jpeg|png|swf|ico)$">
Header set Cache-Control "max-age=604800, public"
</FilesMatch>

The JavaScript and CSS gets cached a little shorter, only for an hour. The images are cached for a week, as Google recommends. If you want to change a image, that was already distributed to the user, make a new upload and redirect the old links to the newly uploaded file. Do not just replace the image, as users, who already viewed the photos, would not see any changes.

If you create “.htaccess” in “wp-content/uploads” the caching time can be made even higher (let’s say a month):

<FilesMatch "\.(gif|jpg|jpeg|png|ico)$">
Header set Cache-Control "max-age=2592000, public"
</FilesMatch>

Same as before, upload a new file and change the links, do not replace the content.

HTML table reorganization for WP-Table Reloaded WordPress plugin

In this post I’ll write about another problem with a WordPress plugin. Blogger, for whom I’m managing the WordPress installation, had another wish for plugin customization. The installed WP-Table Reloaded plugin enables the user to create a table and add contents to it’s cells. But it isn’t able to sort the already added entries in the table or shifting the entries for inserting a new cell.

So the choice was to write another plugin or make a script that rearranges the entries by a specific condition, because the plugin supports the exporting and importing the HTML tables. I chose the letter one and wrote a Python3 script.

The exported HTML file can be reorganized in this ways: specify the new number of wanted columns, choose a horizontal (same as an input file) or vertical rendering and specify a regular expression for sorting the entries. Specifying “0” for sorting disables it and putting “(a)” for regular expression option sorts the elements by text between [a] HTML tags.

Here you can download the Table reorganization for WP-Table Reloaded WordPress plugin script.

Update on 6th of August: Corrected the rendering when multiple [a] tags exist in a single cell. Fixed vertical rendering when there the number of elements didn’t fill the whole columns*rows box.

qTranslate support for »Google XML Sitemaps« plugin

Having qTranslate, multilingual WordPress plugin, installed, brings some problems – between them are slower performance and incompatibility with other plugins. It is said, that having a XML sitemap helps you climb on Google rankings. And you can find a few plugins which support this. One of them is Google XML Sitemaps but it has a broken qTranslate support. This was fixed by a forked Google XML Sitemaps with qTranslate Support, but unfortunately it is not up to date and doesn’t translate categories and tags.

So, some coding was needed. To transfer the enhancement to the original plugin, I wrote two functions, which are stored in a separate file, so the new code does not intermingle with the original one. I also enhanced the functionality, so that besides URLs for home page and posts, the URL is translated for categories, tags, archives and author pages. It also works as correctly for enabling/disabling qTranslate’s “hide_default_language” option.

Download the latest version of »Google XML Sitemaps for qTranslate« plugin. Zip file contains the original and modified file “sitemap-core.php” and a new one “sitemap-qtranslate.php”.

To do: translation of URLs for subposts, static pages, taxonomy pages.

Update on 3rd of June: Corrected display of “last modification time” in “sitemap-qtranslate.php”.
Update on 4th of June: Corrected URL translations when not using qTrasnlate option “URL Modification Mode” set to “Use Pre-Path Mode”.
Update on 19th of August: The code was packed as an official WordPress plugin.

Spam attack through wp-login.php?action=register

On the 15th of May my girlfriend started complaining abut the number of new users with suspicious data that have registered on her blog. Looking at the Apache logs:

111.94.XXX.XXX – – [15/May/2011:07:48:57 +0200] “POST /wp-login.php?action=register HTTP/1.1” 200 2857 “http://www.domain.com/wp-login.php?action=register” “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 GTB6 (.NET CLR 3.5.30729)”

I found out, that all visits came from different IPs and didn’t have the referral specified.

The captcha under the registration form didn’t seem to stop the spammers. I had to find the quick fix, until the registration stops and spam starts to roll in. The closest solution I found was four years old, so I had to do a little update with the help of Apache documentation.

# Disable spammy registrations
RewriteCond %{REQUEST_URI} “^/wp-login.php$” [NC]
RewriteCond %{QUERY_STRING} “action=register” [NC]
RewriteCond %{HTTP_REFERER} “!^http://([^.]+.)?domain.com/.*$”
RewriteRule (.*) “/wp-login.php?” [L,R]

What that means is: if a user doesn’t visit “/wp-login.php?action=register” through clicking on a link on “domain.com”, he will be redirected to “/wp-login.php”.

Update: And don’t forget to delete the spammy users form the database. Backup your data and run select, before you use the delete query. I executed this through phpMyAdmin:

DELETE u, um
— SELECT distinct u.*
FROM wp_users u
left join wp_usermeta um on um.user_id = u.id
WHERE u.id between X and Y
— and u.id not in (‘A’, ‘B’, ‘C’)

To get the numbers for X and Y, use Browse on ‘wp_users’ table to get the first and the last unwanted registration. If you find a legit user (A, B, C, etc.) in this range, exclude it with appropriate filter.

Choosing a CPU for WordPress server

Running a WordPress blog, with lots of plugins installed, requires a reasonably capable server. Running a custom theme and multiple plugins makes the response times skyrocket on an old processor like AMD Sempron 2600+. In my case, the time to generate a page, took from about 2 seconds. Looking at the performance monitor, I found that CPU is to blame. This can be mitigated my installing a cache plugin, but response to the visits, which create the cache files, are still slow.

So a new server, which is not already aged, must be bought. But which one to choose? Expensive server platform of cheaper desktop system? Intel or AMD? I was not looking for a high performance server for lots of simultaneous visitors. To process a few thousands visits per day it is only vital to process the PHP code for a single request as fast as possible. The 10 Mbit/s internet line, which I have, can never fully utilize a common hard drive or any other component.

For a server, certain hardware specifications must be met. That is hard drive redundancy (in my case RAID-1) and ECC memory for detecting/correcting memory faults. I rather be safe then sorry.

I decided to test different platforms by installing a backed-up blog on different computers I have access to – one installation was on the hosting server and others by installing a VirtualBox guest, running Ubuntu Server.

Here are the times (I made this calculations at the end of summer 2010), that it took generate the first page (monitored through Firebug):

Measured times Time to generate
AMD Sempron 2600+ 2200ms
AMD Athlon X2 64 4400+ 800ms
Intel Core2 Duo E8400 420ms
Tom’s Hardware 2009 DT CPU charts Lame, Price, Calc. times to gen.
Intel Core2 Duo E8400 163s
Intel Ci5 660 (3.33 GHz) 127s, 189€, 327ms
Intel Ci3 530 (2.93 GHz) 156s, 112€, 402ms
AMD Phenom II X6 1055T (2.8 ~ 3.3 GHz) 188s, 188€, 484ms
AMD Athlon II X2 250 (3 GHz) 206s, 70€, 531ms

Comparing the measured times of those three CPUs, I noticed the Core2 has much more cache available. So I measured the memory speed of Athlon X2:
L1 = 6688 MB/s, 64 kB
L2 = 3747 MB/s, 512 kB
RAM = 2607 MB/s, 2015 MB

This looks like a bad joke, comparing this to Intel’s Core2 Duo. So to stay on the safe side, I decided the CPU must have a lot of L3 cache. And the cache is a lot faster on the AMD Phenom II – measured on a friend’s computer:
L1 = 33000 MB/s
L2 = 10000 MB/s
L3 = 7000 MB/s
RAM = 4000 MB/s

Unfortunately, Intel i CPU doesn’t support ECC. And the Xeon was too pricey – as mentioned, at the end of summer 2010. And because of L3 cache and budget, Phenom II stays as the only choice. I have chosen a 6 core CPU, because of the web crawlers. But it seems that wasn’t necessary, because you can set the time between a “crawl” in robots.txt file. Looking a the statistics, using “atop”, a dual to quad core CPU would be enough. If I’d make a choice now, I would go for Intel Xeon E3 series – it’s relatively cheap, supports ECC and is probably significantly faster. This way you can enable more plugins. And how does the new Phenom II CPU perform? The page is generated in 370ms. Even better then I hoped it would.

qTranslate plugin performance optimization

Installing a qTranslate, multi-language enabling WordPress plugin, can increase the time in which your blog loads.

My installation of the blog, with all the plugins (including qTranslate), loads in about 600ms. Disabling qTranslate reduces the load time to 320ms.

To see what is happening I installed Xdebug and WebGrind for making a performance review of WordPress and qTranslate. Xdebug is probably a package in your Linux distribution (at least it comes with Ubuntu Server as php5-xdebug). Instructions for installation and usage can be found here.

And the WebGrind screenshot:
qtranslate default install

As you can see, function named “qtrans_use” takes 20% of all loading time. Combined with all “qtrans*” functions, the share goes to more than 30%. That is really inefficient compared to the analysis with qTranslate disabled – note that the loading times are lot longer with Xdebug enabled:
qtranslate disabled

As none of those two options are satisfactory, I tried to optimize qTranslate. Expanding “qtrans_use” function reveals that 48% of calls are recursive (function calls itself) and for 38% a function “qtrans_useCurrentLanguageIfNotFoundUseDefaultLanguage” is to blame:
qtranslate default install qtrans_use expanded

After a little searching I found out that WordPress options are getting translated too. Unfortunately there is no cache implemented to remember the translations so the whole list gets translated every time one option is called. So I disabled (just put a # or // in front) the line “add_filter (‘option_’.$option, ‘qtrans_useCurrentLanguageIfNotFoundUseDefaultLanguage’, 0);” in “qtranslate_core.php”, which is located in “wp-content/plugins/qtranslate”. This improved the loading times a bit – to 480ms. A 120ms improvement. Not much, but it helps when a Google crawler goes through hundreds of pages.

qtranslate optimized

You can see that number of calls of “qtrans_use” were reduced form 13517 to 2221. qTranslate now contributes little more than 12% to the whole loading time. There is some room for improvement, but probably this includes substantial modifying of the plugin. You must also know, that some plugins can have their options translated and in that case add a little “if” conditional clause around that “add_filter” call to let only the desired options through.

Revisions and growing indexes

If you look into the WordPress database, table wp_posts, you’ll see lots of records … much more then the actual displayed posts and attachments combined. The reason lies in revisions.

Among others, by default, every post save creates a revision. For the blog, I’m managing, the statistics are: 809 posts and 6287 revisions. And I must mention that I already deleted the revisions for first 391 posts. That means above 15 revisions per one post.

This does not present a big problem if you do not use a plugin such as Yet Another Related Posts Plugin. These related-posts plugins create a fulltext index on post title and content. And that means, if you want speedy searching for the related posts, big indexes – in my case 55 MB. And of that size, 75% is never used. And if didn’t have deleted some revisions before, the MBs and percentage would be even higher. And on webhosting servers this can present a problem because of high memory usage.

Unfortunately MySQL does not (yet) support partial indexing with which you could disable indexing of the revisions. Because of this revisions must be deleted. Some plugins are addressing this problem, but I was unable to find one, which deleted records older than certain number of days. It’s only all or nothing.

Searching the web (I’ll include the link if I stumble upon it again) I found a deletion code and added a time filter (use at your own risk):

DELETE p, tr, pm
-- SELECT p.*, tr.object_id, pm.meta_id
FROM wp_posts p
JOIN wp_posts pp ON pp.ID = p.post_parent
LEFT JOIN wp_term_relationships tr ON tr.object_id = p.ID
LEFT JOIN wp_postmeta pm ON pm.post_id = p.ID
WHERE p.post_type = 'revision'
AND pp.post_type = 'post'
AND pp.post_status = 'publish'
AND p.post_date < ADDDATE(CURDATE(), INTERVAL -2 MONTH)

This can then be used to make a cron job (Linux) or schedule (Windows) to periodically delete old revisions. Linux example: 'mysql --user=? --pass=? --execute "INSERT CODE HERE" DATABASE'.