Alternative PHP Cache (APC) on Debian 6

APC usage and hit chartAfter spending some time reviewing way to increase WordPress performance in ways beyond simple file caching I decided to give APC (Alternative PHP Cache) and extended try. I use mod_fastcgid on my server, as opposed to mod_fastcgi. Web searches indicated that mod_fastcgi would be a better choice with APC since it could share the same cache between processes. With mod_fastcgid each process has its own cache, which would also mean more memory usage. I have a relatively small server, a 512 MB VPS. But in looking at free memory it appeared I’d be OK as long as I was careful and had some upper limits. Also, I had has some issues getting mod_fastcgi to run in the past so I didn’t want to try that again.

Why APC

APC is an opcode cache. Opcode caches will cache the compiled php script in memory so it does not have to be interpreted each time it is called. WordPress is built with PHP so this should help performance. (Technically “compiled” is probably technically wrong but does get the point across.) Other opcode caches are XCache and eAccelerator.

eAccelerator hasn’t been updated in a couple years so I was hesitant to try it. APC is by the same guys who develop PHP and it’s supposed to be rolled into PHP 6. So I decided to give it a try first. I may give XCache a try after running APC awhile. Although, if I’m honest, I probably won’t bother if APC works OK.  They can’t co-exist so I can only install one at a time.

The one reason not to use APC is memory. I’ll be improving cpu utilization and improving performance at the expense of memory.

Throughout this article the terms cache and caching refer to the opcode caching provided by APC.

Installation

APC is in the Debian 6 repositories so it can be installed from there. I use aptitude, so I run:

aptitude install php-apc

Once it’s installed I reload Apache and APC is running. No admin interface is running so I extract the apc.php file by running:

gzip -dc /usr/share/doc/php-apc/apc.php.gz > /path/to/my/mngmt/website/apc.php

The /path/to/my/mngmt/website is a website I have that’s a catch-all for various server related pages. Access to it is restricted and since it does include some version information and files paths you may want to restrict access. I can open this page and get statistics about APC usage.

When I opened apc.php and did a version check I saw that I was several versions behind. I install 3.1.3p2 from 2009 and the latest stable version was 3.1.9 fro May 2011 and there was a beta version from April 2012. APC was working and limited testing didn’t show any problems but I decided to update.

Update Through PECL

PECL is the PHP Community Extension Library so its installer will be installed along with packages it needs to compile APC. Finally APC will be installed.

sudo aptitude install php-pear php5-dev apache2-dev

Once these are installed I can install (upgrade) apc

sudo pecl install apc

Reloading Apache finishes things off. Because I had first installed apc through aptitude I didn’t have to do any manual edits.

Configuring APC

Configuring APC is where the real work is needed. I did several iterations testing different configurations. At this point I was testing for performance, I was looking to see what worked and what didn’t.

The apc.ini shows the settings I was most interested in and what I’ve currently settled on. Memory is the biggest issue with APC. As I mentioned, each FCGID process creates it’s own cache. I’ve configured my server for a maximum of three processes so I can control memory usage. The easy way to determine how much memory to allocate is to divide my free memory (without caching enabled) by 3 (the max number of FGCID process I’d have) and leave a little for OS buffers and breathing room. For me this worked out to about 100 MB (I had about 350 MB free. But that calculation is a limit, which doesn’t mean it’s a good size.

I’m going to jump ahead and show the configuration I settled on.My /etc/php5/conf.d/apc.ini contains the following entries:

extension=apc.so
apc.enabled=1
apc.cache_by_default=0
apc.filters=”+(osquest.com|stats.rrn.co)/.*”
apc.shm_segments = 1
apc.shm_size=90M
apc.slam_defense=0
apc.stat=1
apc.ttl=0
apc.user_ttl=0

The first two lines tell PHP to load the APC extension and enable caching.

I started off with a 256 MB cache and let it cache all my sites. This was much too large so I’d have to monitor usage in case things started swapping to disk. Cache usage seemed to level off at around 90 MB after a few hours. I did go through a load the home page of all my sites (many are test sites or receive little traffic). This gave me an rough (and lowball) idea of what I’d need to cache everything. So this was right at my 90 MB estimate. In practice a 90 MB cache size resulted in a lot of cache fragmentation and cache reloads which isn’t an optimal situation.

I took the approach of reducing my FCGID processes to 2 and increasing the cache to 128 MB. This provided better cache performance but shortly after midnight I received a series of “[warn] [client 69.164.211.225] mod_fcgid: can’t apply process slot for /home/rayn/public_html/fcgi-bin.d/php5-default/php-fcgi-wrapper” errors. That IP address is my server so some daily maintenance scripts needed more PHP processes than were available. Later in the morning one visitor experienced a timeout error. It may have been unrelated but timeout errors were previously unheard of. So I switched back to three FCGID processes.

At this point it was obvious I had to cache less than everything. There’s only two sites that have enough traffic to benefit from caching so I decided limiting caching to them. I handled this through the next two lines. apc.cache_by_default=0 turns off caching for all sites. It now needs to be turned on explicitly. It’s turned on through the apc.filters line. The + means the filter is to add the matching files to caching. (By default the filter removes matches from caching.) The syntax is a POSIX regular expression and can be comma delimited. I had trouble getting a comma delimited list to work although all indications are that it should. So I ended up combining them into this one expression. Here’s a regular expression cheat sheet. I cache this site and one subdomain on a second site.

Caching doesn’t have to be enabled for a site in order to use it with a WordPress plugin that supports APC. For example, W3 Total Cache will use the cache even if it’s not enabled for the site.

I specified apc.shm_segments = 1 even though 1 is the default setting. I don’t want later version to change the default and start eating more memory. I specify apc.shm_size=90 to create a 90MB cache per FCGID process.

I also specified apc.stat=1 even though that’s the default. There’s debate about whether setting this to 0 will improve performance. A setting of 1 means that APC will check each script on request to see if it’s been modified. If it’s been modified it will be re-cached. A setting of zero disables this feature, The official documentation recommends setting it to 0 on a production server. A setting of 0 can cause problems as PHP changes (such as WordPress plugin updates) would go unnoticed. Once things are solid I may change it to 0 to see if there’s an improvement or any problems. My initial feeling is any improvement would be unnoticeable for my site and wouldn’t be worth the hassle of needing to manually flush the cache when I make changes. I’m also concerned it will cause problems with some plugins.

To make sure memory usage doesn’t get out of control I’ve also limited my FCGID processes to only three and I don’t allow them to spawn any child processes. I also don’t terminate to FCGID processes since this would clear the cache. This could be a potential memory problem so I’ll have to monitor it.  The following settings are specified in my /etc/apache2/conf.d/php-fcgid.conf.

DefaultInitEnv PHP_FCGI_CHILDREN 0
DefaultInitEnv PHP_FCGI_MAX_REQUESTS 0
FcgidMaxProcesses 2

This should keep memory usage below the physical memory in my server. PHP_FCGI_MAX_REQUESTS is set to zero based on this in the Apache mod_fcgid documentation:

By default, PHP FastCGI processes exit after handling 500 requests, and they may exit after this module has already connected to the application and sent the next request. When that occurs, an error will be logged and 500 Internal Server Error will be returned to the client. This PHP behavior can be disabled by setting PHP_FCGI_MAX_REQUESTS to 0, but that can be a problem if the PHP application leaks resources. Alternatively, PHP_FCGI_MAX_REQUESTS can be set to a much higher value than the default to reduce the frequency of this problem. FcgidMaxRequestsPerProcess can be set to a value less than or equal to PHP_FCGI_MAX_REQUESTS to resolve the problem.

I’ll have to monitor this. It’s possible the FCGID memory could keep growing until physical memory is filled.

PHP_FCGI_CHILDREN is set to 0 to prevent children from being spawned. From the Apache documentation:

PHP child process management (PHP_FCGI_CHILDREN) should always be disabled with mod_fcgid, which will only route one request at a time to application processes it has spawned; thus, any child processes created by PHP will not be used effectively. (Additionally, the PHP child processes may not be terminated properly.) By default, and with the environment variable setting PHP_FCGI_CHILDREN=0, PHP child process management is disabled.

Additional Notes

The screenshot below (click to open full size) shows my status (using apc.php) a couple hours after the caching started. The screenshot was done when the cache was still configured for 256 MB. I am running a unique cache for each fdcgid process so refreshing the page moves between the instances (but randomly). The size of the three caches stay pretty close to one another but they are clearly unique caches.

APC Info Page

When running top each of my three fcgid processes levels off at about 25% memory usage. So if my math is right this is 75% of 512 MB. Yet free (and top) show 350 MB still free. If each was allocating the full 256 MB to its own cache then it would be well over 100%, assuming free and top saw the shared memory it was using. So I’m assuming free doesn’t see the shared memory and reports is as available. In my limited memory this makes sense since the shared memory is still available to any application. The actual usage of my three caches never grew about the available free memory.

APC does require some work to configure and optimize it. To start with I’d enabled it for all my sites and PHP. This resulted in a lot of cache fragmentation and a few cache reloads after a few hours.

Benchmarks and timings on a test server are fine for comparisons, but I’ll need to run a configuration for awhile to see how it does, then start tweaking it. Right now my server seems healthy. Using APC did improve performance over no caching of any type, but I’m not sure of it’s benefit once page caching is used, especially if using a preloaded cache.

Right now my main concern is that I have too few FCGID processes, only two. I used some load testing tools and it didn’t cause any errors. I do have page caching enabled to PHP shouldn’t be needed when serving a page from the cache. I’ll be monitoring my error logs.

Anyone else with experience using APC with WordPress?

Optimizing My Website – Low Hanging Fruit

Image a of red rocket shipIn my previous article I looked at five web performance tools to optimize my website. In this article I’ll cover the website optimizations they led me to make. These were all done over a weekend of off and on work and experimentation. The more in depth things were left for a later time but these got me off to a good start, cutting the size of some pages in half.

Image Optimization

The fist thing I looked at was image optimization. This isn’t an art or photo site so anything I could do to reduce image file size would be helpful. Google’s Page Speed tool found plenty of room for improvement in my images. The most obvious area for improvement were the site header and 3 images that appear on every page.

My site header was a GIF, Google Page Speed provided an optimized PNG file for replacement. So I just copied that to my server and changed the template to load the PNG instead of the GIF. This reduced my header by 62% and will save 3KB every time it loads. The other images were ads that weren’t served by me, more on these later.

While taking the optimized image created by Page Speed is fine for one or two images it isn’t viable for more than some testing. Google’s site recommended OptiPNG as one method of compressing images so I downloaded that.

I used OptiPNG in it’s simplest form, just passing the files to compress on the command line. Eventually I set up directory for the files and a batch files to optimize all the files in the directory. I’d copy down an entire directory of files from my site, optimize them and copy them back up to replace the original files (of course, I had a backup). While OptiPNG will convert GIFs to PNGs and I had a few GIFs I didn’t go into the pages and change the links unless I decided to optimize that entire page. Image size reduction would vary but was about 40% overall with larger files being around 55%.

Javascript Optimization

This fell into two categories – offsite javascript and onsite javascript.

For the offsite javascript the choice was simple. Keep it, and the feature it provided, or dump it. In one case it was an ad and the javascript was relatively slow and it also loaded an image, so I dumped it. In another case it was a plugin which I also dumped. More on this in plugin optimization.

As for compressing Javascript files I decided that using a utility to compress the actual files wouldn’t be beneficial to me. I’d have to redo it every time they changed and they were part of the theme and plugins. Since I had already enabled GZip compression on my server I added the configuration option to compress javascript. In my case I added application/javascript to the mod_deflate configuration. (My reading indicated some Apache servers may use a mime type off application/x-javascript for javascript). This was enough to make Page Speed happy.

CSS Optimization

I probably could have also compressed the css files through GZip compression and left it at that. But I also decided to minify the css files since they rarely change. Minifying is the process of removing whitespace from the css files.

Rather than find a tool to minify my css files I simply used the files created by Google’s Page Speed plugin for Firefox. This creates the files and names them cryptically randomly so I had to follow the link to each file and figure out which was which. I only had three files so this was quicker than searching for another tool and doing it myself.

I use the Thesis theme for WordPress and after I minified the css for the theme I would get a “upgrade Thesis” message whenever I went into the Thesis options in the WordPress dashboard. The theme works just fine but if I want to make any changes I need to restore the un-minified files. This is no big deal since I just rename the files on the server and name them back when I’m done. If the css actually changes I’ll need to minify the updated file.

Plugin Optimization

The final item I tackled to optimize my website is the one I spent the most time on. Taking a look at my plugins. The Share This plugin I used to allow my posts to be bookmarked called some external javascript which was a bit slow. I decided to look for a replacement. I wanted to find something that would be completely hosted on my server which eliminated a bunch of replacements. One I found loaded a 172kb image for every page, even ones where the plugin wasn’t used. I wasn’t willing to use an image this big. I finally found one called Social Bookmarks that fit the bill as long as I kept some options off.

The rest of my plugins either didn’t detract from performance or were worth the cost. There wasn’t a lot to review as I’ve always tried to limit the number of plugins I use.

Summary

Some other recommendations would have involved more than I was willing to do. Some of the recommendations were related to css and I wasn’t about to rewrite the css provided with the them.

Compressing the images gave me the biggest overall improvement in performance with minifying the css coming in second. The css compression helped because it loads for every page and there was a significant reduction in file size. (Of course, client side caching can also help the load times.)

While replacing the Share This plugin was only a minor improvement, experimenting with other plugins showed me how some plugins can significantly impact performance.  Some plugins I looked at really hurt perf0rmance.

Back when I started looking at performance Google Webmaster Tools told me my site was slower than 66% of all websites and took an average of 4.2 seconds. On December 31st the Webmaster Tools tell me the that only 49% of all websites are faster than this one and the average load time is 2.9 seconds. This is hardly a scientific analysis of my attempts to optimize my website, but at least it shows an improvement.

Website Performance Tools

Picture of a red sports carI mentioned in my previous Trail Log that my look at the new Site Performance feature in Google Webmaster Tools tools led me down a rat hole looking into ways to optimize my website. Since I use WordPress I approached this as WordPress optimization, but the reality was most of this was basic web optimization. I did use these tools to help decide which WordPress plug-ins & features to use. Plug-ins or features that degrade performance for little benefit were dropped. In this article I’ll discuss the tools it led me to. I’ll cover the optimization changes I made to optimize my website in another article.

Google Webmaster Tools Site Performance

This is a Google Labs feature so it’s experimental. The information is not collected from Google’s web crawlers, rather, it’s collected by Google Toolbar users that have the PageRank feature active.

The top of the Site Performance page will display a graph showing performance history. This is what mine looks like today.

performance

It will also say what percentile your website falls into. When I first went in a couple days ago 73% of sites were faster than mine. Today 66% are faster. While I did make some optimizations this seems like a big drop in such a short time. I suspect much of this is due to the pages people are visiting. Unfortunately I didn’t keep track of load times but today it tells me page loads average 4.2 seconds. I’ve no idea what the September spike is due to. The October spike is around when I changed to the Thesis theme although I’m not sure why it spiked then dropped, it’s not like I changed the theme a lot. There may have been a plug-in I was trying and later stopped using.

They also list example pages along with suggestions to improve performance on the page. They don’t fudge the results in their favor as compressing the Google Analytics code snippet was listed. I didn’t spend much time on this information. Instead I installed the Page Speed add-on for Firefox that was promoted on the page.

Page Speed Add-On for Firefox

The Page Speed add-on plugs into the Firebug add-on so you’ll need that first. The plug-in is simple to use. Just start Firebug, load the page and wait for the load to complete entirely, then click the “Analyze Performance” button.

You’ll get a report showing the different categories on possible improvements. If you click the category name you’ll go to a help page to lean more information. The suggestions are listed in priority order and color coded as red, yellow or green. This was the jumping off point for most of the enhancements I made. Some of them I’d never heard of, like “minify css” while others just required tweaking an existing configuration, such as enabling gzip compression for javascript. There’s also items that aren’t realistic for me to implement, things that are core WordPress code such as an inline script in the header that appears before CSS files.

Page Speed will also provide optimized versions of files that needs it. These can be a little hard to deal with since they are given long, random names. Links are provided so you can open them up and cut/paste. In the cases of images I found it easier to use another tool and just compress the original images rather than finding the Page Speed generated files. In the case of CSS where I only had a few files I just used the Page Speed minified files.

Between the suggestions on this page and the help links I was able to make significant improvements in performance. Unfortunately I didn’t do timings in the beginning and Page Speed doesn’t do actual timings so while things are better I’ve no idea how much better.

The specific optimizations are best covered in a follow-up post.

Web Page Analyzer & Pingdom Tools

As I mentioned, Page Speed didn’t give me any timings. Once I’d hit the big things Page Speed identified I wanted to look at specific timings. Web Page Analyzer is a website that does what it says. I provides a plethora of information. Like Page Speed it provides information on optimization techniques and whether or not they’re in use. It’s report is all text and provides a wealth of information along with suggestions.

Pingdom Tool report Like Web Page Analyzer Pingdom Tools is web based and breaks down the load time for each item on the page. It presents it’s information in a graphical report such as the one to the left. (Click the thumbnail for full size). It also provides a summary box with stats like total load time and number of objects broken down by type. Unlike the other tools it doesn’t offer any suggestions.

The Pingdom tool graph is a nice quick way to see what order things are loading in and how long each item takes. It makes it easy to drill down into what’s taking the most time. Using this I found a plug-in that was trying to load a missing file via CSS. It also called out another plug-in that was loading a file over 170KB via CSS and even on pages the plug-in wasn’t used. Removing that plug-in alone cut the load time on this example page by 50%. That plugin just wasn’t worth the performance cost.

I found Pingdom tools first and it helped me find the big hits that Page Speed wasn’t calling out. I’ll continue to use both, first Pingdom tools to quickly call out the really slow objects then Web Page Analyzer gets into the nitty gritty. Web Page Analyzer provides more detailed information, as does Page Speed, but adds timings. The Web Page Analyzer timings are based on the object size and not the actual response from my server. I like this because it makes it easier to gauge changes I make and provides a consistent benchmark. Pingdom Tools & Site Performance are based on actual experience. This would take into account my server performance and any network congestion. This can be used to gauge the response a user is likely seeing.

OptiPNG

OptiPNG isn’t a benchmarking tool, rather it’s the one program I added to my web page creation workflow. As the name suggests, it optimizes PNG files. Actually it will also convert JPEG and GIF files to PNG and optimize them. It’s an open source command-line utility. At the basic level it’s easy to use. Just pass a file name (wildcards allowed) and it will optimize and replace the file.  There are various optimization levels along with other options (such as saving a backup file) that can be set on the command-line.

YSlow

YSlow is a Firefox add-in from Yahoo that blugs into Firebug like Page Speed. YSlow has been around awhile and I’d heard of it, but it wasn’t at the top of my mind and I didn’t come across it until late in the process. So, while I now have it installed, I haven’t really used it. I provides a nice scorecard to help call out problem issues and it provides links to more information. It looks like it presents information similar to Page Speed but with the addition of some summary statistics. If anything, there’s too much information. At this point I don’t want to take the time to delay this article until I’ve had time to explore YSlow. It would take too long and probably wouldn’t change anything for me. Of course, I may regret that statement in the future. Any YSlow fans out there?

Summary

This covers the five tools that I’ve gravitated to while exploring WordPress optimization. They helped me see what was going on with my website and within WordPress. While it was nice to see some performance improvements, the exploration was great fun which was the primary motivation to continue. (It’s five tools because I exclude YSlow since I didn’t use it, but I wanted to mention it since it’s certainly seems useful.)

Now comes the tedious task of implementing the optimizations on individual pages to fully optimize my website. I’ve already got the site-wide optimizations done. Images that appear on all pages are compressed, my CSS is minified and gzip compression has been enabled for javascript. Unfortunately I hadn’t learned learned about the benchmarking websites so I don’t have solid number for the improvement. But these changes got rid for the red in Page Speed. I’ve also changed my posting work flow so anything new is optimized when it appears. Hopefully my Site Performance stats in Webmaster tools will improve over time.

Next on the agenda to optimize my website  is to optimize my most visited pages. The main effort there will be to compress and optimize any images. The rest of the page optimization was covered by the site wide changes in most cases.

The OS Quest Trail Log #46: Housekeeping Edition

Picture of Santa Celebrating with a beerIt’s been over a month since I’ve done a Trail Log and I got some time for blog updates this weekend so I might aw well do one. The day job has kept me busy and pretty well burnt out by the time I get home so I haven’t dived in depth into anything for awhile. But recently I’ve started to take a look at things that have piqued my interest.

Optimization

I’ve been on a Google kick lately, first looking at Picasa Web (because of the Google Eye-Fi offer) followed by Google Public DNS. Then I went down a unexpected rat hole with the newly added site performance feature in Google Webmaster Tools. This was actually a quit enjoyable foray into site optimization and I learned a lot in a short amount of time. One thing I learned is I have a lot more to learn. I’ll have one or two posts out in the next couple of days on this and other tools along with the optimizations I made. I’ve already made some changes like compressing the header and other images that appear on multiple pages. This was all pretty ad-hoc and I didn’t keep very good notes, but I want to start optimizing some of my most viewed pages so I’ll write up the posts when I do them.

Regular visitors may also notice the social bookmarking links at the bottom of each post have changed. This is also the result of the optimization testing. I tried several and all had a pretty significant performance impact. Several others I tried were just as bad but for other reasons. The one I’m using now, Social Bookmarks, provides the best performance. Even so, I can’t use all it’s features and there was a bug in it’s CSS that I had to fix (it was trying to load a non-existent image. This was all part of the fun, since trying to optimize things required me to try and figure out how things worked. It comes as no surprise that all those web 2.0 bells and whistles are expensive.

Along the optimization lines I’ve also been researching a new web host. I really like Slicehost and have no complaints. If I was running a business I’d be staying with them. They have solid support. The best part is their server has been reliable enough that I’ve rarely needed support and only to have them do things I couldn’t do myself (like kernel upgrades before they added that ability to the console). But the reality is there are now competitors with more hardware for the buck. After the holidays I’ll pick one and give it a try. The question is can I get the same or better for less money. No sense starting now since it would site idle most of the time.

Other Website Changes

I’ve made a couple other changes, only one is worth mentioning. I had been closing comments 90 days after a article was published as a way to stop spam. I turned this off and re-enabled comments. If a certain posts attracts spam and isn’t otherwise worth commenting on I’ll turn off comments for the post. I don’t check the spam queue for false positives so if you post a comment you should limit the number of links. Also, when moderating comments I’m more likely to spam a short “nice post” type comment instead of checking any links.

I’m turning it back on because I’m even worse at answering email than I am with replying to comments. At least with comments others can join in. I still have a couple emails I need to respond to. Still feel free to send an email. I’ll just apologize in advance for the slow response.

Frustrations

Of course, things haven’t been frustration free. As I started writing this article my netbook woke up in the other room and started doing a backup to my Windows Home Server. This serves as a reminder then when a PC does a backup to my WHS it slows the server down to a crawl for everything else. So tonight my streaming video stopped. It’s not usually a problem since the backup occur in the dead of night. But occasionally they aren’t. Eventually I’ll have to dig into this. Unfortunately once the problem occurs I can’t get a remote connection to see what’s going on, so I actually have to set up a test and be prepared. On the plus side, the problem seems consistent so once I set things up it should happen the first time I test.

The new Handbrake also stopped working on me. I’d upgraded it and it was working without a problem. Then I applied Apple’s latest updates and it stopped (that Mac Mini is still on Leopard). I’ll try a Handbrake re-install first. VLC is also pretty old so my next step will be to try and update that since Handbrake uses it’s library. It’s just one of those annoying things that’s not hugely important yet.

DNS Errata

In between writing my Google Public DNS article and now I listened to the latest Security Now podcast by Steve Gibson. In it, he mentions a DNS benchmarking tool that he’s written. He’s written some great utilities so I’ll be sure to check it out. Steve is also my hero because he writes his utilities in assembler. They rarely need updating for bugs and they’re nice and small. His DNS benchmark utility is 150kb while the namebench utility is over 8MB even when compressed. And it’s not because his is command line, it’s a Windows GUI too.

Happy Holidays

SantaHi That’s about it for this trail log. I might get another Trail Log in before the end of the year since I have some days off, but there’s no guarantee. I’ll be happy if I get the optimization article(s) done this weekend. The coffees’s on so we’ll see how long I can go. Thanks to the miracle of WordPress post scheduling I’m hoping to get pretty far into the future so this site isn’t so dormant. So happy holidays to everyone.