PHP factor 30 performance difference from Linux to

Question

Our team is working is developing WordPress plugins and provides hosted instances on a couple of independent servers. Our WordPress installation is managed by Git, all servers have the same source & WordPress setup deployed, only domains & actual data in the database varies. For each installation, MySql is running on the same host. WordPress is running exclusively on each server.

However after having deployed this setup on a Windows Server 2008 RC2, we noticed a drastic performance difference compared to our other servers: page generation time goes up from avg. 400ms to 4000-5000ms for pages generated with PHP. For static resources delivered by Apache only, speed is about the same as on linux.

So we took some steps to narrow down the problem:

1. Make sure there is no antivir-software running or other windows domain stuff interfering
2. Collect profiling data to identify the timekillers during script execution
3. Test different server & hardware setups
4. Double-check both Apache and PHP configuration for obvious configuration errors

After some profiling we quickly noticed that the evaluation of regular expressions is horribly slow on our windows machines. Evaluating 10.000 Regular expressions (preg_match) takes about 90ms on Linux and 3000ms on Windows.

Profiling, system tests and configuration details are provided bellow. We don't want to optimize this script (which we do know how to do). We want to get the script to run approximately the same speed on windows as on Linux (given the same setup regarding opcache/...). No need to optimize the memory footprint of the script too.

Update: After some time, the systems seems to run out of memory, triggering out of memory exceptions and random allocations. See bellow for more details. Restarting Apache/PHP fixed the problem for now.

Trace to _get_browser is:

File (called from)
require wp-blog-header.php (index.php:17)
wp (wp-blog-header.php:14)
WP->main (functions.php:808)
php::do_action_ref_array (class-wp.php:616)
php::call_user_func_array (wp-includes/plugin:507)
wp_slimstat::slimtrack  (php::internal (507))
wp_slimstat::_get_browser (wp-slimstat.php:385)

Update 2: Some some reason I can't remember we went back to activating PHP as an Apache Module on our servers (the same which deliver bad performance). But today they run blazingly fast (~1sec/request). Adding Opcache brings this down to ~400ms/req. Apache/PHP/Windows remained the same.

1) Profiling Results

Profiling was done with XDebug on all machines. Usually we only collected a few runs - those were enough to reveal the location where most of the time (50%+) was spent: the method [get_browser][1] of the WordPress plugin wp-slimstats:

protected static function _get_browser(){
    // Load cache
    @include_once(plugin_dir_path( __FILE__ ).'databases/browscap.php');
    // browscap.php contains $slimstat_patterns and $slimstat_browsers

    $browser = array('browser' => 'Default Browser', 'version' => '1', 'platform' => 'unknown', 'css_version' => 1, 'type' => 1);
    if (empty($slimstat_patterns) || !is_array($slimstat_patterns)) return $browser;

    $user_agent = isset($_SERVER['HTTP_USER_AGENT'])?$_SERVER['HTTP_USER_AGENT']:'';
    $search = array();
    foreach ($slimstat_patterns as $key => $pattern){
        if (preg_match($pattern . 'i', $user_agent)){
            $search = $value = $search + $slimstat_browsers[$key];
            while (array_key_exists(3, $value) && $value[3]) {
                $value = $slimstat_browsers[$value[3]];
                $search += $value;
            }
            break;
        }
    }

    // Lots of other lines to relevant to the profiling results
  }

This function similar to PHP's get_browser detects the browser's capabilities and OS. Most of the script execution time is spent in this foreach loop, evaluating all those preg_match (~approx 8000 - 10000 per page request). This takes about 90ms on Linux and 3000ms on Windows. Results were the same on all setups tested (picture shows data of two executions):

Sure, loading two huge arrays takes some time. Evaluating regular expressions too. But we'd expect them to take approximately the same time on Linux and Windows. This is the profiling result on a linux vm (one page request only). The difference is pretty obvious:

Another time killer was actually the Object-Cache WordPress uses:

function get( $key, $group = 'default', $force = false, &$found = null ) {
    if ( empty( $group ) )
        $group = 'default';

    if ( $this->multisite && ! isset( $this->global_groups[ $group ] ) )
        $key = $this->blog_prefix . $key;

    if ( $this->_exists( $key, $group ) ) {
        $found = true;
        $this->cache_hits += 1;
        if ( is_object($this->cache[$group][$key]) )
            return clone $this->cache[$group][$key];
        else
            return $this->cache[$group][$key];
    }

    $found = false;
    $this->cache_misses += 1;
    return false;
}

Time is spent within this function itself (3 script executions):

On linux:

The last real big time killer were translations. Each translation, loaded from memory, takes anything from 0.2ms to 4ms in WordPress:

On linux:

2) Tested systems

In order to make sure virtualization or Apache do affect this, we tested this on several setups. Antivir was disabled on all setups:

• Linux Debian, Apache 2 & PHP on up to date stable releases. This is the same for developers running in their virtual machines as for staging/live servers. Acting as a reference system of desired performance. Either run in our office or at some hosting provides (shared space). Windows Systems had between 4GB and 8GB of RAM, at all time memory usage was bellow 50%. Virtualizations never run Windows & Apache at the same time.
• Life-Servers, running at T-Systems (managed virtualized servers), on VMWare Player
  • Win 2008 R2. Apache 2.2.25 + PHP 5.4.26 NTS,VC9 as fastcgi module
  • Win 2008 R2. Apache 2.2.25 + PHP 5.5.1 NTS,VC11 as fastcgi module
  • Win 2008 R2. Apache 2.2.25 + PHP 5.5.1 NTS,VC11 as apache module
  • Win 2008 R2, Apache 2.2.25 + PHP 5.5.11 TS,VC11 as apache module (that's the fast one I mentioned in the update 2)
• On a local machine, Host: OpenSuse, Virtualization: VMWare player, same as @T-Systems. To avoid their infrastructure influencing us:
  • Win 2008 R2. Apache 2.2.25 + PHP 5.4.26 NTS,VC9 as fastcgi module
  • Win 2008 R2. IIS7 + PHP 5.4.26 NTS,VC9 as fastcgi module (with and without wincache)
  • Win 2012. IIS * + PHP 5.5.10 NTS,VC11 as fastcgi module (with and without wincache)
• On a local machine without virtualization
  • Win 2008 R2. Apache 2.2.25 + PHP 5.4.26 NTS,VC9 as fastcgi module

Profiling results as mentioned above were the same on the different systems (~10% derivation). Windows was always a significant factor slower then Linux.

Using a fresh install of WordPress & Slimstats resulted in approx. the same results. Rewriting the code is not an option here.

Update: Meanwhile we found two other Windows Systems (both Windows 2008 R2, VM & Phys) where this complete stack runs quite fast. Same configuration though.

Update 2: Running PHP as apache module on the Life-Servers was slightly faster then the fastcgi method: down to ~2sec, 50% less.

Running out of Memory

After some time, our Live-Server stops working at all, triggering these out of memory exceptions:

PHP Fatal error:  Out of memory (allocated 4456448) (tried to allocate 136 bytes)
PHP Fatal error:  Out of memory (allocated 8650752) (tried to allocate 45 bytes) 
PHP Fatal error:  Out of memory (allocated 6815744) (tried to allocate 24 bytes) 

This happens at random script locations. Obviously the Zend Memory Manager is not able to allocate more memory, although the scripts would be allowed to do so. At the time if incident, the server had about 50% of free RAM (2GB+). So the server does not actually run out of ram. Restarting Apache/PHP fixed this problem for now.

Not sure if this problem is related to the performance issues here. Yet as both issues seem to be memory related, its included here. Especially we'll try to reproduce the settings of the Windows-Tests that provided decent performance.

3) Apache & PHP Configuration

... probably do not have any common pitfalls. Output-Buffering is enabled (to default), multibye override disabled, ... If any option(s) are of interest we'll happily provide them.

Output of httpd.exe -V

Server version: Apache/2.4.7 (Win32)
Apache Lounge VC10 Server built:   Nov 26 2013 15:46:56
Server's Module Magic Number: 20120211:27
Server loaded:  APR 1.5.0, APR-UTIL 1.5.3
Compiled using: APR 1.5.0, APR-UTIL 1.5.3
Architecture:   32-bit
Server MPM:     WinNT
  threaded:     yes (fixed thread count)
    forked:     no
Server compiled with....
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses disabled)
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=256
 -D HTTPD_ROOT="/apache"
 -D SUEXEC_BIN="/apache/bin/suexec"
 -D DEFAULT_PIDLOG="logs/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error.log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

mpm_winnt_module configuration:

<IfModule mpm_winnt_module>
    ThreadsPerChild 150
    ThreadStackSize 8388608 
    MaxConnectionsPerChild 0
</IfModule>

Excerpt of php.ini:

realpath_cache_size = 12M
pcre.recursion_limit = 100000

4) Current suspected reason

Old assumption:

All three examples heavily rely on big arrays and string operations. That some kind seems to be the common factory. As the implementation works ok'ish on Linux, we suspect this to be a memory problem on Windows. Given there is no database interaction at the pin-pointed locations, we don't suspect the database or Server <-> PHP integration to be the problem. Somehow PHP's memory interaction just seems to be slow. Maybe there is someone interfering with the memory on Windows making access dramatically slower?

Old assumption 2:

As the same stack runs fine on other Windows machines we assume the problem to be somewhere in the Windows configuration.

New assumption 3:

Actually I am out of assumptions. Why would run PHP that much slower as fastcgi then as apache module>

Any ideas on how to verify this or finding the real problem in here? Any help or direction for fixing this issue is highly welcome.

Answer 1

Windows has lots of services/policies that restrict, prevent, protect, control and etc usage of the computer in every situation.

A good Microsoft certified specialist will be able to solve your question within minutes, because they will have the experience to tell exactly which settings/services/policies to check and disable/enable/change settings, so that the PHP scripts are executed faster.

Out of my memory, I can only suggest you to check everything that deals with RAM, Hard Drive access, Environmental variables, Limits and Security (like Firewall). Everything that can affect the execution of php script, starting with some Remote Procedue Call policies and ending with the operating stack memory.

The logic is that is php.exe calls some external .dll file to execute some operation, there might be checks on the way done by OS, that will slow both sending request via such .dll, and receiving the response from it. If the .dll uses hard drive to access something - hard drive access policies enter into the scene. Also, how everything is situated in the memory - in RAM or hard-drive cache of RAM. Application policies. Threads policies. Limits on max percentage available for use for applications.

I am not saying that Windows-based hosts are bad, just that they are much more difficult to setup properly for a general admin. If you have Microsoft specialist on hands, he can tune your server to be as fast as Linux-based server.

Answer 2

Use NGINX and FCGI for PHP via UNIX socket (not TCP socket).

http://wiki.nginx.org/PHPFcgiExample

You will notice immediately speed improvements even without accelerators. Additionally, above setup would have much lower memory usage footprint.

Answer 3

• enable APC, when using PHP5.4

  • if you do not notice a speed gain, when APC is on, something is misconfigured

    [APC] extension=php_apc.dll apc.enabled=1 apc.shm_segments=1 apc.shm_size=128M apc.num_files_hint=7000 apc.user_entries_hint=4096 apc.ttl=7200 apc.user_ttl=7200

• enable Zend Opcode when on PHP 5.5

  [Zend] zend_extension=ext/php_zend.dll zend_optimizerplus.enable=1 zend_optimizerplus.use_cwd=1 zend_optimizerplus.validate_timestamp=0 zend_optimizerplus.revalidate_freq=2
zend_optimizerplus.revalidate_path=0 zend_optimizerplus.dups_fix=0 zend_optimizerplus.log_verbosity_level=1 zend_optimizerplus.memory_consumption=128 zend_optimizerplus.interned_strings_buffer=16 zend_optimizerplus.max_accelerated_files=2000 zend_optimizerplus.max_wasted_percentage=25 zend_optimizerplus.consistency_checks=0 zend_optimizerplus.force_restart_timeout=60 zend_optimizerplus.blacklist_filename= zend_optimizerplus.fast_shutdown=0 zend_optimizerplus.optimization_level=0xfffffbbf zend_optimizerplus.enable_slow_optimizations=1 opcache.memory_consumption=128 opcache.interned_strings_buffer=8 opcache.max_accelerated_files=10000 opcache.revalidate_freq=60 opcache.fast_shutdown=1 opcache.enable_cli=1

• disable Wordpress extensions step-wise, to find the memory usage monster

• set Wordpress: define('WP_MEMORY_LIMIT', '128M');, unless you use image converting plugins that should suffice
• set unlimited memory in php.ini ini_set('memory_limit', -1);
• profile without running Xdebug, this sounds crazy, but the debugger itself has a high impact
• use memory_get_usage and spread calls all over the system to find the code position, where the memory leaks
• give zend.enable_gc=1 a try, scripts will be slower, but use less memory
• maybe just disable checking for the user browser in the SlimStats settings..
• if that is not possible, try to override SlimStats getBrowser() function, with a faster getBrowser() substitute
• for a speed comparison of user-agent fetchers, see https://github.com/quentin389/ua-speed-tests
• https://github.com/garetjax/phpbrowscap

Answer 4

To troubleshoot an issue of this kind, you'd have to:

• Use system logs, namely error logs: sorting them by error level
  and/or date to identify dates of the issue.
• Compare Windows updates in use on either system, one of them could be faulty.
• Compare software in use on either system, the setup of one of these could be faulty.

If that's the case, you'll want to uninstall the Windows update and/or the software that causes the issue, shutdown your server completely, then reinstall the update or software (to ensure a stable state during setup).

Tools that can help you troubleshoot this issue include the sysinternals suite: http://technet.microsoft.com/en-us/sysinternals/bb842062.aspx

Or more simply, open source VBS scripts to produce comparable lists of updates and applications on a system.

Answer 5

I took a look at that plugin on Github:

https://github.com/wp-plugins/wp-slimstat

And the offending file being included is a file that has been minified to some degree and really is data (not code), there are 5 variations each of which is about 400KB

There is also that maxmind.dat file that is 400KB, although I don't know if it uses both.

You are using an older version of the plugin, version 3.2.3 and there is a much newer one that may solve your problem.

Comparing the differences is hard because the author or whoever has not kept the git history in order, so I had to manually diff the file. Most of the changes related to _get_browser seem to be adding a cache.

It's possible loading that file is slow to parse, but I would expect PHP to be load both files at similar rates in both platforms granted that IO caching is working.

EDIT Looking a bit closer that might not solve your problem. Those files are basically large Regular Expression lookup tables. Did your Linux system have an APC cache on it and this one does not? The APC cache would probably keep the PHP file data cached (although not the compiled regex patterns)