The purpose of this post is not to provide a bunch of cache backend benchmarks, but rather to simply introduce my benchmark code in the hopes that others conduct their own tests and hopefully publish their findings. A link to this post is appreciated. Also, if there are any criticisms of the benchmark I’d love to see a pull request. :)

The benchmark suite is fully-featured:

• Repeatable tests. Dataset is written to static files so the exact same test can be repeated, even with entirely different backends.
• Test datasets can easily be zipped up and copied to different environments or shared for others to use.
• Can relatively easily test multiple pre-generated datasets to compare different scenarios on the same hardware.
• Uses true multi-process benchmarking, each process with a different set of random operations.
• Flexible dataset generation via options to init command. Cache record data size, number of tags, expiration, popularity and volatility are all randomized.

Currently the benchmarks are run via the command line so testing the APC backend or any others that only work via a cgi or apache module environment will not work. This could be remedied easily enough with the use of CuRL and some php copy/paste if you had the desire to test on your actual web server.

Here is an example run using the Redis backend using my dev environment, a Lubuntu VirtualBox guest:

Cache Backend: Zend_Cache_Backend_Redis
Loading 'default' test data...
Loaded 10000 cache records in 29.1080 seconds. Data size is 5008.9K
Analyzing current cache contents...
Counted 10023 cache IDs and 2005 cache tags in 0.2062 seconds
Benchmarking getIdsMatchingTags...
Average: 0.00036 seconds (36.82 ids per tag)
Benchmarking 4 concurrent clients, each with 100000 operations...
4 concurrent clients completed in 62 seconds

         |   reads|  writes|  cleans
------------------------------------
Client  1| 1811.83|  184.66|    6.81
Client  2| 1799.84|  165.29|    6.91
Client  3| 1818.90|  165.17|    6.79
Client  0| 1790.91|  153.56|    7.40
------------------------------------
ops/sec  | 7221.48|  668.68|   27.91

The important numbers to look at are the summed ops/sec. Given the three variables: dataset, hardware and backend, it is easy to change just one of these without affecting the others so this benchmark can be used to test any one of the three variables reliably. The three metrics observed are reads, writes and cleans. The first two are pretty self-explanatory. The third is a clean operation on a single tag using Zend_Cache::CLEANING_MODE_MATCHING_ANY_TAG which is the only mode Magento ever uses other than Zend_Cache::CLEANING_MODE_ALL for manual cache refreshes. Individual read/write operations are very fast so given the large number of operations in a test I did not feel the need to examine min, max, average, or standard deviations.

The test uses (hopefully) sane defaults for dataset generation parameters, but there is plenty of flexibility. I advise you to examine your production environment (number of cache keys, number of cache tags, number of concurrent clients) to tweak the test to more closely match your own environment. Here is the output of the --help cli parameter:

$ php shell/cache-benchmark.php --help
This script will either initialize a new benchmark dataset or run a benchmark.

Usage:  php -f shell/cache-benchmark.php [command] [options]

Commands:
  init [options]        Initialize a new dataset.
  load --name <string>  Load an existing dataset.
  clean                 Flush the cache backend.
  tags                  Benchmark getIdsMatchingTags method.
  ops [options]         Execute a pre-generated set of operations on the existing cache.

'init' options:
  --name <string>       A unique name for this dataset (default to "default")
  --keys <num>          Number of cache keys (default to 10000)
  --tags <num>          Number of cache tags (default to 2000)
  --min-tags <num>      The min number of tags to use for each record (default 0)
  --max-tags <num>      The max number of tags to use for each record (default 15)
  --min-rec-size <num>  The smallest size for a record (default 1)
  --max-rec-size <num>  The largest size for a record (default 1024)
  --clients <num>       The number of clients for multi-threaded testing (defaults to 4)
  --seed <num>          The random number generator seed (default random)

'ops' options:
  --name <string>       The dataset to use (from the --name option from init command)
  --client <num>        Client number (0-n where n is --clients option from init command)
  -q|--quiet            Be less verbose.

To handle multi-process benchmarking the test is actually launched from a shell script which backgrounds each client and sums the results using awk so unless you are doing single-process benchmarks you never need to invoke the ‘ops’ command yourself.

Give me the code already!

The code is hosted at github.com/colinmollenhour/magento-cache-benchmark. If you use modman you can install it like so:

modman cachebench clone git://github.com/colinmollenhour/magento-cache-benchmark.git

Or, you may also download it directly and just extract cache-benchmark.php to the “shell” folder in your Magento installation.

Run a test!

Assuming you’ve cloned/downloaded the code already, here is how you run your first test:

php shell/cache-benchmark.php init
bash var/cachebench/default/run.sh

Could it get any easier?

PS. I included a “Null” backend which is just a black hole for the purpose of getting a general idea of your PHP overhead.

$productIds = array_keys($_products->getItems());
$minimalPriceModel = Mage::getResourceModel('catalogindex/price');
$minimalPriceModel->setStoreId(Mage::app()->getStore()->getId());
$minimalPriceModel->setCustomerGroupId(
  Mage::getSingleton('customer/session')->getCustomerGroupId());
$minimalPrices = $minimalPriceModel->getMinimalPrices($productIds);
foreach ($minimalPrices as $row) {
  $item = $_products->getItemById($row['entity_id']);
  if ($item) {
    $item->setData('minimal_price', $row['value']);
    $item->setData('minimal_tax_class_id', $row['tax_class_id']);
  }
}

There you have it!

This module has been replaced with one that correctly updates the layered navigation counts. See the new version here.

Updated 7/3/09

The in-stock-only module now works properly when multiple websites are in use.

Updated 6/26/09

The in-stock-only module now works with configurable products.

Magento has several criteria for determining if an item should be visible in the store, but one of these is not whether or not the item is in stock. Magento can manage your stock and will set your stock status to “Out of Stock” for you, but what if you want these out of stock items to be hidden from the store.. automatically? Also, in my case I do not want to “disable” the item because the inventory management system will then ignore it. Turns out this was much more difficult to figure out than I expected it to be but the end solution was quite simple.

To hide all out of stock items from Magento’s storefront we are going to create a local module and call it “Lucky_InStockOnly”. This will be a very simple module, so get ready to copy and paste.

Every module needs a config file. Ours is very basic, here it is:

app/code/local/Lucky/InStockOnly/etc/config.xml

<?xml version="1.0"?>
<config>

<global>
  <models>
    <lucky_instockonly>
      <class>Lucky_InStockOnly_Model</class>
    </lucky_instockonly>
  </models>
</global>

<frontend>
  <events>
    <catalog_block_product_list_collection>
      <observers>
        <lucky_instockonly_list>
          <type>singleton</type>
          <class>lucky_instockonly/observer</class>
          <method>addInStockOnlyFilter</method>
        </lucky_instockonly_list>
      </observers>
    </catalog_block_product_list_collection>
  </events>
</frontend>

</config>

As you can see, we declare an event for the frontend that observes the “catalog_block_product_list_collection” event which is dispatched in Mage_Catalog_Block_Product_List in the _beforeToHtml() method. This _beforeToHtml method is called before the corresponding template code is loaded, and the next line after the dispatchEvent loads the collection at which point it is too late to add any further filters, so we’re just in time.

Next, we add the observer code:

app/code/local/Lucky/InStockOnly/Model/Observer.php

<?php

class Lucky_InStockOnly_Model_Observer {

  /**
   * Observes the catalog_block_product_list_collection event
   */
  public function addInStockOnlyFilter($observer){
    $observer->getEvent()->getCollection()
      ->joinField('stock_status','cataloginventory/stock_status','stock_status',
        'product_id=entity_id', array(
          'stock_status' => Mage_CatalogInventory_Model_Stock_Status::STATUS_IN_STOCK,
          'website_id' => Mage::app()->getWebsite()->getWebsiteId(),
        ))
    ;
  }
}

That’s it! That’s it? Like all things Magento, easy for a veteran, not so easy for anyone else; hence, this blog post. All that is left is to enable our module and we’re off!

app/etc/modules/Lucky_InStockOnly.xml

<config>
  <modules>
    <Lucky_InStockOnly>
      <active>true</active>
      <codePool>local</codePool>
    </Lucky_InStockOnly>
  </modules>
</config>

Last thing, you need to refresh your configuration cache so Magento will notice the new module. In the admin backend you can do that under System > Cache Management.

Enjoy!

While modules in Kohana can (and often do) have their own controllers and views, they sit below the application layer in terms of cascading priority, whereas the modules in Zend Framework sit above the main application layer. That is, if a module is specified in the route, classes will be searched for first in the module directory, and then in the default directory. I really like this directory structure and want to apply it to Kohana. Because of the naming conflict, I am calling these sub-application directories components instead of modules. Here is an example of what I am shooting for:

/application
   /cache
   /config
   /controllers
      /template.php (Template_Controller)
      /welcome.php  (Welcome_Controller)
   /libraries
   /models
   /components
      /admin
         /controllers
            /welcome.php (Welcome_Controller)
            /users.php (Users_Controller)
            /posts.php (Posts_Controller)
         /views
            /template.php (overrides /application/views/template.php)
      /blog
         /controllers
            /posts.php (Posts_Controller)
         /views
            /posts.php (from within blog component: new View('posts'))
      /rest
         ...
      /mobile
         ...
      ...
/modules (Traditional Kohana Modules)
   /auth
   /curl
   ...
/system (Kohana Core)

I use the router to determine the current component, implemented inside my branch of the router3 module. The default value can be set for ‘component’ in a route, or the component can be discovered from the uri, or it can fall back to the default component, which by default is the application itself. The default component can also be given a name such as ‘default’ in the config file. To avoid conflicts I have an array in my config file to define the components that are enabled and the router will only attempt to match those components via a regex built into the route.

Hacking the find_file system is not particularly easy, especially when you enable caching because the component is chosen on-the-fly so the results for (‘controller’,'welcome’) will be different from one request to the next. In order to support cascading of all resources in the component directories, the core would have to be hacked. However, I have successfully implemented this as a module which only supports “controllers” and “views” in the component’s directory. Caching of find_file paths is not broken, and components that are not active for the current request are not in the cascading order at all which I believe to be preferable. Any controllers and views that should be shared among all components in the application can go in the application directories, all others can be separated into component directories.

So what does this accomplish? This let’s you easily keep components of your application organized in different directories. I currently do not include the component name in the controller class like Zend does, so your controller class and file names and view paths can be kept short. E.g., for the typical blog example:

/application
/controllers
   /welcome.php (Welcome_Controller for blog frontend)
/views
   /template.php (template for blog frontend)
   /posts.php (render frontend view of blog posts)
/components
   /admin (URLs starting with admin will use this component)
      /controllers
         /welcome.php (Welcome_Controller for admin backend)
      /views
         /template.php (template for admin backend)
         /posts.php (render admin view of blog posts)

Aside from the more compartmentalized directory structure, the template name does not have to be changed for the component’s template to be used, it just works! Also, when instantiating a view, there is no need to specify sub-directories.

// New:
$this->template->content = new View('posts');
// Old:
public $template = 'admin/template';
$this->template->content = new View('admin/posts');

From a code-management standpoint, this could also have some benefits. For example, do you want to revert the admin component back to an older revision but leave all other components alone? Since the admin components files are all contained under one directory, rather than mixed in with every other component this is a simple task.

Update:

The powers that be cannot be convinced that such a directory structure is worthwhile, or that the Kohana class does not have to be “final” so I’ve unfortunately decided to give up and fork Kohana and track it’s progress.. So much for, as the Kohana homepage puts it, “Loosely coupled architecture” and “Extremely easy to extend”… If anyone is interested in a patch that seamlessly adds this feature, let me know!

Update 2 (4/21/09):

It turns out that the Kohana devs have agreed to make Kohana no longer “final” for 2.4, and in fact have moved the Bootstrap.php file into the application directory as well; Icing on the cake! See r4231