Python extension for libmemcache

Contents

Retired

cmemcache is retired as of 2009/12/21 due to lack of maintenance and more viable replacements. You can find the alternatives on the memcached website.

I have tried python-libmemcached briefly in my cache compare benchmark and it scores a little bit better than cmemcache on most operations, and on the rndmulti 10% faster. However, I have not run it in production so I can only endorse it in the sense that it is fast and in active development. If somebody has experience with python-libmemcached handling of faulty and changing memcached servers, cmemcache often crashed in these situations, I would very much like to hear about it.

Many thanks for using and contributing to cmemcache!

Description

cmemcache is about 1.7 times faster than python-memcache with short key names (8 characters), faster with larger key names (I get about 2x for 100 character keys). Using get_multi is faster still, almost 2x for 2 8-character keys. See cachecmp.py for profiling logic.

Motivation

Keep the espresso's flowing:

This extension was created after doing some timings on a simple session caching scheme where I noticed that python-memcache was 'only' three times as fast as PostgreSQL. I expected it to be faster than that, since PostgreSQL does a lot more than memcached. It was suggested that it was perhaps python-memcache, which for instance gets the initial part of the message from memcached byte for byte. After discovering libmemcache it seemed pretty straight forward to create an extension on top of that.

cmemcache has 2 clients: StringClient and Client. StringClient is the extension that only supports python strings for values. The api was copied from python-memcache, except that it excepts strings only. Client is a module that implements caching of arbitrary python objects using Pickle on top of the StringClient. This code is a copy/paste from python-memcache. Most of the test code in test.py is run on the cmemcache Client and the python-memcache Client to make sure that they are interchangeable. Although I have not tested this but it should be possible to mix cmemcache and python-memcache clients in a running system as well, since they use the same constants for encoding object types.

The speed difference would be less if the memcached protocol would be changed to precede the reply header with its size. A client could then read the header size, read the full header (in one read), parse header to get the data size, and read data (this is already done in one read).

Dependencies

Download

To do

Known Bugs

cmemcache:

libmemcache:

Changes

Versions:

0.96

Change Client._set() str(ing) logic to pass unicode strings through the pickle code. Found and patched by Armin Ronacher. Fixed Client() object memory leak, found and patched by Dan Helfman. Updated some docs, fixed build on MacOS.

0.95

Fixed expire time internal type (was int, must be time_t). To be on the save side the expire time is also clamped to 2**31-1 to follow the memcache protocol spec. Old code caused problems when using sys.maxint on 64-bit machines. Reported and patched by Simon Law.
0.94

Added missing debuglog() implementation, copy/paste error from memcache.py. get() now returns None on all error cases. Reported by Simon Law.

Fixed some memory leaks in get_multi().

Fixed get_multi() return values for Client. All results would be returned as strings, instead of the proper types. Reported and fixed by Alfred J Fazio.

cmemcache.py uses debuglog() (memcache.py remnant) but debuglog() is not even defined, doh! Reported by Simon Law. His patch uses python module logging, but to avoid dependencies I added the same stderr logger as used in memcache.py. One can override the cmemcache.log variable to install another log function.

0.93

Fixed memory leak caused by not Py_DECREF of key and val objects when calling PyDict_SetItem(). Reported and fixed by Alfred J Fazio.

Allocated my own context and moved from 'mc_*' api to 'mcm_*' api to use. Install error handler to change the 'cont' state of the memcache_err_ctxt object to 'y' to try not to abort on fatal errors. This is an attempt to not crash python on fatal errors from libmemcache, but libmemcache needs considerable changes to make it work. I am not too sure about my libmemcache so I have not released them yet. Code is compatible with libmemcache-1.4.0 though.

0.92
Changed return values for set, add, and replace to be the same as memcache.py, ie nonzero on success. Reported by Marek Majkowski.
0.91
Remove @staticmethod from _convert method to make it python 2.3 compatible.
0.90
Initial version.

Author

Gijsbert de Haan <gijsbert.de.haan@gmail.com>