Monday, March 30, 2009

How I would like to use Propel and Memcache

In this article I will like to share some ideas that are wandering in my mind but I haven't implemented yet. So beware!

I was thinking of a way to reduce database load by caching some results inside Memcache -idea which has nothing special or revolutionary this days-. This article gave me some ideas that I would like to see implemented in some of the projects I work for.

The code is for using mostly inside a Symfony/Propel project, but could be adapted to a different one with ease.

Propel Peer classes come packed with the following method.

BaseUserPeer::retrieveByPK($pk, $con=null);

I would like to override it in the following way:

public static function retrieveByPK($pk, $con = null)
  $cacheKey = sprintf('user:id:%d', $pk);
  $asArray = $memcache->get($cacheKey);
  if($asArray === null)
    $obj = parent::retrieveByPK($pk, $con);
    if($obj !== null)
      $memcache->set($cacheKey, $obj->toArray(BasePeer::TYPE_FIELDNAME));
    $obj = new User();
    $obj->fromArray($asArray, BasePeer::TYPE_FIELDNAME);
  return $obj;

Note that I've avoided the memcache connection code. I assume that there is a memcache class that abstracts the process. I'm also avoided the details of this class instantiation.

As you can see from the code, I store the object in memcache as an associative array. I like to do so because I don't want to store a serialized version of the Propel object which will take more space. Also by using a native data type I can warm up the cache from -let's say- a batch script without the need of using Propel at all. Also, if a I have code that don't require symfony or Propel, I can still use the cached data. 

The fromArray and toArray methods are built inside propel objects as a convenient way of populating them, so there's no extra effort in our side to get their benefits. 

As key for the cache I'm using as prefix the name of the table, followed by a colon and then the primary key column name, a colon and the primary value. i. e.: table_name:primary_key:value

Then what is left for this to actually work is to override the User::save($con = null) method, so every time the database row is updated the changes will be reflected in the cache.

I came up with the following code:

public function save($con = null)
  $affectedRows = parent::save();
  $memcache->set(sprintf('user:id:%d', $this->getId()), $this->toArray(BasePeer::TYPE_FIELDNAME));
  return $affectedRows;

There by updating the entry after it's saved to the database I used a traditional pattern with memcache to warm up the cache.

So lets say that in a normal login form, the user will submit his nickname and password to be checked against the database. In this case we have to do a SELECT query
using the nickname and password as WHERE parameters. Instead of issuing a query, I would like to do the following inside UserPeer::retrieveByNickname($nickname).

public static function retrieveByNickname($nickname)
  $nicknameCacheKey = sprintf('user:nickname:%s', $nickname);
  $userId = $memcache->get($nicknameCacheKey);
  if($userId !== null)
    return UserPeer::retrieveByPK($userId);
    $c = new Criteria();
    $c->add(UserPeer::NICKNAME, $nickname);
    $user = UserPeer::doSelectOne($c);
    if($user !== null)
      $memcache->set(sprintf('user:id:%d', $user->getId()), $user->toArray(BasePeer::TYPE_FIELDNAME));
      $memcache->set($nicknameCacheKey, $user->getId());
    return $user;

In the last example first I check if there is a memcache entry with the following key: user:nickname:somenickname. The value stored will be the user id which I assume
is the primary key of the table. If the user id is not null then I delegate the call to UserPeer::retrieveByPK to do the job. In the other case I fetch the user from
the database using the nickname as Criteria condition. If the record exists I store inside memcache the user data as an array and also I store the id using the
user:nickname:somenickname key. Now should be clear why in the previous example I used used:id:somenid as key.

Some improvements to do in the retrieveByPK method and in the save method will be to also store the respective values for the user:nickname:somenickname key once
the object has been populated. In this way we increase the chances that retrieveByNickname will successfully hit the memcache.

I hope this article results useful for you and thanks for reading.

NOTE: I know that the code is pretty ugly, some code needs to be refactored out to it's own methods and maybe Propel classes are not the best place for
this caching logic to reside, but I think is a nice example to build upon.


francois said...

DbFinderPlugin implements a query cache with Propel - and supports not only Memcache, but also many other caching engines for which symfony provides an adapter. Check it out!

Alvaro Videla said...

@francois, nice, I will take a look at the plugin

Derek said...

I implemented this using Propel and APC (another cache). I did it by creating my own ObjectBuilder and PeerBuilder classes which extend Propel's built-in ones. This technique is documented, and lets you tell propel to Propel generate classes with your methods built-in to the standard methods. For example, you can override save() for all objects, so it uses your caching layer.

In the end, after I got it totally implemented and working, I found that it did not offer me the flexibility I needed, so I switched back to just doing the caching in the controllers instead of the model itself.

Anyway, if you want help setting it up, just let me know.

Alvaro Videla said...

@Derek, thanks for the comment. I've done some extensions to the Propel builders and also to the BaseObject class, etc.

But as you say I think this code should be moved to another layer and leave the Propel classes as they are.