Sometimes we’re faced with unexpected challenges, and as system administrators, part of our job is to find and test solutions before an issue begins to have a noticeable impact on one of our services. This post outlines an example of one of those situations, and the approach we took to resolve it.

Problem: As the popularity of Master FTP increased, the strain it was placing on our database servers also grew.

Solution: We’d heard that Facebook made use of Memcached to ease load on their database servers, so we decided to give it a try. Happily, after adding a very simple layer of caching to parts of Master FTP (so that operations like adding new packages to the master work immediately), we’ve gained a 50% speed increase on Master FTP performance.

Because Master FTP is a virtual FUSE filesystem, it needs to query our database servers for file information for many filesystem operations (stat, open, close, read, write, seek, etc). As the popularity of Master FTP grew, the amount of queries it was making to our database servers also increased, which began to impact performance on the read/write/directory listing speeds of Master FTP.

A lot of this information is static, or at the very least, infrequently changing. We had the idea that we would add a very thin layer of Memcached between Master FTP and the database servers in the Master FTP FUSE software.

The implementation was very simple:

(a) Compose SQL query

(b) Query Memcached for an answer

(c1) Answer with Memcached’s answer if it has one

(c2) Query the database if not

(d) Cache that answer in Memcached (we chose a cache time of 30 minutes).

We haven’t cached *every* query in Memcached, as it wouldn’t necessarily speed things up but could affect Master FTP negatively: the best example of this being not to cache the list of sites that are inside a Master FTP account (this question is asked quite infrequently, but by not caching it, when you add another package to Master FTP, you don’t need to wait for the cache expiry time to pass before it will work).

If you have any questions, please pose them in the comments section below.

And in the words of the Looney Tunes: That’s all folks!