Master FTP and Memcached | Heart Internet Blog – Focusing on all aspects of the web

Sometimes we’re faced with unexpected challenges, and as system administrators, part of our job is to find and test solutions before an issue begins to have a noticeable impact on one of our services. This post outlines an example of one of those situations, and the approach we took to resolve it.

Problem: As the popularity of Master FTP increased, the strain it was placing on our database servers also grew.

Solution: We’d heard that Facebook made use of Memcached to ease load on their database servers, so we decided to give it a try. Happily, after adding a very simple layer of caching to parts of Master FTP (so that operations like adding new packages to the master work immediately), we’ve gained a 50% speed increase on Master FTP performance.

memcached

Because Master FTP is a virtual FUSE filesystem, it needs to query our database servers for file information for many filesystem operations (stat, open, close, read, write, seek, etc). As the popularity of Master FTP grew, the amount of queries it was making to our database servers also increased, which began to impact performance on the read/write/directory listing speeds of Master FTP.

A lot of this information is static, or at the very least, infrequently changing. We had the idea that we would add a very thin layer of Memcached between Master FTP and the database servers in the Master FTP FUSE software.

The implementation was very simple:

(a) Compose SQL query

(b) Query Memcached for an answer

(c1) Answer with Memcached’s answer if it has one

(c2) Query the database if not

(d) Cache that answer in Memcached (we chose a cache time of 30 minutes).

We haven’t cached *every* query in Memcached, as it wouldn’t necessarily speed things up but could affect Master FTP negatively: the best example of this being not to cache the list of sites that are inside a Master FTP account (this question is asked quite infrequently, but by not caching it, when you add another package to Master FTP, you don’t need to wait for the cache expiry time to pass before it will work).

If you have any questions, please pose them in the comments section below.

And in the words of the Looney Tunes: That’s all folks!

Comments

Please remember that all comments are moderated and any links you paste in your comment will remain as plain text. If your comment looks like spam it will be deleted. We're looking forward to answering your questions and hearing your comments and opinions!

Leave a reply

  • Dean Wall

    14/02/2013

    This is great news, I often found Master FTP considerably slower than using individual FTPs. It’s a great tool for me as a developer having every site I work on in one place but I often found myself configuring individual FTPs as it was so slow! Will test it out!

     

Comments are closed.

Drop us a line 0330 660 0255 or email sales@heartinternet.uk