CodeIgniter, NFS mounts, and Logging
I recently ran into a problem with a site I wrote in CodeIgniter, and I didn’t see much that mentioned it, so I thought I’d post about it in the hopes that if it came up for someone else, maybe I could be of some assistance.
When developing the site locally, everything was fine. On a testing server, it was fine at first, but as more people started testing on it, we found that now and then, the site would simply stop responding. The browser would just stay at the “waiting for server xyz.edu”, apparently indefinitely. We have our subversion server set up to automatically deploy the HEAD tag to the test server, and there’s a web interface to manually deploy a given revision (defaulting to HEAD) whenever. The odd thing was that this hanging problem seemed to be cured (usually, but now always) by pushing the current version out to the server. Sometimes it only cured it for a few seconds or minutes, but often, it seemed fine for hours or days. As time when on, though, it came up more and more often. Oddly, it was only with this site that the problem showed itself. While the site as unresponsive, other sites on the same server seemed fine.
At first we thought the server was just getting bogged down with some process that was using up CPU cycles, or network connections, something like that. Watching htop for a while did show a coworker’s python script coming up, but even that didn’t seem nearly severe enough to be the cause. We also had a bit of trouble with the database server while this was going on, so that was also suspect, though I would be able to connect while the site was unresponsive, so that didn’t seem likely either.
With some help from a coworker, we eventually did come up with the cause, at least in part. We did a list of all files in the site’s directory before the problem happened, and then refreshed the site rapidly using a highly technical method (we hit F5 and held it down until the site died. Then we redeployed from svn, and did another file list, then diff’ed the two. The only difference was CodeIgniter’s own log file that it generates in system/logs by default. After the deployment, the log was cleared. We also noticed that while the site was unresponsive, the log directory had what appeared to be a temporary dot file left by NFS. There was our clue – the directory that hosted sites were stored was in fact an NFS mount.
I’m familiar in brief with what NFS is and what it’s for, and so on, but by no means am I an expert or even a frequent user. It appeared that in this case, though, that maybe the mount wasn’t expected for so much writing as a web log (a fairly heavily verbose one at that, as we we still actively developing the site). It makes sense to me at least – typically you don’t log there, you log in /var/log or something, and the site itself is just a bunch of files you only read from. I wouldn’t be surprised at all if the NFS mount was, or was configured to be, set up for a lot of reads and very few writes, and when the site hammered it with file writes, it slowed to a crawl, as it turned out that if you tailed the log while it was unresponsive, lines were still written to the log, just very slowly. As in, several seconds in between lines indicating nothing more than a PHP class being instantiated.
So, regardless of the gritty technical details of the problem that I haven’t had time to investigate yet, the problem was then easily solved – I just turned off logging altogether, and the problem hasn’t surfaced since.
The lesson learned, for me at least, is to be mindful of where you’re writing.