After posting earlier about performance problems with Dropbox, I got an email from Arash Ferdowsi, the Founder/CTO of the company, saying there is a new version in the final stages of testing. He suggested it might help with some of the problems.

He wrote: "I recommend trying the latest release candidate of dropbox. it's been incubating for the last 9 months and we're planning to release it in the next 2-3 weeks. it's much better at handling large file sets and uses much less cpu/memory. additionally, it's a lot more aggressive about pruning the dropbox cache (the build you're on only clears it when the dropbox process reboots while the new build clears it every 3 days)."

http://forums.dropbox.com/topic.php?id=27419&replies=146

I asked if I could install the software on just some of the machines here, and he said yes, with a caveat. "0.8 now supports syncing of the executable bit and some other xattr data. if you're running a mix of 0.7 and 0.8 clients, this metadata won't sync properly."

I'm installing the new release on the server that's suffering the most, and on one or two other machines. I will of course report the results.

Update: I've tried installing the new software on two machines and both installs have frozen at this point. I'm going to leave the installer running on this machine for a while and see what happens.

Update: A couple of hours later, I was able to get it to install on one of the machines by first uninstalling then re-installing the app. This meant relinking the account, a slow process, but there was an upside -- I stumbled across the new Selective Synch function. This is going to make a big difference. It allows me to push content out through Dropbox that I might only need on one or two other machines, without adding overhead for the others. Nice!

Update: Why does Dropbox download so slowly? 5K/sec. I have no bandwidth constraints in the prefs. See the screen shot.

 11/27/2010; 4:12:11 PM.   .

I've spent the last week wrestling with my servers, dividing up what they do, factoring things so that maybe I can get the performance where it needs to be. It's one of the reasons I was exploring the possibility of using S3 to store my static sites (I gave up, couldn't make it work).

So now things are running in the new configuration, or I should say -- limping along in the new configuration. There is always lots of breakage when I do this kind of surgery. While doing it I wish I could integrate DNS, generating Apache config files and serving into one master outline, where I make a change and the new configuration automatically percolates where it needs to. That, and it would be great to have one server with infinite CPU, disk and net bandwidth, so I'd never have to logically split a server across multiple machines.

Maybe I'll get all this in time for next Christmas!

Anyway...

I think the mistake I made in the last half of 2010 is that I got too Dropbox-happy.

I built too many systems on it, without understanding that it could be brought to its knees, begging for less to do, all the while causing every other application on the server to be starved for CPU cycles.

When I saw how much work Dropbox was doing just to keep current in a relatively stable folder, I realized I needed to lower my dependence on it. It really wasn't doing that much for me, but it was taking up a very large portion of the CPU cycles on every one of my machines. (Only a few small text files were changing every few minutes, the really big stuff came once a night when the servers backed up).

Also it uses a huge amount of disk space. The cache is enormous. I figure it uses at least 1.5 times the size of the folder.

Now, Dropbox is a miracle product. Last year I called it an "idea factory." It gets you thinking about great things could be with a simpler architecture, and closer to the ideal of one big server without boundaries. It's a very easy-to-understand queueing system, and when you add XML or JSON to the mix, you get the magic that comes at the intersection of simplicity and power. But that's the virtuality of it, the reality is that it uses a lot of resources, so you have to use it carefully.

I'm interested in finding out what other people are learning about Dropbox.

 11/27/2010; 4:25:24 AM.   .