What I learned about fs.writeFile today
by Dave Winer Tuesday, September 27, 2016

Thanks to Frank Leahy, I believe I now understand what's causing the persistent but intermittent Node.js fs.writeFile problem, discussed earlier.

Here's the story...

fs.writeFile is not an indivisible operation. It makes lower-level filesystem calls each of which is asynchronous. That's cool, usually -- unless for some reason that a request to write the same file comes in while it already has one outstanding. Then havoc can ensue as explained in this note on a Node.js developer site.

appPrefs.json is probably the most written-to file in an app built on nodeStorage. It's where the state of the user's system is stored. So every keystroke when editing, if the app saves your writing automatically, could result in a write to appPrefs.json. But here's an important constraint, the write can't occur more often than once a second. So if you type three keystrokes in a second, that will just generate one write, not three. And in most apps it's not even that frequent, usually there's a max of three seconds. But in some apps there is no max. So once a second writes will happen.

But add to that the time it takes for the message to be sent to the server, and intuitively it seems that the two writes can't interfere with each other. 

But..

What if the machine is busy. What if there are a lot of users all writing at the same time. What if who knows what else determines the time it takes for a write to complete.

And if one write should happen to be triggered while the previous one hasn't completed, the file could get garbled. It says so in that note, and it explains why, convincingly.

And...

The file doesn't get screwed up very often. It happens rarely and only seems to happen with appPrefs.json, the most often-written file. 

So...

Here's the change I made.

When I start to write a file, I add the path to the file in a JavaScript object named filesCurrentlyBeingWritten. When the write completes I remove the path. And before I start the write I check to see if the file is already being written, and if so, I simply call it an error. The important thing is I don't begin a second write. 

Think of this as a Poor Man's Semaphore. Since there is no way to tell a thread to yield in Node (as far as I know) a full semaphore system isn't possible. But the tradeoff seems reasonable.

This way the user might not get his or her appPrefs.json file updated this one time, but it won't get corrupted. No havoc. It seems unlikely anyone would even notice, because it's probably just text being saved against a different (also unlikely) app failure. In any case, losing a little data is vastly preferable to losing it all. ;-)

I'm have checked these changes into the nodeStorage repo and they will make it out to all servers that have auto-updating turned on. 

PS: The new version is running on this blog, and so far it seems to work. Knock wood. Praise Murphy!