Title photo
frugal technology, simple living and guerrilla large-appliance repair

Regular blog here, 'microblog' there

Many of my traditional blog post live on this site, but a great majority of my social-style posts can be found on my much-busier microbloging site at updates.passthejoe.net. It's busier because my BlogPoster "microblogging" script generates short, Twitter-style posts from the Linux or Windows (or anywhere you can run Ruby with too many Gems) command line, uploads them to the web server and send them out on my Twitter and Mastodon feeds.

I used to post to this blog via scripts and Unix/Linux utilities (curl and Unison) that helped me mirror the files locally and on the server. Since this site recently moved hosts, none of that is set up. I'm just using SFTP and SSH to write posts and manage the site.

Disqus comments are not live just yet because I'm not sure about what I'm going to do for the domain on this site. I'll probably restore the old domain at first just to have some continuity, but for now I like using the "free" domain from this site's new host, NearlyFreeSpeech.net.

Fri, 22 Nov 2013

Keeping a filesystem in sync across two or more servers and local machines, Part 1

You'd think the solution would be easy and ubiquitous. Here's what I wanted to do: My personal blog run with the Perl-based Ode system. Ode doesn't use a database. Instead it stores its entries as text files in "normal" directories on the server.

I wanted to have exact copies of everything in my Ode documents directory on my local computer and the server. And I wanted the freedom to add to or modify anything in this directory on either side (server or laptop) and have everything track on both machines.

Many of us use Dropbox (or Box, or SpiderOak, or Google Drive, or ...) to both back up some or all of our files and mirror them on other desktops and laptops we happen to use.

But what if you want to keep a filesystem in sync across any number of servers and desktops and laptops without using a third-party service?

My first thought was, "I'll just use Dropbox. Certainly there must be a way to use Dropbox on my server/VPS/shared-hosting. Nope. No. It doesn't work that way.

My second thought was, "Holy shit, Dropbox is missing out on a whole lot of revenue and screwing its users besides."

My third thought was, "There's probably a way to hack this together -- to get Dropbox on the server, but I'm tired of filthy hacks."

My fourth thought was, "There has got to be another way." Sure there are things like FTP/SFTP and rsync, both of which I use on a daily basis for accessing and updating the various servers I use.

That works if you're always "creating" on one machine and hosting that created material on another. You just push with S/FTP or rsync and leave it at that.

But what if you create a file directly on the server, or use another computer to create/modify a file? How do you get that file off the server and onto your main "local" machine? Sure you can remember to download it later, but how do you remember? How many files? Which files? Where are they? Are their image files involved, too?

This will make your head hurt. It's just not doable. This is why things like Dropbox were invented. And why something like this is essential for servers and local client machines.

That the solution wasn't quick, easy, obvious and painless, for the lack of a better word, vexed me.

There has to be a way to do this: an automatic way to keep two or more filesystems in sync if there are changes potentially on both -- or all -- sides.

I searched and asked around. Two projects looked good: csync2 and Unison.

They looked good in that they are meant to do just what I wanted: reconcile the differences between two or more filesystems/directories/folders (whatever you want to call them).

Neither looked easy to set up.

Csync2, on its own home page, positions itself as a server-to-server product and recommended Unison for syncing between a server and a local computer. So that's where I went first.

Next: The road to syncing with Unison.