Box Backup

An open source, completely automatic on-line backup system for UNIX.

Project status: Stable, but not feature complete

More information: what it is, and how it works.

Comparisons to other backup systems

Distributed under a BSD license. Please read the license and commentary now.

Development team

Main developers for 0.11: Martin Ebourne, Charles Lecklider, Gary Niemcewicz, James O'Gorman, Ben Summers, Chris Wilson

Main developers for 0.10: Martin Ebourne, Nick Knight, Jonathan Morton, Gary Niemcewicz, Ben Summers, Chris Wilson

Pre-0.10: Ben Summers, ben@fluffy.co.uk

Project status

This project should be considered as stable, but not feature complete. Over 3 years of testing suggests it will be reliable enough to will back up your data, and make it available for restoration.

It has always been possible to retrieve old versions of files. Planned features will implement marked versions of the store to emulute backing up onto sets of tapes in a controlled manner. This will make restoration of old versions much easier, although in the case of retrieving a single old file will not make much difference to the usability as the FTP-like utility is already effective.

Given that this is a backup system, we are being very conservative about when we consider it ready for production use. That said, we trust our data to it! However, as this is quite a young system compared to some of the alternatives, and operates in a slightly different manner, for production use I must recommend that you regularly verify your backups.

The documentation is not yet complete, and the error messages you get when things go wrong are not as clear as they should be. (see the troubleshooting page.) Please follow the instructions carefully and read everything to avoid problems -- following the instructions on this web site will result in a working installation.

Why online backup?

Because tape is not a reliable solution, unless you...

which is hardly likely to happen in real life.

Backing up to media such as CDROMs is not really a solution either. It needs to be something which just happens without user intervention, and without media which is liable to be corrupted.

So, for the same price as a decent tape backup system, you could buy a server with three big hard discs. And for the money you spend on tapes per year (you do replace them regularly, don't you?), you could host it in a nice data centre somewhere.

We all have internet connections these days, so connectivity isn't really a problem. Although it might be best to archive those static files, like MP3s and images, to CDROM or something.

The costs are even less if you pool resources amongst a group of friends. Remember, you don't have to trust the server admin to not look at your files as they're encrypted. You just have to trust them to make the files available when you need them.

Mailing list

Please join the project mailing list, boxbackup@fluffy.co.uk, for announcements of new versions and discussion of the system. Join at the sign up page.

Many thanks to Alaric for hosting this list.

Implementation

There are three main elements

Running the store server is a multi-step process, but the backup client is easy.

TLS (SSL revised) is used to encrypt connections, and more importantly, to authenticate servers and clients with both server and client side certificates. Scripts are provided to generate and manage these certificates.

Stored files are encrypted using AES for file data and Blowfish for metadata. This does mean that the one thing you do need to back up off-site and look after is a 1k file containing your keys -- the data on the server is useless without it. But it never changes, so that's OK.

Platforms

We develop and deploy on Linux, Darwin, Windows and OpenBSD. The software is designed to take advantage of BSD features where useful, but not to rely on them. It should be easily portable to other UNIX like operating systems. Ports are available for the following platforms:

On these platforms, the software compiles and the tests pass. Ben does all his deployment on OpenBSD, Chris on Linux and Windows, but we know of several people who have had good results under other OSes.

More platforms will arrive over time. It takes in general a couple of hours to get it compiling on a BSD-like UNIX or Linux, and a bit longer on something like Solaris.

We welcome patches for other platforms, and volunteers to look after ports for a particular platform. Programmers notes describing how everything works are included in the notes directory within the distribution archive.

Related projects

There are a number of separate projects which are related to Box Backup. These include:

Boxi -- a cross platform GUI front end for Box Backup.

Download

The latest stable release is 0.10. We also have a release candidate for 0.11 available, which fixes some important issues, but may be less stable and less well tested than 0.10. If you don't mind the extra risk, we'd really appreciate your help in testing it. Please read the Betas and Release Candidates and Upgrading to Box Backup 0.11 pages on the Wiki.

See our Wiki for the latest compilation and installation instructions.

Code Signatures

New releases are signed by Chris Wilson (key ID D847F866). Download the key or fetch with:

gpg --recv-keys --keyserver pgp.mit.edu D847F866

Upgrading

A 0.10 or later server requires all clients to be version 0.10 or later too. Existing backup store accounts are compatible with 0.10, and do not require conversion.

Changes

0.11rc2 (29 January 2008)
Fix build and tests on Solaris. Silence const char warnings.
Fix raidfile tests on FreeBSD 7, thanks to James O'Gorman and Mikael Syska.
Don't warn about mounted directories which are explicitly excluded, reported by Matt Brown.
Added Solaris SMF framework scripts by Ben Summers.
Fully support configuring with a prefix, sysconfdir and localstatedir, building them into all the relevant files, to help packagers and porters.
Fix harmless error message in install scripts, reported by Guno Heitman.
Fix support for debian buildds, thanks to Reinhard Tartler for providing access to them.
Reformatted usage messages from bbackupd-config, bbstored-config and raidfile-config scripts for readability.
Update version and contact email address in configure.ac.
Update copyright year to 2008.
0.11rc1 (12 January 2008)
Fixed some bugs with backing up, restoring and comparing files over 2GB in compressed size.
Added new logging infrastructure, allows more control over whether messages are sent to the console or system logs, and at what level of detail.
Changed keepalive and diff timers to run in real time, not CPU time.
Enable KeepAlive time by default on new installations, set to 120 seconds.
Added bbackupctl commands for improved scripting of syncs.
Fixed a bug with restoring symlinks to directories outside of the backed-up location (thanks to Hans-Joachim Baader)
Ported unit tests for Windows.
Added full unit tests for keep-alives and diff timer on most platforms.
Fixed a number of bugs in the Windows port.
Added option to send Extended Logs to a file instead of to system logs.
Added option to log all file access, for debugging when a file is not backed up or causes the backup to fail mysteriously.
Improved error messages to identify the causes of some errors which were difficult to track down before.
Added bbackupd option to set the length of time before unused locations are deleted.
Changed default location of bbackupd.conf on Windows to the same directory as bbackupd.exe.
Fixed a bug where bbstoreaccounts could modify an account while it was locked by a running backup.
Improved command-line option handling.
Added command-line help (-h option) to bbackupd and bbstored.
Add a new -F option for daemons, which runs in the foreground but still accepts multiple connections, which is what SINGLEPROCESS used to do.
Fixed compare of timestamps on filesystems which cannot set them more accurately than 1 second.
Added new backup-start and backup-finish events to the NotifyScript, which can be used to implement more advanced functionality such as snapshotting databases.
Added a new sample NotifySysAdmin script for Windows, written by James O'Gorman in VBscript.
Added support for multiple Box Backup (bbackupd) services on Windows, with different service names and named pipe names, to implement redundancy.
Fixed bbackupd mysteriously failing to back up if one of the location paths did not exist.
Fixed entering of international characters into bbackupquery on Windows (instructions) and Unixes with editline.
Improve Makefiles by reducing verbosity during build, so that any errors and warnings can be seen more easily.
Added saving of the list of unused root directory entries to the StoreObjectInfoFile, so that they will persist across restarts of bbackup (thanks to Gary Niemcewicz).
Updated built-in documentation (program manuals, installation guide and administrator's guide).
Improved build targets (thanks to James O'Gorman).
0.10 (23 February 2006)
Optimised diffing algorithm -- massive speed improvements
Improved diff timeout
Bug fixes
Solaris support
Portability improvements, including new autoconf based build
xattr support (on Linux and Darwin -- backs up resource forks under Mac OS X)
Readline usage clarified for licensing.
Experimental: Keep-alive on long diffing operations
Experimental: Save state for non-server use, speeds up first sync after bbackupd starts (not enabled by default)
Experimental: Native win32 port integrated into main source tree, fixes and improved build
Experimental: Support for MinGW and MSVC under Win32
NOTE Features are marked as experimental if no automated test exists. However, the developers have been running them on live systems. Win32 support is not totally complete.
0.09 (06 December 2004)
Maintenance release
Bug fixes and reliability changes
Add support for large files under Linux
Compile on new platforms
Solaris and Win32 ports for testing
0.08 (23 September 2004)
Minor bug fixes
Server stores old versions of files as patches from the current version
Adjust behaviour of client
Works under 64bit systems
Client recovers cleanly from corruption of its working files
Improvements to build system
Add contributed code for building RPMs
0.07 (28 June 2004)
Minor bug fixes
Improvements to build process, including better detection of environment on Linux, especially SuSE 9.x
Fix wrong space used sizes on store with non-RAID (see notes)
Be more paranoid about crypto startup
Extra checking for file modifications
SyncAllowScript to allow temporary disabling of syncs, eg only backup from a laptop on certain network connections
Add force-sync command to bbackupctl, like sync but ignores AllowSyncScript
0.06 (21 May 2004)
Bug fixes
Use AES for file data
Cygwin port of client
Add usage command to bbackupquery
Add delete [yes] command to bbstoreaccounts
Add check [fix] [quiet] command to bbstoreaccounts
Deleted locations in conf file get deleted from server
0.05 (10 Apr 2004)
Bug fixes
Added configurable "max diffing time" parameter, improved speed of diffing algorithm in edge cases
Added support for old versions of OpenSSL -- experimental only (but appears to work fine)
Read errors on files and directories are reported neatly, and no longer abort the backup scan
NetBSD port fixed
Server more tolerant of being unexpectedly aborted
0.04 (16 Mar 2004)
Bug fixes and code clean up
Change to standard BSD license
Expanded error messages
Improved flexibility with backup timing, including the ability to do snapshot backups
Exclude files explictly or with regular expressions
More work arounds for memory issues with STL in gcc 2.95.x (the gcc 3 version is fine)
In bbackupquery, add a 'ls' alias, and include 'help' command to document all commands
The *-config programs are improved and validates options and the configuration more to catch common problems
bbackupd-config excludes keys file if it might be backed up
Add bbackupctl program for controlling bbackupd daemon
bbstoreaccounts takes sizes in blocks, Mb or Gb with unit suffix
Start and end of last sync recorded in bbackupd working dir
bbackupquery compares check last modification time to see if this explains any difference
bbackupd will run a script run to alert administator that store is full (the default script sends an email)
0.03 (27 Jan 2004)
Improvements to the build system, especially configuring for random Linux systems.
Worked round an issue with the gcc 2.95.x libraries which meant that the backup daemon slowly used more and more memory.
Efficiency improvements.
0.02 (22 Jan 2004)
First public release

Documentation

The Box Backup wiki contains user contributed documentation, based on the documents below. Please use it for up to date information about Box Backup.

This is basic documentation, which will be improved. Note that configuration uses a number of scripts to do the hard work -- these all output a lot of detailed instructions on exactly what to do. Please read it.

Programmers notes on how the system works can be found within the notes directory inside the distribution archive. In particular, read notes/backup_encryption.txt for details of how the encryption scheme works, and where to look in the code to verify it.

Planned future work

Thanks

In addition to the developer credits for each release, we would like to thank:

Content © Ben Summers and Chris Wilson, 2003-2008

Web design by Joe Gillespie