Backup

If using the current default Badger backend, you can do online backups so never have to shutdown DVID. This works well for us and it's easy to make this a cron job so you can do incremental updates to some archival disk storage. Here's an example of a script that copies the mutation logs (both JSON and protobuf) and separate databases we use for metadata, labelmaps, meshes, and then everything else.

# preferably done on older backup just to minimize the rsync times.
# see: https://dgraph.io/docs/badger/get-started/#database-backup

set -o history
set -o histexpand

# backup the mutation logs in JSON and protobuf formats
rsync -av --delete /data1/mutlogs/aso-json/ /data2/backups/aso/mutlogs-json
rsync -av --delete /data1/mutlogs/aso/ /data2/backups/aso/mutlogs

# online backup of Badger database directories
rsync -av --delete /data1/dbs/aso/metadata /data2/backups/aso
while !! | grep -q "(MANIFEST\|\.sst)$"; do :; done

rsync -av --delete /data1/dbs/aso/default /data2/backups/aso
while !! | grep -q "(MANIFEST\|\.sst)$"; do :; done

rsync -av --delete /data1/dbs/aso/labelmaps /data2/backups/aso
while !! | grep -q "(MANIFEST\|\.sst)$"; do :; done

rsync -av --delete /data1/dbs/aso/meshes /data2/backups/aso
while !! | grep -q "(MANIFEST\|\.sst)$"; do :; done

You can configure DVID to use separate databases for different classes of data types or even assign particular data instances to its own database. This has the advantage of being able to restore just portions of your dataset to previously backed-up copies even if you haven't made a commit.

DVID has a bundled legacy backup program: dvid-backup but it was used for the basho-tuned leveldb. For basholeveldb databases, DVID must be halted very briefly (typically less than 10 seconds) while dvid-backup runs.

% dvid-backup -delete  /path/to/my/db/dir  /path/to/backup/dir

The dvid-backup command works in two stages. First, it copies all the mutable files at the top of the leveldb directory, then it creates hard links to the immutable files. The mutable files and hard links will be copied to a directory /path/to/my/db/dir-backup. The -delete option will delete and overwrite any dir-backup directory that already existed. This first stage will only take a few seconds, dvid-backup exits and you can restart DVID immediately. The dvid-backup program will also start rsync in the background, copying the bulk of the immutable data that is kept alive via the hard links.