Google BigTable

(very experimental)

Installation

To enable BigTable support when building DVID, include 'bigtable' in the DVID_BACKEND flag as in:

cmake -DDVID_BACKEND="bigtable" -DBUILDEM_DIR=$BUILDEM_DIR ..

Start BigTable Cluster

(requires a google account and a billable project)

Start a BigTable cluster on google: directions.

Record the cluster name, the project the cluster is under, and the zone.

Configure DVID

Point to the BigTable cluster by modifying the DVID toml configuration file. Here is a sample toml that will cause DVID to run on port 8000 and will point to a BigTable cluster called "nameofcluster" in a project "nameofproject" that is running in zone "us-central1-c".

[server]
httpAddress = "0.0.0.0:8000"
rpcAddress = "0.0.0.0:8001"
webClient = "/usr/local/dvid-console"
# How new data instance ids are generated.
# Is one of "random" or "sequential".  If "sequential" can set "start_instance_id" property.
instance_id_gen = "sequential"
instance_id_start = 100  # new ids start at least from this.
[logging]
logfile = "/tmp/logs/dvid.log"
max_log_size = 500 # MB
max_log_age = 30   # days
[store]  
# if no store.metadata is provided, it defaults to store.mutable configuration
[store.mutable]
    engine = "bigtable"
    project = "nameofproject"
    cluster = "nameofcluster"
    table = "mutable"
    zone = "us-central1-c" 

Launching DVID

Running DVID on Google VM

Create a VM instance using the desired hardware configuration:

1. The zones should match between BigTable and the VM instance.
2. Check 'Allow http traffic'
3. Check 'Allow API access to all Google Cloud services in the same project.' so that BigTable can be accessed from the VM. This assumes that the BigTable cluster is running on the same account as the VM.

Running DVID outside of Google

To access BigTable, the machine / environment running DVID should have access to google services. One should just use application default credentials so that DVID is given access to everything. (Access to particular resources in BigTable should be managed by DVID.) There are instructions here: https://developers.google.com/identity/protocols/application-default-credentials. The easiest solution is to set the GOOGLE_APPLICATION_CREDENTIALS environment to point to a JSON that can be downloaded from the developers console.