Simple Example

In this example, we'll initialize a new datastore, create grayscale data, load some images, and then use a web browser to see the results. Although this demo uses the command line, in the near future we'll likely deprecate all command line interaction (except serve and repair commands) and force interaction through HTTP requests.

Getting help

If DVID was installed correctly, you should be able to ask dvid for help:

% dvid help

Configure DVID

Read the Configuration wiki page and make sure you have a valid configuration TOML file, say at /path/to/config.toml. For this simple example, you can skip specifying any [logging] options so DVID will log directly to standard output.

In earlier versions of DVID, creation of a DVID datastore was an explicit create command. However the current version of DVID will auto-create any local database if a database does not exist at the local database path in the configuration TOML. Obviously, data service backends (e.g., Janelia KVAutobus service or Google data services) are permanent and do not require creation.

Start the DVID server

The "-verbose" option makes DVID quite chatty, logging almost all of its requests and timing information.

% dvid -verbose serve /path/to/config.toml

Open another terminal and run "dvid help" again. You'll see more information because DVID can detect a running server and describe the data types available. You can also ask for just the data types supported by this DVID server:

% dvid types

Note that there is one DVID process handling requests on the RPC and HTTP ports either configured in the TOML file or 8000 (HTTP) and 8001 (RPC) by default. When you run a second dvid command, that command-line process sets up a RPC connection to the DVID server process. If you've configured a non-default RPC or HTTP port, you must specify this port:

% dvid -rpc=:6001 types

The above DVID command contacts the server listening at RPC port 6001 and asks for its types. The commands below must include the rpc option if you are connecting to a DVID server with a non-default RPC port.

Once you start a DVID server, you can get help on server-supported commands, in this case from a server operating at the default ports (so no -rpc flag is necessary):

% dvid help server

In addition to the dvid command-line requests you can enter into the terminal, you can also use the larger HTTP API from either a browser (by entering the URL into the address bar) or via a command-line program like httpie (recommended) or curl (usually already installed in most systems but more unwieldy than httpie). For example, supported dvid types can be retrieved via the HTTP API endpoint /api/server/types:

% curl http://localhost:6000/api/server/types

The above command queries the dvid server with HTTP port 6000.

Stopping the Server

While DVID catches various termination signals, it's possible that abrupt termination could cause corruption of operations being processed, so we recommend using the "shutdown" command:

% dvid -rpc=:6001 shutdown

This will clean up the service and wait a few seconds for outstanding processing to finish.

DVID API Documentation

The DVID API documentation can be found (using a browser) at:

 [HTTP address of DVID]/api/help.

The help pages are embedded in every DVID server and should be the most current documentation for the command-line and HTTP API associated with each data type. If you are writing a client for DVID, you can either directly access the HTTP API or use a DVID client library that wraps the HTTP API in C++ or python.

Documentation for direct HTTP API access can be found in two places:

 [HTTP address of DVID]/api/help.
 [HTTP address of DVID]/api/help/[datatype name]

The first help page describes the most basic HTTP commands like creating a repo, creating a data instance, etc. That page also includes links to each supported datatype and its HTTP API.

For the rest of this simple example, we will use dvid on a command line rather than the HTTP API. Certain features like deleting a repo or data instance can only be accessed via the command line at this time. Rather than clutter the commands with server ports like -rpc=:7001, we assume that a DVID server is running on the default ports (HTTP 8000, RPC 8001), so we can omit the rpc specification from dvid commands.

Note that this is a toy example and that for real production use, we always use scripts that target the HTTP endpoints, particularly for ingestion of data. After we introduce an authentication/authorization system into DVID, we'll add a HTTP-only simple example page.

Create a new repository (repo)

One DVID server can manage different repositories (repos), each of which correspond to a set of possibly versioned data in a consistent coordinate system. We think of a repo as all the data associated with a particular alignment of images. If the underlying image data is changed dramatically, e.g., through realignment, warping, etc, it should be given a separate repo. We create a new repo like so:

% dvid repos new Dataset1 "Dataset for testing DVID use"

A hexadecimal string will be printed in response. Repos are identified by the global UUID of its root version, usually printed and read in hexadecimal format (e.g., "c78a0"). When supplying a UUID, you only need enough letters to uniquely identify the UUID within that DVID server. For example, if there are only two repos, each with only one version, and the root versions for each repo are "c78a0..." and "da0a4...", respectively, then you can uniquely specify the repos using "c7" and "da". (We might use one letter, but generally two or more letters are better.)

% dvid types

Entering the above command will show the new repo but there are no data under it.

Create new data for a repo

We can create an instance of a supported data type for the new repo:

% dvid repo c7 new uint8blk grayscale

Replace "c7" with the first two letters of whatever UUID was printed when you created a new repo. You can also see the UUIDs for repos by using the "dvid types" command.

After adding new data, you can see the result via the "dvid types" command. It will show new data named "grayscale" that is of uint8blk data type. DVID allows a variety of data types, each implemented by some code that determines that data type's commands, HTTP API, and method of storing and retrieving data. The data type implementation is uniquely identified by the URL of that implementation's package or file.

For almost all commands, there's a HTTP API equivalent that you can find by browsing /api/help or /api/help/myGreatDatatype. In the case of creating a data instance, you can POST JSON to /api/repo/{uuid}/instance. This can be easily done using the httpie command (or curl, not shown):

% echo "{\"typename\": \"keyvalue\", \"dataname\": \"mydata\"}" | http POST http://path-to-dvid/api/repo/c7/instance

The above creates a new keyvalue data instance with name "mydata", rooted at the UUID c7.

Get data type-specific help

Since each data type has its own set of commands and HTTP API, we need to know what's available from each type:

% dvid types uint8blk help

The above asks the uint8blk data type implementation for its usage. In the next step, we will use the "load local" command described in the help response.

Loading some sample data

Note: For real uses of DVID at the scale of most reconstructions, we highly recommend writing scripts to ingest data via the HTTP API. For the purposes of this tutorial, this simpler ingestion approach is used to add an image volume.

Download a small stack of grayscale images from github.

You can either clone it using git or use the "Download ZIP" button on the right. Once you've downloaded that 250 x 250 x 250 grayscale volume, enter the following:

% dvid node c7 grayscale load 100,100,2600 "/absolute/path/to/sample/*.png"

While you can use relative paths here, note that they are relative to the current working directory of the DVID server process, not the working directory of the DVID process sending the load command.

Viewing data using a browser

Once some images are loaded, you can view them using a web browser or specialized clients for DVID. For this simple example, open a browser and enter the following URL to retrieve a 300x300 pixel image at offset (75,75,2610):

[HTTP address of DVID]/api/node/c7/grayscale/raw/xy/300_300/75_75_2610

You should see an image displayed in your browser with some black area surrounding it. The full REST API can be viewed by visiting the built-in help pages from your browser:

[HTTP address of DVID]/api/help/uint8blk
[HTTP address of DVID]/api/help

Deleting a data instance

You've grown tired of this data and want to delete it from your DVID server. You can do this via the following command:

% dvid repo c7 delete grayscale

Deletions are asynchronous so while it the command returns immediately and you will no longer be able to see the data instance on the web console, the actual deletion of key-value pairs may take some time depending on the size of the data instance.