Skip to content

VictoriaMetrics/VictoriaMetrics

VictoriaMetrics

Latest Release Docker Pulls victoriametrics Slack GitHub license Go Report Build Status codecov

VictoriaMetrics logo

VictoriaMetrics is a fast, cost-effective and scalable monitoring solution and time series database. See case studies for VictoriaMetrics.

VictoriaMetrics is available in binary releases, Docker images, Snap packages and source code.

Documentation for the cluster version of VictoriaMetrics is available here.

Learn more about key concepts of VictoriaMetrics and follow the quick start guide for a better experience.

If you have questions about VictoriaMetrics, then feel free asking them in the VictoriaMetrics community Slack chat, you can join it via Slack Inviter.

Contact us if you need enterprise support for VictoriaMetrics. See features available in enterprise package. Enterprise binaries can be downloaded and evaluated for free from the releases page. You can also request a free trial license.

VictoriaMetrics is developed at a fast pace, so it is recommended to check the CHANGELOG periodically, and to perform regular upgrades.

VictoriaMetrics enterprise provides long-term support lines of releases (LTS releases) - see these docs.

VictoriaMetrics has achieved security certifications for Database Software Development and Software-Based Monitoring Services. We apply strict security measures in everything we do. See Security page for more details.

Prominent features

VictoriaMetrics has the following prominent features:

See case studies for VictoriaMetrics and various Articles about VictoriaMetrics.

Components

VictoriaMetrics ecosystem contains the following components additionally to single-node VictoriaMetrics:

  • vmagent - lightweight agent for receiving metrics via pull-based and push-based protocols, transforming and sending them to the configured Prometheus-compatible remote storage systems such as VictoriaMetrics.
  • vmalert - a service for processing Prometheus-compatible alerting and recording rules.
  • vmalert-tool - a tool for validating alerting and recording rules.
  • vmauth - authorization proxy and load balancer optimized for VictoriaMetrics products.
  • vmgateway - authorization proxy with per-tenant rate limiting capabilities.
  • vmctl - a tool for migrating and copying data between different storage systems for metrics.
  • vmbackup, vmrestore and vmbackupmanager - tools for creating backups and restoring from backups for VictoriaMetrics data.
  • vminsert, vmselect and vmstorage - components of VictoriaMetrics cluster.
  • VictoriaLogs - user-friendly cost-efficient database for logs.

Operation

Install

To quickly try VictoriaMetrics, just download the VictoriaMetrics executable or Docker image and start it with the desired command-line flags. See also QuickStart guide for additional information.

VictoriaMetrics can also be installed via these installation methods:

How to start VictoriaMetrics

The following command-line flags are used the most:

  • -storageDataPath - VictoriaMetrics stores all the data in this directory. The default path is victoria-metrics-data in the current working directory.
  • -retentionPeriod - retention for stored data. Older data is automatically deleted. Default retention is 1 month (31 days). The minimum retention period is 24h or 1d. See these docs for more details.

Other flags have good enough default values, so set them only if you really need to. Pass -help to see all the available flags with description and default values.

The following docs may be useful during initial VictoriaMetrics setup:

VictoriaMetrics accepts Prometheus querying API requests on port 8428 by default.

It is recommended setting up monitoring for VictoriaMetrics.

Environment variables

All the VictoriaMetrics components allow referring environment variables in yaml configuration files (such as -promscrape.config) and in command-line flags via %{ENV_VAR} syntax. For example, -metricsAuthKey=%{METRICS_AUTH_KEY} is automatically expanded to -metricsAuthKey=top-secret if METRICS_AUTH_KEY=top-secret environment variable exists at VictoriaMetrics startup. This expansion is performed by VictoriaMetrics itself.

VictoriaMetrics recursively expands %{ENV_VAR} references in environment variables on startup. For example, FOO=%{BAR} environment variable is expanded to FOO=abc if BAR=a%{BAZ} and BAZ=bc.

Additionally, all the VictoriaMetrics components allow setting flag values via environment variables according to these rules:

  • The -envflag.enable flag must be set.
  • Each . char in flag name must be substituted with _ (for example -insert.maxQueueDuration <duration> will translate to insert_maxQueueDuration=<duration>).
  • For repeating flags an alternative syntax can be used by joining the different values into one using , char as separator (for example -storageNode <nodeA> -storageNode <nodeB> will translate to storageNode=<nodeA>,<nodeB>).
  • Environment var prefix can be set via -envflag.prefix flag. For instance, if -envflag.prefix=VM_, then env vars must be prepended with VM_.

Configuration with snap package

Snap package for VictoriaMetrics is available here.

Command-line flags for Snap package can be set with following command:

echo 'FLAGS="-selfScrapeInterval=10s -search.logSlowQueryDuration=20s"' > $SNAP_DATA/var/snap/victoriametrics/current/extra_flags
snap restart victoriametrics

Do not change value for -storageDataPath flag, because snap package has limited access to host filesystem.

Changing scrape configuration is possible with text editor:

vi $SNAP_DATA/var/snap/victoriametrics/current/etc/victoriametrics-scrape-config.yaml

After changes were made, trigger config re-read with the command curl 127.0.0.1:8428/-/reload.

Running as Windows service

In order to run VictoriaMetrics as a Windows service it is required to create a service configuration for WinSW and then install it as a service according to the following guide:

  1. Create a service configuration:

    <service>
      <id>VictoriaMetrics</id>
      <name>VictoriaMetrics</name>
      <description>VictoriaMetrics</description>
      <executable>%BASE%\victoria-metrics-windows-amd64-prod.exe"</executable>
    
      <onfailure action="restart" delay="10 sec"/>
      <onfailure action="restart" delay="20 sec"/>
    
      <resetfailure>1 hour</resetfailure>
    
      <arguments>-envflag.enable</arguments>
    
      <priority>Normal</priority>
    
      <stoptimeout>15 sec</stoptimeout>
    
      <stopparentprocessfirst>true</stopparentprocessfirst>
        <startmode>Automatic</startmode>
        <waithint>15 sec</waithint>
        <sleeptime>1 sec</sleeptime>
    
      <logpath>%BASE%\logs</logpath>
      <log mode="roll">
        <sizeThreshold>10240</sizeThreshold>
        <keepFiles>8</keepFiles>
      </log>
    
      <env name="loggerFormat" value="json" />
      <env name="loggerOutput" value="stderr" />
      <env name="promscrape_config" value="C:\Program Files\victoria-metrics\promscrape.yml" />
    
    </service>
  2. Install WinSW by following this documentation.

  3. Install VictoriaMetrics as a service by running the following from elevated PowerShell:

    winsw install VictoriaMetrics.xml
    Get-Service VictoriaMetrics | Start-Service

See this issue for more details.

Prometheus setup

Add the following lines to Prometheus config file (it is usually located at /etc/prometheus/prometheus.yml) in order to send data to VictoriaMetrics:

remote_write:
  - url: http://<victoriametrics-addr>:8428/api/v1/write

Substitute <victoriametrics-addr> with hostname or IP address of VictoriaMetrics. Then apply new config via the following command:

kill -HUP `pidof prometheus`

Prometheus writes incoming data to local storage and replicates it to remote storage in parallel. This means that data remains available in local storage for --storage.tsdb.retention.time duration even if remote storage is unavailable.

If you plan sending data to VictoriaMetrics from multiple Prometheus instances, then add the following lines into global section of Prometheus config:

global:
  external_labels:
    datacenter: dc-123

This instructs Prometheus to add datacenter=dc-123 label to each sample before sending it to remote storage. The label name can be arbitrary - datacenter is just an example. The label value must be unique across Prometheus instances, so time series could be filtered and grouped by this label.

For highly loaded Prometheus instances (200k+ samples per second) the following tuning may be applied:

remote_write:
  - url: http://<victoriametrics-addr>:8428/api/v1/write
    queue_config:
      max_samples_per_send: 10000
      capacity: 20000
      max_shards: 30

Using remote write increases memory usage for Prometheus by up to ~25%. If you are experiencing issues with too high memory consumption of Prometheus, then try to lower max_samples_per_send and capacity params. Keep in mind that these two params are tightly connected. Read more about tuning remote write for Prometheus here.

It is recommended upgrading Prometheus to v2.12.0 or newer, since previous versions may have issues with remote_write.

Take a look also at vmagent and vmalert, which can be used as faster and less resource-hungry alternative to Prometheus.

Grafana setup

Create Prometheus datasource in Grafana with the following url:

http://<victoriametrics-addr>:8428

Substitute <victoriametrics-addr> with the hostname or IP address of VictoriaMetrics.

In the "Type and version" section it is recommended to set the type to "Prometheus" and the version to at least "2.24.x":

Grafana datasource

This allows Grafana to use a more efficient API to get label values.

Then build graphs and dashboards for the created datasource using PromQL or MetricsQL.

Alternatively, use VictoriaMetrics datasource plugin with support of extra features. See more in description.

Creating a datasource may require specific permissions. If you don't see an option to create a data source - try contacting system administrator.

Grafana playground is available for viewing at our sandbox.

How to upgrade VictoriaMetrics

VictoriaMetrics is developed at a fast pace, so it is recommended periodically checking the CHANGELOG page and performing regular upgrades.

It is safe upgrading VictoriaMetrics to new versions unless release notes say otherwise. It is safe skipping multiple versions during the upgrade unless release notes say otherwise. It is recommended performing regular upgrades to the latest version, since it may contain important bug fixes, performance optimizations or new features.

It is also safe downgrading to older versions unless release notes say otherwise.

The following steps must be performed during the upgrade / downgrade procedure:

  • Send SIGINT signal to VictoriaMetrics process in order to gracefully stop it. See how to send signals to processes.
  • Wait until the process stops. This can take a few seconds.
  • Start the upgraded VictoriaMetrics.

Prometheus doesn't drop data during VictoriaMetrics restart. See this article for details. The same applies also to vmagent.

vmui

VictoriaMetrics provides UI for query troubleshooting and exploration. The UI is available at http://victoriametrics:8428/vmui (or at http://<vmselect>:8481/select/<accountID>/vmui/ in cluster version of VictoriaMetrics). The UI allows exploring query results via graphs and tables. It also provides the following features:

VMUI provides auto-completion for MetricsQL functions, metric names, label names and label values. The auto-completion can be enabled by checking the Autocomplete toggle. When the auto-completion is disabled, it can still be triggered for the current cursor position by pressing ctrl+space.

VMUI automatically switches from graph view to heatmap view when the query returns histogram buckets (both Prometheus histograms and VictoriaMetrics histograms are supported). Try, for example, this query.

Graphs in vmui support scrolling and zooming:

  • Select the needed time range on the graph in order to zoom in into the selected time range. Hold ctrl (or cmd on MacOS) and scroll down in order to zoom out.
  • Hold ctrl (or cmd on MacOS) and scroll up in order to zoom in the area under cursor.
  • Hold ctrl (or cmd on MacOS) and drag the graph to the left / right in order to move the displayed time range into the future / past.

Query history can be navigated by holding Ctrl (or Cmd on MacOS) and pressing up or down arrows on the keyboard while the cursor is located in the query input field.

Multi-line queries can be entered by pressing Shift-Enter in query input field.

When querying the backfilled data or during query troubleshooting, it may be useful disabling response cache by clicking Disable cache checkbox.

VMUI automatically adjusts the interval between datapoints on the graph depending on the horizontal resolution and on the selected time range. The step value can be customized by changing Step value input.

VMUI allows investigating correlations between multiple queries on the same graph. Just click Add Query button, enter an additional query in the newly appeared input field and press Enter. Results for all the queries are displayed simultaneously on the same graph. Graphs for a particular query can be temporarily hidden by clicking the eye icon on the right side of the input field. When the eye icon is clicked while holding the ctrl key, then query results for the rest of queries become hidden except of the current query results.

VMUI allows sharing query and trace results by clicking on Export query button in top right corner of the graph area. The query and trace will be exported as a file that later can be loaded in VMUI via Query Analyzer tool.

See the example VMUI at VictoriaMetrics playground.

Top queries

VMUI provides top queries tab, which can help determining the following query types:

  • the most frequently executed queries;
  • queries with the biggest average execution duration;
  • queries that took the most summary time for execution.

This information is obtained from the /api/v1/status/top_queries HTTP endpoint.

Active queries

VMUI provides active queries tab, which shows currently execute queries. It provides the following information per each query:

  • The query itself, together with the time range and step args passed to /api/v1/query_range.
  • The duration of the query execution.
  • The client address, who initiated the query execution.

This information is obtained from the /api/v1/status/active_queries HTTP endpoint.

Metrics explorer

VMUI provides an ability to explore metrics exported by a particular job / instance in the following way:

  1. Open the vmui at http://victoriametrics:8428/vmui/.
  2. Click the Explore Prometheus metrics tab.
  3. Select the job you want to explore.
  4. Optionally select the instance for the selected job to explore.
  5. Select metrics you want to explore and compare.

It is possible to change the selected time range for the graphs in the top right corner.

Cardinality explorer

VictoriaMetrics provides an ability to explore time series cardinality at Explore cardinality tab in vmui in the following ways:

  • To identify metric names with the highest number of series.
  • To identify labels with the highest number of series.
  • To identify values with the highest number of series for the selected label (aka focusLabel).
  • To identify label=name pairs with the highest number of series.
  • To identify labels with the highest number of unique values. Note that cluster version of VictoriaMetrics may show lower than expected number of unique label values for labels with small number of unique values. This is because of implementation limits.

By default, cardinality explorer analyzes time series for the current date. It provides the ability to select different day at the top right corner. By default, all the time series for the selected date are analyzed. It is possible to narrow down the analysis to series matching the specified series selector.

Cardinality explorer is built on top of /api/v1/status/tsdb.

See cardinality explorer playground. See the example of using the cardinality explorer here.

Cardinality explorer statistic inaccuracy

In cluster version of VictoriaMetrics each vmstorage tracks the stored time series individually. vmselect requests stats via /api/v1/status/tsdb API from each vmstorage node and merges the results by summing per-series stats. This may lead to inflated values when samples for the same time series are spread across multiple vmstorage nodes due to replication or rerouting.

How to apply new config to VictoriaMetrics

VictoriaMetrics is configured via command-line flags, so it must be restarted when new command-line flags should be applied:

  • Send SIGINT signal to VictoriaMetrics process in order to gracefully stop it.
  • Wait until the process stops. This can take a few seconds.
  • Start VictoriaMetrics with the new command-line flags.

Prometheus doesn't drop data during VictoriaMetrics restart. See this article for details. The same applies also to vmagent.

How to scrape Prometheus exporters such as node-exporter

VictoriaMetrics can be used as drop-in replacement for Prometheus for scraping targets configured in prometheus.yml config file according to the specification. Just set -promscrape.config command-line flag to the path to prometheus.yml config - and VictoriaMetrics should start scraping the configured targets. If the provided configuration file contains unsupported options, then either delete them from the file or just pass -promscrape.config.strictParse=false command-line flag to VictoriaMetrics, so it will ignore unsupported options.

The file pointed by -promscrape.config may contain %{ENV_VAR} placeholders, which are substituted by the corresponding ENV_VAR environment variable values.

See also:

VictoriaMetrics also supports importing data in Prometheus exposition format.

See also vmagent, which can be used as drop-in replacement for Prometheus.

How to send data from DataDog agent

VictoriaMetrics accepts data from DataDog agent, DogStatsD and DataDog Lambda Extension via "submit metrics" API at /datadog/api/v2/series or via "sketches" API at /datadog/api/beta/sketches.

Sending metrics to VictoriaMetrics

DataDog agent allows configuring destinations for metrics sending via ENV variable DD_DD_URL or via configuration file in section dd_url.

To configure DataDog agent via ENV variable add the following prefix: