Metric types in Datadog

With Datadog, we offer three different submission methods: agent checks, dogstatsd, and the HTTP API, each with their own understanding of metric types.

Furthermore the web application (app.datadoghq.com) has its own set of metric types that don't always intuitively follow from the submission-time types. This page is meant to document how submission metric types relate to the in-app types.

Table Of Contents

  • Quick view
  • Metric Types During Submission
    1. Agent Check Submission
    2. Dogstatsd Submission
    3. HTTP API Submission
  • Metric Types In the web application

Quick View

Here's a summary of how submission-time types map to in-app types:

Submission Source Submission Method (python) Submission Type Datadog In-App Type
API api.Metric.send(...) gauge gauge
dogstatsd dog.gauge(...) gauge gauge
dogstatsd dog.increment(...) counter rate
dogstatsd dog.histogram(...) histogram gauge, rate
dogstatsd dog.set(...) set gauge
agent check self.gauge(...) gauge gauge
agent check self.increment(...) counter rate
agent check self.rate(...) rate gauge
agent check self.count(...) count count
agent check self.monotonic_count(...) monotonic_count count
agent check self.histogram(...) histogram gauge, rate
agent check self.set(...) set gauge

 

Metric Types During Submission

Agent Check Submission

Here is some information about metric submission in an agent check:

self.gauge( ... )

  • If called multiple times during a check's execution for a metric only the last sample will be used.
  • Stored as a Web App GAUGE type

self.increment( ... )

  • Usage: Used to modify a count of events identified by the metric key string.
  • Can be called multiple times during a check's execution.
  • Stored as a RATE type in the datadog web application. Each value in the stored timeseries is a delta of the counter's value between samples (time-normalized by the aggregation interval which defaults to 1 for agent checks - so the value is generally the raw count value).
  • Handled by the aggregator Counter class

self.decrement( ... )

  • Usage: Used to modify a count of events identified by the metric key string.
  • Can be called multiple times during a check's execution.
  • Stored as RATE type in the datadog web application. Each value in the stored timeseries is a delta of the counter's value between samples (time-normalized by the aggregation interval which defaults to 1 for agent checks - so the value is generally the raw count value).
  • Handled by the aggregator Counter class

self.rate( ... )

  • Usage: Submit the sampled raw value of your counter. Don't normalize the values to a rate, or calculate the deltas before submitting - the agent does both for you.
  • Should only be called once during a check.
  • Throws away any value that is less than a previously submitted value. IE the counter should be monotonically increasing.
  • Stored as a GAUGE type in the datadog web application. Each value in the stored timeseries is a time-normalized delta of the counter's value between samples.

self.count( ... )

  • Usage: Submit the number of events that occurred during the check interval. If you're tracking a counter value that persists between checks, this means you must calculate the delta before submission.
  • Should only be called once during a check.
  • Stored as a COUNT type in the datadog web application. Each value in the stored timeseries is a delta of the counter's value between samples (not time-normalized).

self.monotonic_count( ... )

  • Usage: Submit the sampled raw value of your counter. Don't normalize the values to a rate, or calculate the deltas before submitting. If the value of your counter ever decreases between submissions the resulting stored value for that submission is 0.
  • Should only be called once during a check.
  • Throws away any value that is less than a previously submitted value. IE the counter should be monotonically increasing.
  • Stored as a COUNT type in the datadog web application. Each value in the stored timeseries is a delta of the counter's value between samples (not time-normalized).

self.histogram( ... )

  • Usage: Used to track the statistical distribution of a set of values.
  • Should be called multiple times during an agent check (otherwise you have no distribution).
  • Actually submits as multiple metrics:

Name | Web App type
-----|------------
 metric.max | GAUGE
 metric.avg | GAUGE
 metric.median | GAUGE
 metric.95percentile | GAUGE
 metric.count | RATE

self.set( ... )

  • Usage: Used count the number of unique elements in a group.
  • Should be called multiple times during an agent check.
  • Stored as a GAUGE type in the datadog web application

Dogstatsd Submission

Here is some information about metric submission in an agent check:

Because dogstatsd flushes at a regular interval (default 10s) all metrics submitted via this method will be stored with associated interval metadata.

dog.gauge(...)

  • Stored as a GAUGE type in the datadog web application. Each value in the stored timeseries is the last gauge value submitted for that metric during the statsd flush period.

dog.increment(...)

  • Usage: Used to increment a counter of events.
  • Stored as a RATE type in the datadog web application. Each value in the stored timeseries is a time-normalized delta of the counter's value over that statsd flush period.

dog.decrement(...)

  • Usage: Used to decrement a counter of events.
  • Stored as a RATE type in the datadog web application. Each value in the stored timeseries is a time-normalized delta of the counter's value over that statsd flush period.

dog.histogram(...)

  • Usage: Used to track the statistical distribution of a set of values over a statsd flush period.
  • Actually submits as multiple metrics:


name | Web App type
-----|------------
metric.max | GAUGE
metric.avg | GAUGE
metric.median | GAUGE
metric.95percentile | GAUGE
metric.count | RATE

dog.set(...)

  • Usage: Used count the number of unique elements in a group.
  • Stored as GAUGE type in the datadog web application. Each value in the stored timeseries is the count of unique values submitted to statsd for a metric over that flush period.

HTTP API Submission

API Documentation

For the API all metrics are submitted the same way, with the type specified as a parameter.

Gauge

  • Stored in Web App as GAUGE type

Metric Types in the web application

In the web app there are 3 metric types: GAUGE, RATE, COUNT (and COUNTER, now deprecated). A metric's type is stored as metrics metadata and is used to determine how a metric is interpreted throughout the app by determining default time aggregation function and as_rate()/as_count() behavior.

The as_count() and as_rate() modifiers

The as_count() and as_rate() modifiers behave differently for different Web App metric types. For count type metrics (the count and rate in app metric types), as_count() and as_rate have the effect of fill(0) (https://help.datadoghq.com/hc/en-us/articles/209279966-Interpolation-the-fill-modifier-explained).

GAUGE

  • Effect of as_count(): None
  • Effect of as_rate(): None

RATE:

  • Effect of as_count():
    • Sets the time aggregator to SUM.
    • Uses the metadata interval to convert from raw rates to counts. Does not work if no metadata interval exists for the metric.
  • Effect of as_rate():
    • Sets the time aggregator to SUM.
    • Uses the query interval and metadata interval to calculate the time-aggregated rate. Does not work if no metadata interval exists for the metric.
  • Known Issue: Agent check submitted RATE metrics have no interval metadata, so as_rate() and as_count() don't work.

COUNT:

  • Effect of as_count():
    • Sets the time aggregator to SUM.
  • Effect of as_rate():
    • Sets the time aggregator to SUM
    • Normalizes the input timeseries values by the query (rollup) interval. For example [1,1,1,1].as_rate() for rollup interval of 20s produces [0.05, 0.05, 0.05, 0.05].
  • The raw metric itself will default to the time aggregator AVG, so querying the metric without either as_rate() or as_count()will become non-sensical when time aggregation is applied.
  • Note that on very small intervals when no time-aggregation occurs, there is no normalization, and you get the raw metric value counts.
Have more questions? Submit a request

0 Comments

Article is closed for comments.
Powered by Zendesk