It's possible to get percentiles in Datadog by submitting data as a histogram metric through dogstatsd. The datadog-agent embeds a dogstatsd server that will receive dogstatsd packets, perform data aggregation, and send final percentile metrics to Datadog.
Since this aggregation is taken care of on the collection side, this isn't currently available as a graphing function in the GUI.
Out of your histogram data you'll get: 95th percentile, 50th percentile, avg, max, count.
- A quick introduction to dogstatsd: Can-I-use-my-own-StatsD-client-What-is-DogStatsD
- Dogstatsd clients available for each programming language: http://docs.datadoghq.com/libraries/
Via the "histogram_percentiles" line of the configuration file of the agent, you can get extra percentiles, e.g.:
histogram_percentiles: 0.95, 0.75
- More about histogram here: http://docs.datadoghq.com/guides/metrics/#histograms
Characteristics of Datadog percentiles
- They are computed on a host per host (and tag list per tag list) basis.
- They're computed every 10 seconds
See more details below and the full explanation in this other article: https://help.datadoghq.com/hc/en-us/articles/211545826
Global percentile beta
Datadog makes it now possible to have the percentiles computed globally on the backend side across your host (no more local aggregation only), see description here: https://help.datadoghq.com/hc/en-us/articles/115005362583-Global-Percentiles.
If you're interested in this beta please drop a note to email@example.com!
Local Aggregations (i.e. without the beta)
Histograms are computed every 10 seconds on a host per host basis by the datadog-agents. This collection model comes with its advantages and its limitations.
- Raw datapoints used to compute histogram metrics are not exposed nor relayed to our site.
- Statsd handles the relevant aggregations and submits the calculated data package to our server directly.
- If you have two reporting streams of aggregated data, it is not possible today to aggregate across the raw datapoints from both streams, only aggregate across the aggregates.
- EX: Averaging across METRIC_NAME.Avg for all regions, takes the average stream values for each region and produces an average of averages.
- Making a change to increase tag complexity (adding additional tags to be more specific) will lead to changes in the behavior of a rolled up metric visualization
- EX: Whereas before the change METRIC_NAME.avg (without any tags) would be aggregating across all raw points (statsd takes all the raw datapoints, aggregates it and then ships over a single metric stream), adding a tag like region (US, EU) tag causes statsd to bin raw datapoints into two region bins, aggregate them, and ship over two streams. This means when graphing METRIC_NAME.avg AVG by * means an aggregate across the two streams rather than a single one