What is a custom metric and what is the limit on the number of custom metrics I can have?

Datadog allows for custom metrics submission through multiple ways to offer a full understanding of what is happening on your infrastructure.

This article will explain:

  • What a custom metric is, and how you can submit it to Datadog.
  • How many custom metrics we allow for out of the box.
  • How to check your custom metric count over time.
  • Some best practices for using custom metrics.  

 

Prerequisite - how can I send a custom metric to Datadog ?  

How is a custom metric defined ?

A custom metric refers to a single, unique combination of a metric name, host, and any tags.

Custom metrics generally refer to any metric that you send using statsd, dogstatsd, or through extensions made to the Datadog Agent. Some integrations can potentially emit an unlimited number of metrics that can also count as custom, further details can be found here.

In order to fully leverage the capabilities of the Datadog product through scoping and alerting, you’ll probably be using tags. As a consequence, one submitted metric will actually lead to multiple unique tag combinations - counting towards your custom metrics count.

For example:

  • You submit the following metric name: auth.exceptionCount
  • Your code instrumentation plans the following tags associated with that metric: "method:X", "method:Y", "exception:A", "exception:B".
  • The logic behind your metric is the following :

Image_2017-03-23_at_3.31.51_PM.png

 

The given unique metrics on a given host will therefore be :

  • auth.exceptionCount with tag “method:X
  • auth.exceptionCount with tag “method:Y”
  • auth.exceptionCount with tags “method:X” and “exception:A” //unique because of new tag “exception:A
  • auth.exceptionCount with tags “method:X” and “exception:B
  • auth.exceptionCount with tags “method:Y” and “exception:A
  • auth.exceptionCount with tags “method:Y” and “exception:B”

In this situation, you would end up with 6 different metrics.

Note that the ordering of tags does not matter, so the following two metrics would be considered non-unique:

  • auth.exceptionCount with tags “method:X” and “exception:A”
  • auth.exceptionCount with tags “exception:A” and “method:X

 

How many custom metrics am I allowed?  

Datadog offers 2 plans - Pro & Enterprise. Pro customers are allotted 100 custom metrics per host & Enterprise customers are allotted 200 custom metrics per host across their entire infrastructure rather than on a per-host basis. For example, if you were on the Pro plan and are licensed for 3 hosts, you would have 300 custom metrics by default - these 300 metrics may be divided equally amongst each individual host, or all 300 metrics could be sent from a single host.

Using the aforementioned example, below shows three scenarios which would all be acceptable without exceeding the default metric count for three hosts:

To sum it up, your global custom metric count is aggregated across your monitored cluster :

custom-metrics-1.jpg

That said, we do not enforce any fixed rate limit on custom metric submission, we'll simply reach out to you if you're exceeding your default allotment.

 

How do I check my custom metrics count?

 

When creating a custom metric, all the host tags will be automatically added to that metric as one unique tag combination, to which you'll add the tags linked to the metric itself. Those are the most important as they will add to the actual metric count.

Let's say you want to have insight into the request.count from different services across your infrastructure.

  • You create your metric service.request.count
  • You want to separate the requests that were successful from the failures. You create two tags to that effect:
  1. status:success
  2. status:failure
  • You want this metric to be reported by each service running on your infrastructure. Let's say you have 3 services per host:
  1. service:database
  2. service:api
  3. service:webserver

The logic behind your metric is the following :

Image_2017-03-23_at_3.38.10_PM.png 

 

From there, you can see that on each host reporting this metric, if all services report both successes and failures, you can have up to 1x2x3 = 6 custom metrics.

Let’s say you have 3 hosts:

  • host1 is reporting all possible configurations
  • host2 is reporting only successes across all services
  • host3 is reporting success and failures, but only for database and webserver services

Across your 3 hosts, you’d have 13 distinct metrics, here is why :

 

Image_2017-03-23_at_3.38.37_PM.png

 

If you are an admin, you can see your total custom metrics per hour as well as the top 500 custom metrics by cardinality in your account in the usage details page. You can also see this metric count on your metric summary page, where you’d see, clicking on the service.request.count metric, the exact number of unique tag combinations:

So if you only had the first host from the example above reporting, you’d have this:

 

_312faf88c063aa6a3a0fe2e6b7da538f__Image_2017-02-15_at_11.31.41_AM.png

Adding the second host:

_7dfdc35e7ae8f87043fc0b6b46453bee__Image_2017-03-14_at_5.27.41_PM.png

 

Adding the third host as per the table above, you get your 13 distinct metrics:

_2c3e1e6e061ed5d97b7caacdf6222cb1__Image_2017-03-14_at_6.39.01_PM.png

 

Using the query editor, you can also find this using the count: aggregator

_70d03ccfa85ab9ab3cfbdcb2688fbdec__Image_2017-04-28_at_7.09.11_PM.png

 

Ultimately, you’ll have 13 metrics using the following query:

count:service.request.count{*}

Image_2017-03-14_at_6.46.15_PM.png

 

Custom metrics best practices:

  • For querying purposes, we encourage you to limit the number of tags applied to 1,000 tags per metric. Going over this amount will slow down the graphs in your dashboards due to the increase in cardinality.
  • You can check the number of "distinct metrics" in the metric summary page (click a metric name to see the number of distinct metrics associated). If you need a higher custom metric limit, please email support@datadoghq.com and we'll connect you with your Customer Success Manager.
  • Additional information about billing and custom metrics is available here.

 

 

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.
Powered by Zendesk