Creating Composite Monitors

Query Grammar
 
Monitors are defined using a boolean expression involving 2 or more monitors, currently up to 10 total. An alert state is defined when the expression is matched anytime during the last 5 minutes
 
For example you might have a query like:
  • 1234 && 5678 
This would reference monitors at:

More advanced boolean expressions are supported like:

  • 1234 && (!4567 || 3456)

And can be defined in the app using the Composite Monitors type here:

https://app.datadoghq.com/monitors#create/composite

composite.png

 
For Multi and Simple alerts we will only trigger if a group is in all groups. For example if monitor A has host:X, host:Y and host:Z and monitor B has host:X, only host:X will be included in the final result.
 
 
API Usage
 
In addition to the UI, the API endpoints for creating composite monitors are the same as for other monitor types, available here: http://docs.datadoghq.com/api/#monitors.
 
CREATE:
 
POST /api/v1/monitor?api_key=:api_key&application_key=:app_key
{
    "type" : "composite",
    "query" : "1234 && 5678",
    "name" : "My first composite monitor",
    "message" : "This will trigger when both monitors are triggered!",
}
 
UPDATE:
 
PUT /api/v1/monitor/6789?api_key=:api_key&application_key=:app_key
{
    "type" : "composite",
    "query" : "1234 && 5678",
    "name" : "UPDATED: My first composite monitor",
    "message" : "This will trigger when both monitors are triggered!",
    "options" : {
        "renotify_interval": 5
    }
}
 
DELETE:
 
DELETE /api/v1/monitor/6789?api_key=:api_key&application_key=:app_key
 
Note: The following options will not apply to composite monitors:
  • notify_no_data
  • no_data_timeframe

 

Monitor Groups
 
The multi-alert monitors that a composite monitor evaluates over must currently have matching monitor-groups. For simple and multi-alert monitors, the results from a simple monitor will always be evaluated alongside any multi-alerts. So, a composite monitor that observes two multi-alert monitors will only evaluate on what groups are held in common by both those multi-alert monitors. If the same composite monitor is combined with a simple alert, the result of that simple monitor will be added into the evaluation as well.
 
 
How Composite Monitors Evaluate
 
Datadog monitors can be in any of the following 7 states, and each state has an increasing level of "criticality" as follows:
OK < Skipped < Ignored < No Data < Unknown < Warn < Alert
 
Of these, Composite Monitors consider the last 4 as "alert-worthy" states (No Data < Unknown < Warn < Alert). Logic with && will evaluate as "alert-worthy" if all values in the expression are "alert-worthy." Logic with || will evaluate as "alert-worthy" if any values in the expression are "alert-worthy."  
 
When a Composite Monitor evaluates as "alert-worthy" it inherits the state of the monitor whose "alert-worthy" state has the highest "criticality." So long as the Composite Monitor evaluates as not "alert-worthy," it inherits the state of the monitor whose non-alert-worthy state has the lowest "criticality."
 
 
Example
 
A composite monitor is setup over 123456 && 234567.
  • 123456 = OK, 234567 = OK: Composite Monitor = OK
  • 123456 = OK, 234567 = Alert: Composite Monitor = OK
  • 123456 = Skipped, 234567 = Alert: Composite Monitor = Skipped
  • 123456 = Warning, 234567 = Warning: Composite Monitor = Warning
  • 123456 = No Data, 234567 = Alert: Composite Monitor = Alert
  • 123456 = Warning, 234567 = Alert: Composite Monitor = Alert
  • 123456 = Alert, 234567 = Alert: Composite Monitor = Alert

A composite monitor is setup over 123456 || 234567.

  • 123456 = OK, 234567 = OK: Composite Monitor = OK
  • 123456 = OK, 234567 = Alert: Composite Monitor = Alert
  • 123456 = Skipped, 234567 = Alert: Composite Monitor = Alert
  • 123456 = Warning, 234567 = Warning: Composite Monitor = Warning
  • 123456 = No Data, 234567 = Alert: Composite Monitor = Alert
  • 123456 = Warning, 234567 = Alert: Composite Monitor = Alert
  • 123456 = Alert, 234567 = Alert: Composite Monitor = Alert

A composite monitor is setup over (123456 || 234567) && 345678.

  • 123456 = OK, 234567 = Alert, 345678 = Alert: Composite Monitor = Alert
  • 123456 = Skipped, 234567 = OK, 345678 = Alert: Composite Monitor = OK
  • 123456 = Alert, 234567 = Alert, 345678 = OK: Composite Monitor = OK
  • 123456 = OK, 234567 = No Data, 345678 = Alert: Composite Monitor = Alert
  • 123456 = OK, 234567 = Warning, 345678 = No Data: Composite Monitor = Warning
  • 123456 = OK, 234567 = No Data, 345678 = No Data: Composite Monitor = No Data

 

 
 
 
Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.
Powered by Zendesk