How to Send Logs to Datadog via External Log Shippers

The best and easiest way to send logs to Datadog is through the Datadog Agent. You can read how to configure the dd-agent to send logs to Datadog here

That said, you can also send logs to Datadog using many common non-Datadog log shippers, like the following:

Rsyslog

FluentD

Forwarding logs from other shippers to the Datadog Log Agent

The Datadog Log Agent can be configured (A) to tail logs from files, and (B) to listen for logs via UDP or TCP over a given port. So whatever your log shipper is, one option is just to have that shipper forward its logs to the Datadog Log Agent; it is often easy to configure this kind of setup, both from the dd-agent side, and from your log shipper. With this approach, you don't need to add your Datadog API key, hostname, or source values in your log shipper's configurations, since that will be handled by the Datadog Log Agent. 

This approach can be especially useful for sending to Datadog logs that have heightened permission requirements. The dd-agent does not run as root (and as a best practice we do not encourage running it as root), so that can block the Datadog Logs Agent from tailing some log files directly, such as /var/log/syslog. If you do not want to modify the permissions on these files or the access that you give to the dd-agent user, many of these open source log shippers do run as root, and can be used to forward logs to the Datadog Logs Agent over UDP / TCP. 

Rsyslog

1. (Optional)Activate Rsyslog file monitoring module

If you want to watch/monitor specific log files, then you have to activate the imfile module by adding this to  your rsyslog.conf:

Rsyslog Version <8

$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/spool/rsyslog

Rsyslog Version >= 8

module(load="imfile" PollingInterval="10") #needs to be done just once

2. Create a /etc/rsyslog.d/datadog.conf file

3. (Optional) Set the file to monitor

Add the following in /etc/rsyslog.d/datadog.conf

Rsyslog Version <8

# Input for FILE1
$InputFileName /<path_to_file1>
$InputFileTag <app_name_file1>
$InputFileStateFile <unique_file_id1>
$InputFileSeverity info
$InputRunFileMonitor

Rsyslog Version >= 8

#For each file to send
input(type="imfile" ruleset="infiles" Tag="<app_name_file1>" File="<path_to_file1>" StateFile="<unique_file_id1>")

4. Send the logs to your Datadog platform

To send logs directly to your Datadog account from Rsyslog over TCP, we firstly need to to define the format in /etc/rsyslog.d/datadog.conf:

$template DatadogFormat,"YOURAPIKEY <%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% - - - %msg%\n"

Then define the endpoint:
Rsyslog Version <8

*.* @@intake.logs.datadoghq.com:10516;DatadogFormat

Rsyslog Version >= 8

ruleset(name="infiles") {
    action(type="omfwd" target="intake.logs.datadoghq.com" protocol="tcp" port="10516" template="DatadogFormat")
}

This assumes that you have SSL enabled for your Rsyslog--if you do not, then you should use port 10514 instead of 10516

Alternatively, to send logs from Rsyslog to your Datadog Logs Agent, configure your dd-agent to expect logs over UDP/TCP on a port of your choosing, add the following content to the end of your/etc/rsyslog.d/datadog.conf:

$template DatadogFormat,"%msg%\n"
*.* @@localhost:PORT;DatadogFormat  # @@ for TCP, @ for UDP

5. Restart Rsyslog and your new logs will get forwarded directly to your Datadog account.

6. Associate those logs with the host metrics and tags

In order to make sure that in your Datadog account these logs are associated with the metrics and tags from the same host, it is important to set the same HOSTNAME in your rsyslog.conf so that its value matches the hostname of your Datadog metrics.

Please note that if you did not specify any hostname in your configuration file for the metrics, then you do not need to change anything.
If you did specify a custom Hostname for your metric, make sure to replace the %HOSTNAME% value in the format to match the same custom name.

7. Enjoy Datadog Integrations

In order to get the best use out of your logs in Datadog, you need to set the source on your logs. The source can be set directly in the agent if you forward your logs to the Datadog agent.

Otherwise you need a specific format per log source which means you need a specific configuration file per source in /etc/rsyslog.d/

To set the source, use the following format (if you have several sources, please change the name of the format in each file):

$template DatadogFormat,"YOURAPIKEY <%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% - - [metas ddsource=\"mysourcename\"] %msg%\n"

Do not forge to replace mysourcename by the wanted value.

 

FluentD

As long as you can forward your FluentD logs over tcp/udp to a specific port, you can use that approach to forward your FluentD logs to your Datadog agent. But another option is to use the Datadog FluentD plugin to forward the logs directly from FluentD to your Datadog account. 

In order to get the best use out of your logs in Datadog, it is important to have the proper metadata associated with your logs (including hostname and source). For the current version of the Datadog FluentD plugin, you will have to include this metadata in the logs that you're sending to FluentD, using the following format:

{
    "syslog.hostname": "myhostname",
"syslog.appname": "myappname",
"ddsource": "mysourcename" }

 

 

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.
Powered by Zendesk