Proxy for logs collection with NGINX

If your network configuration restricts outbound traffic, you can use a proxy to send logs from either the Datadog agent or 3rd party log collectors you utilize to the Datadog logs intake.

Unlike the metrics intake API, which listens on HTTPS 443, the logs intake utilizes TCP (Layer 4) on port 10516 (for TLS and 10514 for plaintext). In this article we're providing an example for proxying logs with NGINX.

agent ---> nginx ---> Datadog

or

external log shipper ---> nginx ---> Datadog

Here's a basic nginx.conf used to proxy logs to the Datadog intake. In this example nginx also does TLS wrapping ensuring internal plaintext logs are encrypted between your proxy and Datadog's log intake API endpoint:

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

include /usr/share/nginx/modules/*.conf;
events {
    worker_connections 1024;
}
# TCP Proxy for Datadog logs
stream {
    server {
        listen 10514; #proxy listen port
        proxy_ssl on;
        proxy_pass intake.logs.datadoghq.com:10516;
    }
}

When using the Datadog agent as the logs collector the agent itself would also need to be instructed to use the newly created proxy instead of establishing a connection directly with the logs intake. This is done with the following options in datadog.yaml:

logs_config:
  dd_url: myProxyServer.myDomain
  dd_port: 10516
  dev_mode_no_ssl: true

Notice the dev_mode_no_ssl: true line. It's ok for us to employ this parameter, because establishing the SSL/TLS connection is handled by nginx's proxy_ssl on; option. Do not run with this option if you aren't intending to use a proxy, which can encrypt the connection to the logs intake.

In Windows, NXLogs is also frequently used as a logs collector. The configuration changes there are minimal - changing the host and port in the <Output> block to those of your proxy instead of the Datadog intake. Example:

<Output out>
  Module om_tcp
  Host myProxyServer.myDomain
  Port 10514
  Exec to_syslog_ietf();
  Exec $raw_event="Datadog_API_KEY "+$raw_event;
</Output>

In container environments Fluentd has established itself as the log collector of choice, this is what the configuration would look like for Fluentd:

<match datadog.**>
  @type datadog
  @id awesome_agent
  api_key Datadog_API_KEY
  host myProxyServer.myDomain
  use_ssl false
  #port 10514
</match>

The option port 10514 above is not necessary as Fluentd will default to port 10514 if use_ssl false is present. We've only included it for clarity.

While using the Datadog agent as a proxy server for metrics submitted by other agents is supported, using it as a proxy for logs is not. This feature is under development. For that you can use your existing Layer 4 proxies or utilize this nginx configuration.

We've successfully testing these configurations on Windows Server 2012 R2, CentOS 7, Ubuntu 16 with NXLog, Datadog agent ver. 6, and Fluentd

If using other 3rd part log shippers see this article for details on their configuration options.