If your network configuration restricts outbound traffic, you can use a proxy to send logs from either the Datadog agent or 3rd party log collectors you utilize to the Datadog logs intake.
Unlike the metrics intake API, which listens on HTTPS 443, the logs intake utilizes TCP (Layer 4) on port 10516 (for TLS and 10514 for plaintext). In this article we're providing an example for proxying logs with NGINX.
agent ---> nginx ---> Datadog
external log shipper ---> nginx ---> Datadog
Here's a basic nginx.conf used to proxy logs to the Datadog intake. In this example nginx also does TLS wrapping ensuring internal plaintext logs are encrypted between your proxy and Datadog's log intake API endpoint:
# TCP Proxy for Datadog logs
listen 10514; #proxy listen port
When using the Datadog agent as the logs collector the agent itself would also need to be instructed to use the newly created proxy instead of establishing a connection directly with the logs intake. This is done with the following options in datadog.yaml:
dev_mode_no_ssl: true line. It's ok for us to employ this parameter, because establishing the SSL/TLS connection is handled by nginx's
proxy_ssl on; option. Do not run with this option if you aren't intending to use a proxy, which can encrypt the connection to the logs intake.
In Windows, NXLogs is also frequently used as a logs collector. The configuration changes there are minimal - changing the host and port in the <Output> block to those of your proxy instead of the Datadog intake. Example:
Exec $raw_event="Datadog_API_KEY "+$raw_event;
In container environments Fluentd has established itself as the log collector of choice, this is what the configuration would look like for Fluentd:
port 10514 above is not necessary as Fluentd will default to port 10514 if
use_ssl false is present. We've only included it for clarity.
While using the Datadog agent as a proxy server for metrics submitted by other agents is supported, using it as a proxy for logs is not. This feature is under development. For that you can use your existing Layer 4 proxies or utilize this nginx configuration.
We've successfully testing these configurations on Windows Server 2012 R2, CentOS 7, Ubuntu 16 with NXLog, Datadog agent ver. 6, and Fluentd
If using other 3rd part log shippers see this article for details on their configuration options.