Skip to content
GridNMS Docs

Recommended Docker Host Configuration

If you’re running Docker on the host where you’ve installed a Log Forwarder, turning on the Docker Logs and Container Stats modules gives you full visibility into every container on that host — logs, CPU, memory, and network, all in GridNMS alongside everything else you monitor. This page walks through the configuration we recommend, and a gotcha worth understanding before you turn it on: a feedback loop that can turn one container’s routine log output into a runaway flood.

Section titled “The recommended setup”

Three things make up full container observability on a Docker host, and only one of them needs anything from you:

  1. Point each container’s logging at the host’s system log. Add this to the service in your docker-compose.yml (or the equivalent docker run flags):

    logging:
      driver: journald
      options:
        tag: "{{.Name}}"

    Do this for the containers whose logs you actually want in GridNMS — this is exactly the snippet in Forwarder Management.

  2. Turn on the Docker Logs and Container Stats modules in a Forwarder Profile applied to that host’s group. Both are off by default — nothing is collected until you turn them on. See Forwarder Management for the full Groups and Profiles walkthrough.

  3. Host performance metrics ship automatically. CPU, memory, and disk for the host itself need no module and no extra configuration — they start flowing the moment the forwarder is approved.

Once that’s in place, you’ll see container logs in Log Search, resource usage and trend sparklines in Container Monitoring, and the host’s own CPU/memory/disk in the forwarder’s Metrics tab under Configure → Log Forwarders.

Watch for a feedback loop in your own logging stack

Section titled “Watch for a feedback loop in your own logging stack”

The Docker Logs module has no way to pick and choose individual containers — it tails the host’s whole system log, and any container logging through journald ends up in it. That’s fine for almost everything you run. It’s a real problem for exactly one kind of container: anything that is itself part of your logging pipeline — a log database, a log shipper, or any service whose job is to receive and store the logs you’re collecting.

Here’s the mechanism. A container like that writes a line to its own output every time it does its job — for example, logging that it just received and stored a batch of entries, or logging an error while doing so. If that container’s own output is also flowing through the same Docker Logs module, that line gets collected, forwarded, and stored too — which is itself a write that container may log about, which gets collected again, and so on. Each pass can produce more log volume than the one before it, and the loop accelerates with nothing external driving it.

This isn’t hypothetical — it’s a real failure mode we’ve hit running our own log pipeline on Docker. A single container caught in this loop generated over six million log entries in a single hour, with volume peaking above two thousand entries per second, and pushed the collector delivering that host’s logs to sustained full CPU load. None of it came from real network or device activity — it was the pipeline’s own routine output feeding back into itself. Storage costs for that log pipeline, over the six days before it was caught, came to more than twenty times what a normal month costs.

Two layers, and why you need both

Section titled “Two layers, and why you need both”

Layer 1 — keep that container off journald. In its own service definition, set:

logging:
  driver: local

local is Docker’s efficient log driver, and keeps docker logs fully usable for that container — it just never writes into the host’s system log, so the Docker Logs module never sees it, and the loop has no way to start. This is the layer that actually stops it, because — as above — there’s no per-container switch inside the module itself; exclusion has to happen at the Docker layer, before the forwarder ever sees the container’s output.

Layer 2 — cap that container’s own log verbosity. If the software running in that container has its own log-level setting, set it to warnings-or-higher rather than the info/debug level many databases and logging tools ship with by default. This doesn’t stop the loop by itself — if journald is ever turned back on for that container by mistake, a chatty info-level log would start the same feedback loop right away. What it does is keep that container’s routine output quiet, so docker logs and any local troubleshooting stay useful instead of being buried in noise, and it shrinks the blast radius of the rare moment layer 1 gets reverted.

Use both. Layer 1 is the fix; layer 2 is what keeps you covered if it’s ever undone.

Recognizing a loop if it happens

Section titled “Recognizing a loop if it happens”
  • Log volume climbing quickly in Log Search or on a forwarder’s Logs (past hour) trend, with no matching increase in real device or network activity.
  • The same message, or a small handful of messages, repeating rapidly — with your own logging or storage service listed as the source.
  • A collector or forwarder pinned at unusually high CPU with nothing else to explain it.

If you see this, exclude the container per layer 1 above, then confirm in Log Search that the volume from that source drops back to a normal, steady rate.

Next steps

Section titled “Next steps”

docs built 2026-09-26 · 195c6d00