Host gone stale in the fleet: find out why
The host exists in your fleet view but its data stopped updating. Start with the last-seen timestamp — it tells you which class of failure you have.
What this looks like
Unlike a host that never connected, a stale host reported for a while before its charts flatlined. Something changed in the collector, host, network path, or credentials.
Allow for your collector's export interval and pipeline delay. Treat a host as stale only when last-seen is well beyond that interval and not recovering.
Common root causes
Collector service stopped or crashed
An OS upgrade, out-of-memory kill, or manual service stop can leave Alloy or your OpenTelemetry Collector down while the host still runs.
Host offline or rebuilt
If the machine is down, terminated, or rebuilt without restoring its collector configuration, the fleet entry becomes stale.
Network or firewall egress change
A firewall, security group, or proxy policy that blocks outbound HTTPS to ingest can silence a collector without stopping its process.
API key rotated
If the project key was rotated without updating collector configuration, exports fail authentication from that moment.
Step-by-step diagnosis
Read the dashboard first, then confirm on the host — the last-seen timestamp narrows it fast.
- 1
Check the last-seen timestamp
In the fleet view, note exactly when the host last reported. A gap of a minute or two is normal heartbeat cadence; correlate a longer cutoff time with deploys, network changes, or maintenance in that window.
- 2
Confirm the host itself is up
Ping or SSH into the machine. If the host is down or was terminated, the fleet entry is correct — restore the host or remove the entry deliberately.
- 3
Check and restart your collector
Check the Grafana Alloy or OpenTelemetry Collector service you configured. If it failed, restart it and watch the host's last-seen timestamp.
- 4
Read the journal for the failure pattern
Read your collector service logs. Authentication errors point to credentials; connection timeouts or TLS failures point to the network path.
- 5
Verify egress and re-install if the key changed
Curl the ingest host over HTTPS to confirm egress. If logs show authentication failures, update the collector's project credentials through your secret manager.
Prevent it from recurring
- Scan the fleet view's status column regularly — stale hosts stand out at a glance.
- Route infrastructure events to Slack with a notification rule so silence gets noticed early.
- Include your collector service in post-maintenance checks after OS upgrades and reboots.
- Treat key rotation as a fleet-wide operation: rotate, then re-enroll every host in the same window.
Still stuck?
If the collector runs cleanly, egress works, and credentials are current but last-seen does not advance, check the AllStak status page, then email [email protected] with the hostname, project, and cutoff time.
Frequently asked questions
How long a gap in last-seen is normal?
Use your collector's export interval as the baseline. Investigate when the gap clearly exceeds that interval and keeps growing.
The host was rebuilt from an image — why is it stale?
A rebuilt machine may have lost its collector configuration or retained stale credentials. Restore the reviewed Alloy, OpenTelemetry, or exporter configuration through provisioning.
Can I get alerted when a host stops reporting?
Route infrastructure and server events to Slack or another channel with a notification rule, so fleet problems surface in the channels your team already watches.
Is a stale host different from a decommissioned one?
Yes. A decommissioned host was deliberately removed and its collector was stopped on purpose — that is expected. A stale host should be reporting and is not; that is what this guide diagnoses.
Explore more
Capabilities
By framework
Compare
A fleet you can trust at a glance
Every Linux server in one live view with health, last-seen, metrics, and operational events — next to your errors and logs. Start free.