Customize your dedicated server image at runtime
Overview
AMS Watchdog supports two optional ways to customize your dedicated server image at runtime, without changing your upload process: installing additional apt packages, and running a companion process alongside your dedicated server (DS). Both are enabled by adding a specific file to the root of the folder you upload.
Installing additional apt packages at runtime
Your dedicated server build may depend on shared libraries that are not present in the base AMS Watchdog container, such as graphics or audio libraries. To add these libraries to the container image on startup, you can list additional Debian/Ubuntu (apt) packages that Watchdog should install automatically before your DS starts.
To do this, add a file named apt-packages.txt to the root of the folder you upload. List one package name per line. Blank lines and lines starting with # are ignored:
# Graphics libraries needed by our DS
libgl1-mesa-glx
libvulkan1
# Audio support
libopenal1
Each package name must match the Debian package naming convention (^[a-z0-9][a-z0-9.+\-]*(:[a-z0-9]+)?$), which also allows an optional architecture qualifier such as libexample:amd64. If a package name doesn't match this pattern, or if the package install fails, your DS will not start.
When Watchdog downloads and extracts your image, it checks for apt-packages.txt and, if present, runs apt-get update followed by apt-get install for the listed packages before starting any DS process from that image. Packages are installed from the default Debian/Ubuntu repositories configured in the AMS Watchdog base image. This install happens once per image download — if multiple DS processes are started from the same cached image (for example, in on-demand mode), the packages are only installed the first time.
Running a companion process alongside your dedicated server
Your dedicated server build can optionally include a companion script that Watchdog starts and keeps running alongside your DS — for example, an agent that ships your own telemetry to your own observability backend, independent of AMS's own built-in monitoring.
To do this, add a file named demon.sh to the root of the folder you upload. Watchdog runs it with a shell, so it does not need to be marked executable. Watchdog starts it automatically and restarts it with a backoff delay if it exits. This companion process does not need to implement any handshake or protocol with Watchdog — Watchdog only checks whether it is still running.
This companion process's health has no effect on your dedicated servers — however many times it exits or crashes, your DS processes continue to start and run normally.
This is not currently supported for development fleets.
Available environment variables
Watchdog sets the following environment variables for demon.sh:
| Variable | Description |
|---|---|
DS_LOG_DIR | Directory containing your dedicated servers' log files |
ARTIFACT_COREDUMPS_PATH | Directory containing coredump artifacts, if configured |
WD_ID, WD_REGION, WD_FLEET_ID, WD_NAMESPACE | Identifying information for the current Watchdog instance |
Watchdog also sets PATH and a DS_ID variable inherited from the underlying process runner it shares with your dedicated server. DS_ID's value is not meaningful for demon.sh and should not be relied on for anything.
Ports already in use on the node
AMS runs its own observability collector alongside every Watchdog, for AMS's internal monitoring of the fleet infrastructure. It listens on the same default ports most observability agents use:
| Port | Protocol |
|---|---|
8125 | statsd (UDP) |
4317 | OTLP gRPC |
4318 | OTLP HTTP |
If your companion process's agent also listens on any of these ports by default, it fails to bind. Configure your agent to listen on different ports instead — for example, 18125, 14317, and 14318, as used in the examples below.
If you rebind these ports, also point your dedicated server's own metrics and traces exporter at the same non-default port. Many statsd and OTLP client libraries default to the standard ports above — if your DS keeps exporting to the default port after you move your companion process's receiver off of it, your companion process never receives that traffic, even though it starts up without error.
Example: shipping DS logs, metrics, and traces with an observability agent
A common use for demon.sh is running an agent that tails $DS_LOG_DIR and forwards your dedicated servers' logs, metrics, and traces to your own backend. Below are starting points for three popular agents, each covering logs, metrics, traces, and host-level metrics (CPU, memory, disk, network) for the node — adjust the pipeline/config to fit your own setup.
OpenTelemetry Collector — include the otelcol-contrib binary (not the plain otelcol build — the file_log and statsd receivers below are contrib-only) and a config in your upload:
receivers:
file_log:
include: ["${env:DS_LOG_DIR}/*.log"]
start_at: beginning
statsd:
endpoint: "0.0.0.0:18125"
aggregation_interval: 60s
otlp:
protocols:
grpc:
endpoint: "0.0.0.0:14317"
http:
endpoint: "0.0.0.0:14318"
host_metrics:
collection_interval: 60s
scrapers:
cpu:
memory:
disk:
filesystem:
load:
network:
exporters:
debug:
verbosity: detailed
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 80
spike_limit_percentage: 15
batch:
send_batch_size: 10000
timeout: 30s
service:
pipelines:
logs:
receivers: [file_log]
processors: [memory_limiter, batch]
exporters: [debug]
metrics:
receivers: [statsd, otlp, host_metrics]
processors: [memory_limiter, batch]
exporters: [debug]
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [debug]
#!/bin/sh
export GOMEMLIMIT=128MiB
exec ./otelcol-contrib --config ./otel-config.yaml
Setting GOMEMLIMIT caps the collector's Go runtime memory use. This is worth doing for any long-running Go-based agent, since demon.sh keeps running for as long as Watchdog does, unlike the one-time install that apt-packages.txt triggers.
Grafana Alloy — include the alloy-linux-amd64 binary and a River config in your upload:
local.file_match "ds_logs" {
path_targets = [{"__path__" = sys.env("DS_LOG_DIR") + "/*.log"}]
}
loki.source.file "ds_logs" {
targets = local.file_match.ds_logs.targets
forward_to = [loki.write.default.receiver]
}
loki.write "default" {
endpoint {
url = "https://<your-loki-endpoint>/loki/api/v1/push"
}
}
prometheus.exporter.statsd "ds_metrics" {
listen_udp = "0.0.0.0:18125"
}
prometheus.scrape "ds_metrics" {
targets = prometheus.exporter.statsd.ds_metrics.targets
forward_to = [otelcol.receiver.prometheus.default.receiver]
}
prometheus.exporter.unix "host" {
}
prometheus.scrape "host_metrics" {
targets = prometheus.exporter.unix.host.targets
forward_to = [otelcol.receiver.prometheus.default.receiver]
}
otelcol.receiver.prometheus "default" {
output {
metrics = [otelcol.exporter.debug.default.input]
}
}
otelcol.receiver.otlp "ds_traces" {
grpc {
endpoint = "0.0.0.0:14317"
}
http {
endpoint = "0.0.0.0:14318"
}
output {
traces = [otelcol.exporter.debug.default.input]
}
}
otelcol.exporter.debug "default" {
}
#!/bin/sh
exec ./alloy-linux-amd64 run ./config.alloy --stability.level experimental
This config's debug output for metrics and traces (otelcol.exporter.debug) is an experimental Alloy component, so --stability.level experimental is required for Alloy to start with this config. Alloy's loki.write component has no console/debug equivalent for logs — it always sends to a real Loki-compatible endpoint, so point the url above at an actual Loki instance to see log output.
Vector — include the vector binary and a config in your upload:
data_dir: "./vector-data"
sources:
ds_logs:
type: file
include:
- "${DS_LOG_DIR}/*.log"
read_from: beginning
ds_metrics:
type: statsd
mode: udp
address: "0.0.0.0:18125"
ds_traces:
type: opentelemetry
grpc:
address: "0.0.0.0:14317"
http:
address: "0.0.0.0:14318"
ds_host_metrics:
type: host_metrics
scrape_interval_secs: 60
collectors: [cpu, memory, disk, filesystem, load, network]
sinks:
console:
type: console
inputs: [ds_logs, ds_metrics, ds_traces.traces, ds_host_metrics]
encoding:
codec: json
#!/bin/sh
mkdir -p vector-data
exec ./vector --config ./vector.yaml
Vector's data_dir must exist before it starts — the demon.sh above creates it.