Skip to main content
Preview. This part of the console is still being built. What you see here describes the intended behavior and can change before general availability.

Troubleshoot an outpost

Something ran on an outpost and did not end the way you expected, or never started. Start from what the console shows, then read the outpost's logs with the commands in Operate an outpost. Almost every failure is one of the cases below.

It never shows up, or stays at Waiting for first heartbeat​

What you seeLikely causeWhat to do
Nothing appears in Outposts, and the error reads "enrollment refused: this enrollment token was already used", "was cancelled in the console" or "expired (tokens work for one hour)", followed by "create a new one in the console (Scan Center › Outposts › Deploy outpost)"The token no longer works. Each enrolls once, within one hour, and can be cancelled.Create a new token in the Deploy outpost drawer and enroll again.
Nothing appears, and the error reads "enrollment refused: the console does not know this enrollment token (or --console names another console)"Dwaar does not know the token: it was mistyped, or it came from another console.Copy the token again from the drawer, and check --console or OUTPOST_CONSOLE is the address you open in the browser.
Nothing appears, and the error is a connection or certificate errorThe host cannot reach the console.Check the host reaches the console's address over HTTPS, and see Behind a proxy.
It appears, then stays at Waiting for first heartbeatIt enrolled but run is not running, or stopped right after. outpost status says the same: "waiting for its first heartbeat".Start the service or container and read its logs.
The logs say the token is ignored/var/lib/outpost/outpost.json already exists, so the outpost is already enrolled.Nothing, if that is the outpost you meant. To start fresh, revoke the old one in Dwaar, delete the file and enroll with a new token.
On ECS, the stack completed but nothing enrollsThe subnets have no route out, or the stack was created more than an hour after the token.Use subnets behind a NAT gateway, set HttpsProxy, or set AssignPublicIp=true in public subnets. For an expired token, deploy again with a new EnrollmentToken.

The enroll command prints the error. An outpost that enrolls on its first start, as with Compose and ECS, logs it.

Refused on an offline outpost​

The queue shows Refused on with the outpost's name, and the discovery's error reads like outpost unavailable: the outpost <name> is offline (last seen ...). Test connection fails with the same reason.

If the error says "the outpost is stopping; try again when it is back", the outpost is Stopping: it is being restarted or upgraded and lets its running scans finish first. Wait until it is Online, then press Sync now or Test connection again.

For offline, the outpost has not sent a heartbeat for two minutes. Check the host is up, the outpost is running, and it can reach the console. Once it reports again it turns Online and takes work: press Sync now to run the discovery again. Scans waiting for it start on their own.

If the outpost is gone for good, revoke it and move its connections to another one under Where it runs.

A job was not taken or stopped reporting​

ErrorMeaning
the outpost did not take the job within 00:05:00A discovery waited five minutes and the outpost did not take it, usually because it was busy with other discovery and test jobs or stuck. Read its logs.
the outpost stopped reporting 3 timesThe outpost took the job three times and lost it each time, usually because it restarted or lost its connection to Dwaar mid job.
Unreachable: the outpost did not answer within 1m30sTest connection waited 90 seconds for the outpost.
this outpost has no driver for ...The outpost does not support that provider or sign in method. Upgrade it, or see what does not run on an outpost yet.

Image pull failed​

The scan fails with ImagePullFailed before anything starts. The outpost could not get one of the scan's analyzer images.

  • The host cannot reach the registry. Check outbound HTTPS to Dwaar's image registry, listed in Plan an outpost. On a host behind a proxy, configure Docker's own proxy too: Docker pulls the images, not the outpost.
  • The pull token could not be issued. Retry the scan. If every scan on every outpost fails this way at once, contact Dwaar support.
  • The disk is full. Free space on the host. The images take about 16 GB.

Insufficient capacity​

The scan fails at once with InsufficientCapacity and a message such as:

the plan's containers need 30720 MB of memory at their peak (35840 MB of limits in all); this host offers 28511 MB (raise Memory overcommit in Scan Center › Outposts if its limits are known to be generous)

The scan's containers, by their limits, need more memory at once than the whole host offers. Nothing was started. Waiting will not help. The same error names the other two limits when they are the cause: a container that asks for more CPUs than the host has, or a scan whose disk estimate is more than the scan root's free space. For memory, choose one:

  • run the outpost on a host with more memory,
  • raise the outpost's Memory overcommit in its pane, accepting that an analyzer may be stopped if the containers really use that much, or
  • turn off scanner categories you do not need for this connection.

Plan an outpost explains the numbers. A scan that fits the host but not what is free right now is not refused: it waits for a running scan to end, and the outpost takes no other scan meanwhile.

A scan waits and never starts​

The queue says why a scan waits, under its stage, and a running scan that waits shows Waiting instead of a stage.

What the queue saysLikely causeWhat to do
"Waiting for : offline since ..."The outpost is down.Bring it back, or move the connection. Scans fail if they wait much more than an hour.
"Waiting for : stopping"The outpost is draining for a restart or an upgrade.Nothing. It runs once the outpost is back.
"Waiting for : it has not connected yet"The outpost enrolled but run has not started.Start it. See It never shows up.
"Waiting on : 2 of 2 scans running"The outpost already runs its Scan concurrency.Let them finish. If this is the usual state and Free now shows memory to spare, raise Scan concurrency. Otherwise add memory or a second outpost. See Capacity and queueing.
"Waiting on : not enough memory for this scan", and the logs show job.waitingThe scan fits the host, but not beside the scans running now.It starts when one ends. If this is the usual state, the host is too small for the work.
"Waiting: 2 scans are running, the limit is 2", with no outpost namedThe scan runs on Dwaar, not on an outpost, and waits for Dwaar's limit for your tenant.Check the connection's Runs on if you expected an outpost.

Scan concurrency does not change​

You saved a new Scan concurrency or Memory overcommit and the pane shows the old value "set on the host". A flag or environment variable on the host wins: --max-jobs or --memory-overcommit on outpost run, OUTPOST_MAX_JOBS or OUTPOST_MEMORY_OVERCOMMIT, or a number in the ECS stack's MaxJobs. Remove it and restart the outpost. An outpost older than 0.1.1 ignores the console's settings altogether: upgrade it as in Operate an outpost.

An AWS test fails on the outpost​

Test connection on an AWS connection whose Role assumed by is An outpost fails with "the outpost could not assume" the role, followed by which role the outpost runs as. The connection role's trust policy does not name that role, or the external id differs. Put the role the message names, which is also the outpost's AWS identity in its pane, in the trust policy with the connection's external id. If the message says the outpost reported no AWS identity, its host has no AWS credentials: give it an instance profile or, on ECS, the stack's task role, and restart the outpost. See AWS connections the outpost assumes.

An analyzer was refused a host​

The logs show outpost.egress_denied with a task and a host. That container asked for a host that is not on its task's list, and the outpost refused it. Most of the time that is the protection working, not a fault. If the analyzer then failed and the host is one it plainly needs, contact Dwaar support with the log line.

It keeps restarting after a revoke​

Its credential was revoked, and it never works again. An outpost that was running when you revoked it logs claim.failed with the error credential refused, then outpost.stopped, and exits with status 3 (outpost run: credential refused). One that starts already revoked, as it does each time Docker restarts it, logs heartbeat.refused instead, and outpost.reenroll_refused when its environment still holds a spent token. systemd does not restart it. Docker restarts the container every few seconds into the same refusal, so with Compose run docker compose down, or docker rm -f dwaar-outpost for a container you started with docker run. ECS restarts the task whatever the status: delete the stack, or give it a new token as below. systemd restarts it too when the outpost is older than 0.1.2 or its unit lacks RestartPreventExitStatus=3: stop and disable the service, or upgrade as in Operate an outpost.

To run an outpost on that host again, give it a new token. It enrolls as a new outpost (outpost.reenrolled). Its name is --name or OUTPOST_NAME when given, else the token's name, else the old outpost's name, else the host's name.

Deployed withEnroll again
systemdAdd OUTPOST_ENROLLMENT_TOKEN=<enrollment-token> and OUTPOST_NAME=<outpost-name> to /etc/default/outpost, then sudo systemctl restart outpost, which starts it again after the exit. Remove both lines once it is online: the token is spent.
Dockerdocker rm -f dwaar-outpost, then the second docker run command of Deploy an outpost again, with -e OUTPOST_ENROLLMENT_TOKEN=<enrollment-token> -e OUTPOST_NAME=<outpost-name> added before the image. The token is spent once it enrolls.
Docker ComposePut the new token in OUTPOST_ENROLLMENT_TOKEN in .env, keep OUTPOST_NAME as it was, and docker compose up -d.
AWS ECSDeploy the stack again with EnrollmentToken set to the new token and OutpostName as it was.

It went offline after Dwaar was updated​

The logs show heartbeat.failed on every attempt, and the outpost is Offline though the host is fine. If its version is well behind the one the Deploy outpost drawer shows, Dwaar no longer accepts its protocol. Upgrade it as in Operate an outpost.

Behind a proxy or a TLS inspecting proxy​

SymptomFix
Enrollment or heartbeats fail with a certificate errorEnroll again with --ca-file naming your proxy's CA. The outpost trusts it besides the system's roots.
The outpost reaches Dwaar but scans fail to pull imagesConfigure Docker's proxy on the host as well. On ECS, HttpsProxy does this for you.
A provider on your network is sent to the proxyAdd its host to NO_PROXY.
On ECS, everything fails with certificate errorsThe stack cannot take your proxy's CA. See Behind a TLS inspecting proxy.
AWS discovery fails behind a TLS inspecting proxyAWS calls use the proxy from the environment but the system's trust store, not --ca-file. Add the CA to the host's trust store, or let AWS traffic bypass inspection.

Try this next​

Keep this page next to the outpost's logs the first week it runs. Each time something fails, find the row above before you change anything. If a failure fits no row, that is worth telling Dwaar support about.