# Troubleshooting (agent runbook)

Fixes for the problems people actually hit installing and running self-hosted Agent Relay on EKS and GKE.

Rendered page: https://agentrelay.com/docs/self-hosting/agents/troubleshooting
Markdown endpoint: https://agentrelay.com/docs/self-hosting/agents/troubleshooting.md

---

## First: stop and keep the evidence

Don't uninstall, roll back, delete volumes, retry migrations or rerun with
`--atomic`. Collect read-only evidence first, then fix forward:

```sh
helm -n agent-relay status agent-relay
helm -n agent-relay history agent-relay
kubectl -n agent-relay get jobs,deploy,statefulset,pods
kubectl -n agent-relay get events --sort-by=.lastTimestamp | tail -40
kubectl -n agent-relay logs job/<job name> --all-containers --tail=100
```

Remove any secret from logs before you share them. If nothing below matches,
go to [Support](https://agentrelay.com/docs/self-hosting/agents/support).

> **Agent:** Run only the read-only commands above without asking. Check every output for
> secrets (URLs with passwords, tokens, keys, PEM blocks) and redact before
> showing it. Propose a fix from this page, show the exact command, and ASK THE
> HUMAN before running anything that changes the cluster or cloud account.

## Expected during the install (not a problem)

- **The `search-indexer-bootstrap` Job fails after step 11,** with
  `consumer bootstrap: trusted credential probe unavailable`, and the read API,
  ingest edge and search indexer restart. Its database login doesn't exist until
  step 13 (`rds-setup.sh post` or `cloud-sql-setup.sh post`), which reruns it.
- **The search indexer restarts until the bootstrap Job completes** in step 13,
  with `permission denied for schema sessions`. It recovers by itself.
- **`helm upgrade --install` prints `STATUS: deployed` with a failed migration
  Job.** Helm doesn't wait for Jobs here: always run the migration check.
- **On GKE, the Ingress has no address for 15 to 20 minutes after step 11,**
  and its events may show one `Translation failed ... no BackendConfig for
  service port exists` warning. The load balancer controller read the Services
  before their BackendConfigs; a later sync fixes it by itself. Worry only if
  the address is still empty, or the warning still repeats, after about 20
  minutes (see [step 14 of the GKE guide](https://agentrelay.com/docs/self-hosting/agents/install-gke#14-wait-for-the-certificate-and-check-up-to-60-minutes)).
- **On GKE, `/internal` and other closed paths answer 502.** The chart sends
  them to the `...-cast-edge-deny` Service, which has no endpoints on purpose.
  Without a Cloud Armor policy the load balancer answers 502 there, not 403 or
  404. This is not an outage: the four checks in step 14 tell you whether the
  install is up.

## Common problems

| Symptom | Likely cause and fix |
|---|---|
| A pod shows `CreateContainerConfigError` | A Secret or key is missing. Recheck step 10, or [Secrets](https://agentrelay.com/docs/self-hosting/agents/secrets) |
| The control API exits at startup naming a variable | A Secret value is too short, or a URL isn't a public https origin |
| `ImagePullBackOff` | The nodes can't pull the image: check ECR or Artifact Registry pull access, and that the mirror copied the pinned digest |
| `cloud-sql-setup.sh pre` waits about 10 minutes, then says its Job did not succeed | On a new Google Cloud project the node service account has no roles, so the Job's image pull fails with `403`. Grant it `roles/artifactregistry.reader` on the mirror repository ([step 5 of the GKE guide](https://agentrelay.com/docs/self-hosting/agents/install-gke)), then run `pre` again |
| `billing account ... disabled` from Cloud Storage or Service Networking | Your Application Default Credentials use a quota project without billing: `gcloud auth application-default set-quota-project <project id>` |
| `PERMISSION_DENIED ... resourcemanager.projects.create` | You can't create projects in the organization. Someone with that permission and `billing.user` on the billing account creates the project, then grants you Owner on it |
| S3 or OpenSearch `AccessDenied` | A Pod Identity association is missing, or the namespace or release name differs from Terraform's. Create it, then `kubectl -n agent-relay rollout restart deploy` |
| OpenSearch 403 | The role mapping in step 7 of the EKS guide is missing, or step 7 ran as the wrong role |
| Self-signed or certificate errors to the database | The `rds-ca` ConfigMap or `__RDS_CA_SHA256__` is wrong: rerun `rds-setup.sh pre` before `post`, and compare the hash |
| `PostgresTlsConfigError` on GKE | `cloud-sql-ca` doesn't match the pin in `managed-values.json`: see step 13 of the GKE guide |
| 504 from the load balancer on EKS | A load balancer subnet isn't listed in `values.yaml`'s `networkPolicy` blocks. Add its CIDR and upgrade |
| `Could not resolve host` in the smoke checks | DNS hasn't propagated yet: wait a couple of minutes |
| Sessions show as `Session · <id>` | Titles and summaries aren't on: see [After the install](https://agentrelay.com/docs/self-hosting/agents/after-install#3-turn-on-titles-and-summaries) |
| Output or digest endpoints answer `*_backfill_incomplete` | The coverage Job hasn't run: see [After the install](https://agentrelay.com/docs/self-hosting/agents/after-install#2-establish-output-and-digest-coverage) |
| The agent filter answers `501 agent_filter_not_ready`, or the dashboard says "The session agent index isn't ready yet" | Expected until you [activate it](https://agentrelay.com/docs/self-hosting/agents/after-install#4-optional-the-agent-name-filter); on GKE, [step 16 of the GKE guide](https://agentrelay.com/docs/self-hosting/agents/install-gke#16-turn-on-agent-search-once-after-first-sign-in) |
| Transcript reads answer `501` on GKE for sessions uploaded before an upgrade | Releases 2026.11.1 to 2026.11.5 recorded Cloud Storage sessions as if they were in S3. After upgrading, run [Re-stamp sessions uploaded before 2026.11.6](https://agentrelay.com/docs/self-hosting/agents/install-gke#re-stamp-sessions-uploaded-before-2026116-upgrades-only) once. A `501 turns_canonical_deadline` on a long session means the read deadline is too short: keep `READ_API_CANONICAL_DEADLINE_MS=15000` from the downloadable `values.yaml` |
| The setup key is refused with `409 admin_already_signed_in` | The admin already signed in another way. Sign in with that method; the key is only for the very first sign-in |
| The setup key is expired or lost | Mint a new one with the bootstrap token: step 16 of the EKS guide |

## A migration Job failed

Read its log:

```sh
kubectl -n agent-relay logs job/<job name> --tail=100
```

Fix the cause, delete the failed Job, and run the same `helm upgrade` again,
which recreates it. A Job name includes a hash of its spec, so a changed
setting gives a new Job.

- **`role "<name>" does not exist`:** the database's runtime role has another
  name than the chart expects. Check with your migration owner connection:
  `SELECT rolname FROM pg_roles WHERE rolname NOT LIKE 'pg_%' ORDER BY 1;`
  and share only role names.
- **`42501` with `role check failed bootstrap:...`:** the database roles weren't
  set up. Run `rds-setup.sh pre` (or `cloud-sql-setup.sh pre`) on a new install.
- **A migration checksum mismatch:** the image changed a migration the database
  already applied. Don't edit the migration ledger:
  [escalate](https://agentrelay.com/docs/self-hosting/agents/support) with the old and new image digests,
  the migration number and the Job log.
- **`field is immutable` on a migration Job during an upgrade:** delete the
  completed old Job and run the upgrade again.

## "already belongs to another HoopDb install"

One database instance holds only one Agent Relay install, because one of its
roles is server-wide. Use a separate instance.

## EKS: load balancer, certificate and Pod Identity

- **The Ingress has no hostname, or no healthy targets:** check the
  IngressClass is `alb`, that the AWS Load Balancer Controller is running, and
  that `values.yaml` admits the load balancer subnets' CIDRs.
- **TLS fails, or `https` origins are refused:** the ACM certificate must be
  `ISSUED` in the cluster's region and cover all four hostnames.
- **Requests to `//internal` reach the API:** the Ingress was edited by hand.
  Regenerate it with `eks-alb-ingress.sh` (step 14) rather than editing rules.
- **`CreateRule` AccessDenied on EKS Auto Mode:** custom tags on the load
  balancer. Auto Mode only allows its own tag keys; remove the tags.
- **The sessions bucket is denied, or a locker-init Job failed:** the Pod
  Identity associations were missing when the pods started. Create them,
  restart the pods, and recreate the failed Job.

## GKE: certificate, sockets and private clusters

### TLS resets on every host after adding a host

GKE uses one managed certificate for all hostnames. Adding a hostname replaces
the certificate, and the load balancer serves no valid certificate for any host
until the new one is `Active`, often 10 to 15 minutes. Schedule hostname changes
as maintenance, create the DNS record **before** adding the host, then wait:

```sh
kubectl -n agent-relay get managedcertificate -o wide
```

Don't roll back mid-provisioning: that starts another replacement.

### relay-control exits: signer and storage must be set together

relay-control restarts again and again, and its log ends with
`SELFHOST_DESKTOP_FEED_SIGNER and SELFHOST_DESKTOP_FEED_STORAGE must be set together`.
`desktop_feed_bucket_name` is set in `terraform.tfvars`, so
`managed-values.json` carries the desktop feed's storage settings, but the
feed's signing key isn't in place yet.

Keep the bucket, and take the storage settings out until the key exists. From
the install folder:

```sh
jq 'del(.["relay-control"].extraEnv)' managed-values.json > managed-values.tmp && mv managed-values.tmp managed-values.json
```

Also remove any `SELFHOST_DESKTOP_FEED_*` entries you copied into
`values.yaml`. The service account annotations can stay. Then run step 11's
`helm upgrade` command again. The Terraform in `agent-relay-2026.11.16` has
no setting that does this for you, so use the `jq` line.

When the key is in place, follow
[Signed desktop update feed](https://agentrelay.com/docs/self-hosting/agents/install-gke#signed-desktop-update-feed-optional):
its first item writes `managed-values.json` again with the storage settings.

### Other GKE issues

- **Certificate stuck in `Provisioning` or `FailedNotVisible`:** a hostname
  doesn't resolve to the reserved address yet, or a CAA record blocks
  `pki.goog`.
- **WebSockets on the cast host drop every 30 seconds:** the Relaycast Service
  isn't bound to the cast BackendConfig, so the load balancer keeps its
  30-second default. Keep `relaycast.service.backendConfig` from the GCP profile.
- **`kubectl` can't reach a private control plane:** use the cluster's DNS
  endpoint (`gcloud container clusters get-credentials <cluster> --location <region> --dns-endpoint`),
  or run the install from a machine inside the VPC with `--internal-ip`.
- **gcloud asks you to reauthenticate midway:** run `gcloud auth login` again in
  your own terminal. Never copy access tokens anywhere.
- **The database setup Job is evicted (`low on resource: ephemeral-storage`):**
  the nodes' disks are too small; use 100 GB.
- **Empty Secrets after a vals retry:** delete `secrets.yaml.dec` and run again
  (see [Secrets](https://agentrelay.com/docs/self-hosting/agents/secrets#vals-with-google-secret-manager)).

## The desktop app

- **Sessions don't appear:** check that the app is signed in to your domain,
  that the upload policy (**Settings › Policies**) allows them, and give it a
  minute or two.
- **Sessions inside WSL don't appear:** you need a desktop build from
  2026.11.11 or later.
- **Copilot chats in VS Code don't appear:** they're captured on VS Code 1.143
  or later with the default Copilot harness, not the older "Local" harness.
