Service audiences, listeners, health checks and telemetry

Context

A service is called by very different parties: customers over the internet, other services inside the platform, the organization's own people running the business, and the platform itself checking whether the service is alive. Each of these expects a different level of exposure and a different kind of authentication, and most services serve only some of them. Many services are called by other services and nobody else.

Without shared names for these parties, every service decided its exposure on its own. Each protocol a service learned (gRPC, Connect, MCP, plain HTTP) invited a new port, and each port then needed its own container port, Service port, probe, ingress rule and NetworkPolicy, so the deployment grew faster than the service did. Whether a port was reachable from the internet depended on which ingress rules happened to exist, not on a decision anyone wrote down.

Health checks drifted the same way. Some services probed a TCP socket, some an HTTP /healthz, some the gRPC health service. A few moved probes to a management port, where a probe can report healthy while the port that carries traffic is hung, out of capacity or failing TLS. Others checked the database from the liveness probe, so a single database hiccup restarted every pod at the same moment and turned a short blip into an outage. Paths drifted too: some services answer at the root, others under /~/health, so every chart, ingress exclusion and dashboard had to learn each service's spelling.

Telemetry is the last piece. OpenTelemetry is how we observe services: traces, metrics and logs leave the process over OTLP and an OpenTelemetry Collector decides where they go. A per-service Prometheus scrape endpoint is a second metrics path that has to be secured, routed and kept in step with the first. Probes add noise on top: a probe every few seconds per pod produces more spans than real traffic on a quiet service.

Resolution

Audiences

Every caller of a service belongs to exactly one audience. The audience, not the protocol, decides where a request enters the service and what it must prove.

Audience	Who calls	Reaches the service through
Public	Clients outside the organization's network	The edge: an ingress or Gateway API route
Private	Other workloads inside the platform	The Service, never the edge
Internal	The organization's own people	An internal tool that calls Private APIs
Platform	The platform running the service	The pod network or loopback

• Every service MUST document which of Public and Private it serves. A service that does not serve Public MUST NOT be referenced by any edge route. Serving only Private is the expected default; Public is the exception that has to be argued for.
• Every service serves Platform. Internal is never served by the service directly.
• Internal names who is calling, not where a network reaches. Roles such as admin or support are permissions granted inside Internal, not audiences of their own, and "admin" is left free for a customer's own administrators. (audience research).

Public

• Public traffic MUST enter through the edge, which terminates TLS and is the only component exposed to the internet. Rate limiting and request size limits SHOULD be enforced at the edge.
• Every Public request MUST be authenticated with credentials issued by the identity provider, except for operations explicitly designed to be anonymous, which MUST be listed as such in the service's API definition.
• The service MUST NOT trust forwarding headers such as X-Forwarded-For from anything other than the edge.

Private

• Private traffic MUST stay inside the platform network. A caller in another cluster reaches it through a load balancer that is not exposed to the internet or through the service mesh, never through the public edge.
• Every Private request MUST authenticate the calling workload, with mesh mTLS or a workload identity token. Being inside the cluster is not an identity.
• When a Private call acts on behalf of a user, the caller MUST forward that user's identity, and the service MUST authorize against it rather than against the caller's own privileges.
• A forwarded user identity MUST be a signed token that the service validates itself, checking its signature, issuer, audience and expiry, such as the user's access token or one exchanged for it with OAuth 2.0 Token Exchange (RFC 8693). Workload authentication proves which workload is calling, not which user, so a service MUST NOT accept a plain header naming the user.
• A NetworkPolicy SHOULD admit only the workloads that are documented callers.

Internal

• Internal tools, such as dashboards and support consoles, MUST be separate applications. A service MUST NOT host an internal UI or open a port for one; every service that grew one would also grow its own login, CSRF protection, session handling and UI dependencies.
• Operations that only the organization's own people may perform MUST be ordinary Private API operations guarded by authorization, so they share the service's validation, audit trail and telemetry with every other call.
• An internal tool MUST authenticate people through single sign-on with multi-factor authentication, MUST call services with that person's own identity rather than a shared privileged account, and MUST be reachable only through an identity-aware proxy or the private network.
• Every state-changing operation made through an internal tool MUST be audit logged with that person's identity.
• Internal tools receive errors at PRIVATE visibility, the same as any other Private caller, and never INTERNAL detail. A person investigating a failure follows the error's id and trace context into the producing service's telemetry (ADR#0129349218).

Platform

• Platform callers are the kubelet, load balancer health checks, the service mesh's own health and control traffic, and engineers debugging a single pod. They reach the service only through operational paths and the debug port.
• A service mesh that carries requests for other callers is transport, not a caller: each request keeps the audience of the workload that sent it.
• Platform endpoints MUST NOT carry business data and MUST NOT change business state.

Listeners

• A service MUST open one serving port per audience it serves, and no more: a private port, a public port, or both. Everything one audience calls (gRPC, Connect, MCP, plain HTTP) MUST share that audience's port and be told apart by path.
• You MUST NOT add a port for a new protocol. A port exists only to put a trust boundary between audiences, so the network can expose one without the other. gRPC is HTTP/2 requests to /{Service}/{Method}, so it shares a port with other routes the way Google's own APIs serve gRPC and REST/JSON on one host and port (Google Cloud research; port research).
• Serving container ports and Service ports MUST be named after their audience: public or private. Callers, routes and policies then refer to the audience by name instead of to a number.
• Every serving port MUST accept both HTTP/1.1 and HTTP/2. gRPC requires HTTP/2, while health probes and most MCP and browser clients use HTTP/1.1.
• You MAY run a debug port for profiling and debugging. It MUST bind loopback by default, MUST require an explicit opt-in to bind any other interface, and MUST NOT be referenced by a Service, a route or a probe. Its container port MUST be named debug. When it binds anything other than loopback, a NetworkPolicy MUST admit only the callers allowed to debug it, since a pod port is reachable without a Service.
• The debug port carries what looks inside the running process: CPU and heap profiles, async runtime inspectors such as tokio-console, and runtime switches such as a log level. It MUST NOT move to a /-/debug path on a serving port. Heap dumps can hold tokens and user data, profiles cost real CPU and memory, and everyone who reaches a serving port would reach them. On loopback, only someone allowed to kubectl port-forward to the pod can.

Path layout

• gRPC MUST be served at its canonical root paths, /{Service}/{Method}, where {Service} is the fully qualified service name (acme.billing.v1.Invoices, or just Invoices for a proto file with no package), and MUST NOT be mounted under a prefix such as /rpc/. gRPC clients, grpcurl, reflection, gRPC probes, load balancer health checks and service meshes all expect it there, and gRPC servers put it there by default.
• Every other route MUST NOT start with a fully qualified proto service name or with /-/. /mcp is fine; /acme.billing.v1.Invoices is not.

Operational paths

• The /-/ prefix MUST be reserved for operational endpoints on every listener, and application routes MUST NOT live under it. A - segment cannot collide with a username, a resource ID or an API version, and GitLab and Prometheus already use the same prefix, so platform engineers recognize it.
• You MUST NOT use /~/ or other prefixes for operational endpoints. ~ conventionally names a user's home directory in URLs, and a single prefix lets every proxy, route and instrumentation filter match one pattern.
• Operational endpoints MUST be mounted ahead of application routing and authentication, so a broken router or auth backend cannot hide them.
• The grpc.health.v1 methods are the one exception to the prefix. Their paths, such as /grpc.health.v1.Health/Check and /grpc.health.v1.Health/Watch, are fixed by the gRPC protocol and MUST be treated as operational paths wherever this ADR says /-/.

Path	Port	Purpose
/-/readiness	every serving port	Readiness and load balancer health checks
/-/liveness	every serving port	Liveness and startup
/-/startup	every serving port	Startup, only when liveness cannot serve
/-/metrics	private only	Scraping, only where OTLP cannot reach

Health checks

• Every service MUST answer /-/readiness and /-/liveness, whether or not a deployment probes them today. Which probes run is a deployment choice that changes with the platform and the load, and turning one on MUST NOT wait for a code change.

• Every deployment MUST declare a readiness probe. It is the only probe that protects callers: without it, a pod gets traffic before it can serve and keeps getting it while it cannot.

• A deployment SHOULD declare a startup probe when initialization can outlast a few readiness periods, such as loading data or warming caches. The startup probe SHOULD target /-/liveness with a higher failureThreshold, the way the Kubernetes control plane probes its own components: it asks the same question as liveness with a longer budget, and liveness does not run until it passes.

• A service MAY answer /-/startup as well when its startup cannot be judged by /-/liveness, such as a one-time step whose result the process forgets once it is done. It MUST NOT add other probe paths.

• A deployment MAY declare a liveness probe, and SHOULD do so when the process can hang without exiting, such as a deadlocked event loop. A process that crashes is restarted without one, and a liveness probe that checks the wrong thing restarts healthy pods.

• Probes MUST use the operational paths with httpGet. The paths take the names of the Kubernetes probe fields and of the grpc.health.v1 services below, so each check has one name everywhere. HTTP is the one check every platform can run: kubelet, Cloud Run and every load balancer speak it, while GKE Ingress cannot build a gRPC health check and Cloud Run's HTTP probes require HTTP/1.

• A probe path MAY list its individual checks with ?verbose, one check name and its pass or fail per line, so a platform engineer can see why a pod is not ready.

• A deployment whose probed port serves only gRPC over plaintext MAY use Kubernetes grpc probes against grpc.health.v1 instead, naming the readiness and liveness services, and startup where the service answers it. The kubelet gRPC probe uses TLS only behind an alpha feature gate, so a TLS port MUST use httpGet with scheme: HTTPS.

• Kubernetes probes MUST target the private port when the service has one, and the public port otherwise. A load balancer that routes to public then needs its health check declared explicitly, as described under load balancers below.

• A service that serves gRPC or Connect MUST also register the standard grpc.health.v1 service on every serving port, next to its own services and in addition to the /-/ paths, not instead of them. gRPC tooling checks health only through it: clients that stop sending to an unhealthy backend, gRPC-aware proxies such as Envoy, Istio and GKE Gateway, grpcurl and grpc-health-probe. Most gRPC frameworks ship it ready to register, so it costs little, while adding it later means touching every service once a proxy starts relying on it.

• Registering grpc.health.v1 on the public port does not expose it to the internet. Like the /-/ paths, it answers callers that reach the pod directly, such as kubelet and load balancer health checks, while the edge answers /grpc.health.v1.Health/ itself and never forwards it.

• grpc.health.v1 MUST give the same answers as the /-/ paths, so both report one state:

  Service name	Answers as
  readiness	/-/readiness
  liveness	/-/liveness
  startup	/-/startup
  empty name	/-/readiness

  The empty name is what gRPC clients and load balancers ask for by default. startup is reported only by a service that answers /-/startup.

• A service MUST NOT register grpc.health.v1 service names other than those above, such as one per gRPC service or per dependency. Every caller can then check any service the same way, and the state cannot split across names that answer differently. Individual checks belong in ?verbose.

• A probe on a serving port takes the same listener, TLS and concurrency limits as real traffic, so a hung or saturated listener fails it. A probe on the debug port MUST NOT stand in for it.

What each probe may check

Probe	Answers	Failing it	May check
Startup	initialization finished	kills the container	its own initialization
Liveness	process can make progress	restarts the container	its own initialization, then nothing else
Readiness	requests should reach it	removes it from rotation	its connections to required services

• Startup SHOULD be used instead of a long initial delay on liveness. It MUST cover only the service's own initialization; a dependency that is down at boot is a readiness concern, not a reason to kill the container.
• /-/liveness MUST fail until the service's own initialization is done, so it can serve the startup probe. This is safe because liveness does not run until startup passes.
• Liveness MUST NOT depend on a database, broker or any other service, directly or through a shared check. A restart cannot fix a dependency, and every pod would restart at the same moment and reconnect to a dependency that is already struggling.
• Readiness MAY reflect dependencies that every request needs, so traffic avoids a pod that can only answer with errors. It MUST read the state the process already holds, such as whether its connection to the broker is up, and MUST NOT call out to a dependency or another pod on each probe. It MUST NOT include optional dependencies, and SHOULD fail only after that state has been bad for longer than a transient blip, since a shared dependency takes every pod out of rotation together.
• Readiness MUST go not ready as soon as shutdown begins, before in-flight requests drain, while liveness keeps passing until the process exits.
• Liveness SHOULD use a higher failureThreshold than readiness, so a pod is out of rotation for a while before it is killed.
• Health checks MUST NOT require authentication and MUST NOT return anything beyond status.

Telemetry

• You MUST export traces, metrics and logs over OTLP to an OpenTelemetry Collector, configured through the standard OTEL_* environment variables (OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES).
• Metrics MUST leave over OTLP, and a service SHOULD NOT expose a metrics endpoint for scraping. A backend that wants to scrape reads from the Collector's Prometheus exporter, and Prometheus itself can receive OTLP directly, so almost no consumer needs the service to be scrapeable.
• Only when a consumer can reach neither OTLP nor a Collector, such as an installation run by someone else with nothing but a Prometheus server, MAY a service also serve GET /-/metrics in the Prometheus exposition format. It MUST be off unless explicitly enabled, MUST be served only on the private port, MUST expose the same instruments as the OTLP export rather than metrics of its own, and the reason MUST be documented with the service. OTLP export stays on beside it.
• You MUST instrument every protocol on every serving port with the same pipeline, following the OpenTelemetry RPC semantic conventions for gRPC and Connect and the HTTP semantic conventions for everything else, so one dashboard covers every caller.
• Requests to operational paths, both under /-/ and to grpc.health.v1, MUST NOT start spans. Exclude them where the instrumentation is installed rather than filtering in the Collector, so health checks never consume sampling budget or export bandwidth.

Kubernetes

• The Service MUST expose only serving ports, never debug. When a serving port speaks HTTP/2 without TLS, it MUST set appProtocol: kubernetes.io/h2c.

• An edge route MUST reference only the public port. It SHOULD forward the / prefix and leave path routing to the service.

• A / prefix also matches every operational path, so the edge MUST carry a more specific rule for /-/ and /grpc.health.v1.Health/ that answers without forwarding to the service. Both Ingress and Gateway API pick the longest matching prefix, so that rule takes precedence over /: on Gateway API it is an HTTPRoute rule with a PathPrefix match and no backendRefs, and on Ingress it is a Prefix path pointing at a backend that answers 404. Load balancer health checks are unaffected, since they reach the pod directly rather than through the route.

• A load balancer in front of a port that carries gRPC and other routes MUST send both to that one port, and you MUST NOT open a second port to suit it. Where a backend protocol carries both, use one backend: on a Google Cloud Application Load Balancer that is HTTP2 for a TLS port and H2C for a plaintext one, which classic Application Load Balancers do not support. Where it does not, such as an AWS Application Load Balancer, whose HTTP/2 target groups reject HTTP/1.1 clients, point two target groups at the same port: a gRPC one for the gRPC paths and an HTTP/1.1 one for everything else (AWS research; Google Cloud research).

• A load balancer health check MUST target /-/readiness on the port it routes to, except that a target group or backend that speaks only gRPC MUST check grpc.health.v1 with the empty service name and expect SERVING. Where the controller does not read it from the readiness probe, such as GKE Gateway, you MUST declare it explicitly (a HealthCheckPolicy on GKE), since the default check requests a path the service may not answer. GKE Ingress and the Application Gateway Ingress Controller read it from the readiness probe only when that probe targets the port they route to, so a service that has both public and private, whose probes target private, MUST declare the health check for public explicitly: a BackendConfig on GKE Ingress, and the appgw.ingress.kubernetes.io/health-probe-path and health-probe-port annotations on the Application Gateway Ingress Controller (Google Cloud research; Azure research). On AWS, a gRPC health check MUST set its path to /grpc.health.v1.Health/Check and expect status 0, since the defaults, /AWS.ALB/healthcheck and 12 (UNIMPLEMENTED), pass a service that does not answer health at all.

• A NetworkPolicy SHOULD admit the edge to the public port only, and documented callers to the private port only.

  pod
  ├── public port    Public audience: gRPC, Connect, MCP, HTTP
  │                  /-/liveness, /-/readiness, grpc.health.v1
  │                  ← edge (prefix /, never /-/) ← internet
  ├── private port   Private audience: gRPC, Connect, MCP, HTTP
  │                  /-/liveness, /-/readiness, grpc.health.v1
  │                  ← Service ← other workloads, internal tools
  │                  ← kubelet probes
  ├── debug port     profiling and debugging
  │                  loopback by default, never in a Service, route or probe
  └── OTLP out  ──→  OpenTelemetry Collector  ──→  backends

Links

• ADR#1631648331: Rust is the language for new services and programs
• ADR#0129349218: error visibility levels, and where INTERNAL detail goes
• Kubernetes API health endpoints
• Configure liveness, readiness and startup probes
• Service application protocol
• Network policies
• Cloud Run container health checks
• GKE Ingress health checks
• GKE Gateway health check policy
• gRPC health checking protocol
• GitLab health check endpoints
• Prometheus management API
• Liveness probes are dangerous
• OTLP metrics export to Prometheus
• Using Prometheus as an OpenTelemetry backend
• OpenTelemetry SDK environment variables
• OpenTelemetry RPC semantic conventions
• OpenTelemetry HTTP semantic conventions

Provider research

How each provider names exposure levels and runs health checks, verified against its own documentation:

• Google Cloud
• AWS
• Azure
• Kubernetes
• Platforms and frameworks
• Audience names
• Ports