Observability
Your OrderInventory service answers on orders.inventory.check, and the
discovery page lets you enumerate it and target a
single instance by ID. So far you can find the service, and this page shows you
how it's doing.
The framework keeps a running tally for every endpoint (how many requests it's handled, how many failed, how long each took) without a single line of metrics code on your part. This page reads that tally, then shows the one way a handler influences it: by returning a service error.
There are two new ideas here: per-endpoint stats and the service error that the stats record.
Per-endpoint stats the framework keeps for you
Every endpoint you add carries a counter set that the framework updates on each
request, under the service lock, before your handler ever sees the next one. You
don't register the counter set, increment it, or flush it yourself. It's part of
what AddEndpoint gives you.
Five fields matter for day-to-day work:
num_requests: total requests this endpoint has handlednum_errors: how many of those ended in a service errorlast_error: the description string of the most recent errorprocessing_time: total handler time across all requests, in nanosecondsaverage_processing_time:processing_timedivided bynum_requests, in nanoseconds
You read them through the third discovery verb, STATS, on the $SRV
prefix. The same three address levels from the discovery page apply:
$SRV.STATS hits every service, $SRV.STATS.OrderInventory hits every instance
of one service, and $SRV.STATS.OrderInventory.<id> hits one instance.
Send a few requests to orders.inventory.check, then read the accumulated
stats back:
- CLI
#!/bin/bash
# Read the per-endpoint stats the framework keeps for OrderInventory.
#
# First send a few real requests so the counters have something to show.
# Each request carries the canonical Acme order payload to the check
# endpoint on orders.inventory.check.
nats request orders.inventory.check \
'{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}'
nats request orders.inventory.check \
'{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}'
nats request orders.inventory.check \
'{"order_id":"ord_8w2k","customer":"acme-co","total_cents":4200,"ts":"2026-05-22T10:14:22Z"}'
# Now read the accumulated stats. This is the STATS discovery verb on the
# $SRV prefix, scoped to the OrderInventory service. Every instance replies
# with one stats_response listing each endpoint and its counters:
# num_requests, num_errors, last_error, processing_time, average_processing_time.
nats service stats OrderInventory
# Expected (one row per instance): the check endpoint shows
# Requests: 3 Errors: 0 and a non-zero average processing time.
The response is one stats_response per instance, and each lists its endpoints
with the five counters above. After three successful check requests you'd
see num_requests: 3, num_errors: 0, and a non-zero average_processing_time
that reflects how long your handler ran.
The CLI has a shortcut that does the STATS query and formats the table for you:
nats service stats OrderInventory
╭────────────────────────────────────────────────────────────────────────╮
│ OrderInventory Statistics │
├──────────────────────┬──────────┬──────────┬─────────┬─────────────────┤
│ ID │ Endpoint │ Requests │ Errors │ Avg Time │
├──────────────────────┼──────────┼──────────┼─────────┼─────────────────┤
│ NCXY...A9 │ check │ 3 │ 0 │ 412µs │
╰──────────────────────┴──────────┴──────────┴─────────┴─────────────────╯
The counters tracked here are per endpoint, not per service: a service with
check and a second endpoint keeps a separate row for each. That's the level
where you reason about load, telling you which handler is busy and which one is
slow.
Reference documents the full set of fields in the STATS
response, including the per-endpoint queue_group and an optional custom
data blob. We only need the five counters here.
A service error increments the error count
Of those five fields, four move on their own. Only num_errors and
last_error depend on you: they move when a handler returns a service error
instead of a normal response.
A service error is a response that carries two headers: Nats-Service-Error
holds a human-readable description, and Nats-Service-Error-Code holds a short
code string like "400". You don't set those headers by hand. The handler
calls req.Error(code, description, data), and the framework attaches both
headers, sends the response, and bumps num_errors and last_error for that
endpoint in the same step.
Here's the check handler returning a service error when the order total
isn't a positive amount, followed by the stats showing the error recorded:
- CLI
#!/bin/bash
# Make the check handler return a service error, and watch num_errors move.
#
# The check handler rejects an order whose total is not a positive amount by
# calling req.Error("400", ...). The framework attaches the Nats-Service-Error
# and Nats-Service-Error-Code headers, sends the reply, and bumps num_errors
# and last_error for the endpoint.
#
# Send one bad request (total_cents is 0). Use --raw and inspect headers so
# you can see the error response the handler returned.
nats request orders.inventory.check \
'{"order_id":"ord_8w2k","customer":"acme-co","total_cents":0,"ts":"2026-05-22T10:14:22Z"}' \
--raw
# Expected: the reply carries headers like
# Nats-Service-Error-Code: 400
# Nats-Service-Error: order total must be positive
# Now read the stats. The errored request is recorded: num_errors is 1 and
# last_error holds the description string the handler passed to req.Error.
nats service stats OrderInventory
Watch the stats accumulate as requests flow, and the one errored request bump the error counter while the rest tick the request counter:
A service error is still a delivered reply, not a transport failure. The request reached the handler, the handler chose to answer with an error, and that answer came back over the same reply subject as any success. This is different from no-responders, where nothing is listening at all. That case belongs to request-reply itself and is covered in request-reply.
When to return a service error is your call. Bad input is one reason to call
req.Error, a downstream dependency that's unavailable is another, and a
business rule that rejects the order is a third; calling req.Error means the
failure is both reported to the caller and counted in the stats. A handler that
responds normally instead of reporting the failure leaves num_errors at zero
and hides the problem.
Pitfalls
Two traps catch people the first time they rely on service stats. Both have the same cause: a service error looks like a normal reply unless you check for it.
A service error arrives as headers, not as a failed call. The framework
sends the error response over the reply subject like any success, so the
caller's Request returns a message with no transport error. If the caller
only checks "did I get a reply," every service error reads as a success, and
the num_errors you can see in stats is invisible to the code making the
request. Do check Nats-Service-Error-Code on every response; do not treat a
returned message as proof the request succeeded.
Handling it takes one header read on the reply. The CLI shows the headers
directly; in code you read Nats-Service-Error-Code and branch on whether
it's set. Send a request the handler rejects, inspect the headers on the reply,
and confirm the error in the stats:
- CLI
#!/bin/bash
# Make the check handler return a service error, and watch num_errors move.
#
# The check handler rejects an order whose total is not a positive amount by
# calling req.Error("400", ...). The framework attaches the Nats-Service-Error
# and Nats-Service-Error-Code headers, sends the reply, and bumps num_errors
# and last_error for the endpoint.
#
# Send one bad request (total_cents is 0). Use --raw and inspect headers so
# you can see the error response the handler returned.
nats request orders.inventory.check \
'{"order_id":"ord_8w2k","customer":"acme-co","total_cents":0,"ts":"2026-05-22T10:14:22Z"}' \
--raw
# Expected: the reply carries headers like
# Nats-Service-Error-Code: 400
# Nats-Service-Error: order total must be positive
# Now read the stats. The errored request is recorded: num_errors is 1 and
# last_error holds the description string the handler passed to req.Error.
nats service stats OrderInventory
Stats are per instance, and Reset() zeroes them. Each running instance
keeps its own counters, so $SRV.STATS.OrderInventory returns one
stats_response per instance and you must sum num_requests across IDs
yourself to get a service-wide total; the server keeps no aggregate for you.
And a call to Reset() on an instance sets its counters back to zero
and resets the started timestamp, so a dashboard that assumes
monotonically increasing counters will see a drop. Do aggregate across IDs in
the reader; do not assume the numbers only ever grow.
For richer observability, see Monitoring: a custom
StatsHandler that adds your own data blob, or the server-side
service-latency advisories that measure round-trip time from outside the
service. Those are separate mechanisms layered on top of the counters this
page reads.
Where you are
OrderInventory now reports on itself. You can read its per-endpoint stats
through $SRV.STATS or the nats service stats shortcut, you can see
num_requests and average_processing_time move as traffic flows, and you can
make a handler return a service error so num_errors and last_error record
the failure. You also know to read Nats-Service-Error-Code on the caller side
so an error never passes for a success.
What's next
One service answering one request at a time is a single point of contention.
The last mechanism is scaling: run more instances of OrderInventory and
let the default queue group "q" spread requests across them, with no
coordinator and no config change.
Continue to Scaling.
See also
- Discovery — the PING and INFO verbs that sit
alongside STATS on the
$SRVprefix. - Monitoring — custom stats handlers and server-side service-latency advisories.
- Reference — the full STATS response schema and every field it carries.