Internal / Gossip API
These endpoints carry inter-node CRDT replication traffic and internal service operations. They are served on the same port as the public OAuth2/OIDC endpoints and protected at the application layer — port-level firewall rules cannot isolate them without also blocking public clients.
Gossip endpoints
| Method | Path | Description |
|---|---|---|
POST | /api/gossip/sync | Accept an incoming CRDT gossip push from a peer node. The body is a CMS SignedData(EnvelopedData) blob (ECDSA P-256 outer signature, ML-KEM-768 inner encryption). The receiver verifies the signature, decrypts, applies admission filters, merges the CRDT, and replies with its own state (delta or full, based on the request_delta_since field in the inbound envelope) in the same CMS format. |
GET | /api/gossip/kem-info | Return this node’s ML-KEM-768 public key (base64url SPKI DER) and node_id. Unauthenticated. Returns 404 Not Found with {"registered": false} when this node has no CRDT entry yet (i.e. the node has not yet completed its first gossip round); callers should treat 404 as “not yet registered”. |
GET | /api/gossip/wrapping-key | Return the cluster AEAD wrapping key sealed to the requester’s ML-KEM-768 public key (SignedData(EnvelopedData) DER, application/pkcs7-mime). The requester identifies itself via the X-Ahdapa-Node-Id header and must have a KEM key already in the CRDT. Unauthenticated at the HTTP level; confidentiality is ensured by the ML-KEM-768 encryption — the blob is useless without the requester’s private key. |
POST | /api/gossip/register-kem | Kerberos-authenticated self-registration of both the ML-KEM-768 public key and the ECDSA P-256 gossip signing public key. An IPA-enrolled peer presents Authorization: Negotiate <kerberos-token> (Kerberos AP-REQ for the local HTTP service) and a JSON body {"node_id":"<hostname>","kem_public_key_der":"<base64url-SPKI>","gossip_signing_pub_key_der":"<base64url-SPKI>"}. All three fields are required; missing or empty fields return 400. The server validates that the authenticated principal is HTTP/<hostname>@<REALM>, that node_id matches <hostname>, and (when gossip.kerberos_realm is set) that the principal’s realm matches. Both keys are stored in the CRDT in a three-case match: insert-fresh, upsert-signing-key-only, or no-op (both already present). Returns 503 when the GSSAPI server credential is unavailable or when the DB persist fails; 401 with WWW-Authenticate: Negotiate when no token is presented or invalid; 400 when required fields are missing; 403 on principal or allowlist rejection; and 200 OK on success. |
GET | /api/gossip/stats | Return runtime gossip statistics for this node. Unauthenticated. Returns a JSON object with node_id, crdt_generation, per-collection live counts under counts, configured and topology-discovered peers, active_signing_kid, kem_enrolled, gossip_signing_enrolled, and a gossip sub-object with started_at, rounds_completed, last_round_at, peer_last_sync (map of peer node_id → last inbound sync unix timestamp), persist_errors, and wrapping_key_pull_errors. See Gossip Protocol — Node statistics endpoint for the full response schema. |
GET | /api/gossip/await-client?client_id=ID&timeout_ms=MS | Long-poll endpoint that blocks until client_id appears in the local CRDT or timeout_ms elapses (default: 30000). Unauthenticated. Uses crdt_changed Notify to detect convergence without polling. Returns 200 OK when the client is found, 408 Request Timeout on timeout. Primarily used by ahdapa-bench converge to measure gossip propagation time. |
Credential exchange (OIDC-to-Kerberos)
| Method | Path | Auth | Description |
|---|---|---|---|
POST | /api/internal/ccache | Authorization: Bearer <access_token> with krb5:ccache scope | Exchange a valid OIDC access token for an exported GSSAPI credential (Kerberos ccache) via S4U2Self constrained delegation. |
This endpoint bridges the OAuth2/OIDC world to Kerberos: a FreeIPA server-side component presents a valid access token and receives an exported credential that can be used to perform Kerberos operations on behalf of the authenticated user.
Security model:
- Localhost restriction: when
server.ccache_localhost_onlyistrue(the default), the endpoint rejects requests from non-local source IPs with403 access_denied. “Local” means loopback addresses or any IP address assigned to a local network interface (enumerated viagetifaddrs, cached for 30 seconds). This prevents accidental exposure when the endpoint is reachable over the network. - Bearer token: the request must carry a valid JWT access token in the
Authorization: Bearerheader. The token must contain asubclaim in Kerberos principal form (user@REALM), aclient_idclaim, anaudclaim that includes theclient_id, and thekrb5:ccachescope. - Proof-of-possession (DPoP / mTLS): when the access token contains a
cnfclaim (RFC 9449 DPoPjktor RFC 8705 mTLSx5t#S256), the endpoint validates the corresponding proof. DPoP proofs are verified against the endpoint URL{issuer}/api/internal/ccachewith methodPOST. mTLS certificate thumbprints are extracted from the TLS handshake or a trusted proxy header. Missing or mismatched proofs return401 invalid_tokenor401 invalid_dpop_proof. - User existence check: before attempting S4U2Self, the endpoint verifies
that the
subprincipal exists in the directory (static users file or FreeIPA LDAP). Unknown users are rejected with403 access_denied. - HBAC enforcement (fail-open): when no HBAC rules have been created,
ccache requests pass through (the
krb5:ccachescope check alone gates access). Once at least one HBAC rule is created, the full evaluation runs: user/group membership, client match, scope grant (must includekrb5:ccache), source network CIDR, MFA requirement (via AMR), and required ACR. In FreeIPA co-deployments, when the built-inallow_allHBAC rule is enabled (the default), HBAC evaluation is skipped entirely for this endpoint – matching FreeIPA’s default open policy. See Identity HBAC Policy – FreeIPA allow_all. - Rate limiting: subject to the same per-IP rate limit as other
authentication endpoints (
server.auth_rate_limit). - Audit logging: every ccache issuance and denial is recorded in the audit
journal with the event type
ccache.issuedorccache.deniedrespectively, including the denial reason. - Credential encryption: the exported credential bytes are encrypted with gssproxy’s master key and are only usable on the same machine. The credential cannot be replayed on a different host.
Request
POST /api/internal/ccache
Authorization: Bearer <access-token>
The request body must be empty (the route enforces a zero-byte body limit).
Response
On success, 200 OK with Content-Type: application/octet-stream. The
response body is a complete MIT Kerberos ccache v4 file (version tag
0x0504). The credential is obtained via S4U2Self and stored into a
temporary FILE: ccache via gss_store_cred_into; the file bytes are read
back and returned as the response body. The consumer can write the bytes
to disk and use them with KRB5CCNAME=FILE:/path, or load them into a
MEMORY: ccache via krb5_cc_resolve + krb5_cc_initialize +
krb5_unmarshal_credentials.
The response includes Cache-Control: no-store, Pragma: no-cache, and
Content-Disposition: attachment.
Error codes
| Status | Error | Condition |
|---|---|---|
400 | invalid_request | sub claim missing or not a valid Kerberos principal (user@REALM), or client_id claim missing, or aud does not contain client_id. |
401 | invalid_token | No Authorization: Bearer header, JWT is invalid/expired/revoked, or cnf proof-of-possession binding failed (DPoP key mismatch or missing mTLS certificate). |
401 | invalid_dpop_proof | A DPoP header was sent but the proof JWT is invalid (bad signature, wrong method/URL, expired). |
401 | use_dpop_nonce | DPoP proof JTI was already used (replay). |
403 | access_denied | Source IP is not local (when ccache_localhost_only = true), HBAC denied the request, or the user does not exist in the directory. |
403 | insufficient_scope | Token does not contain the krb5:ccache scope, or HBAC did not grant that scope. |
429 | slow_down | Rate limit exceeded for the source IP. |
500 | server_error | S4U2Self delegation failed or credential export failed. |
503 | temporarily_unavailable | No GSSAPI credential is available for S4U2Self (server not configured with [gssapi] initiator_principal). |
All error responses include a WWW-Authenticate: Bearer error="<error>" header
and a JSON body {"error": "<error>"}.
Prerequisites
[gssapi] initiator_principalmust be configured so the server can perform S4U2Self constrained delegation.- The HTTP service principal must have
ok_to_auth_as_delegateset so the KDC issues forwardable S4U2Self tickets (required for S4U2Proxy):
Without this flag the KDC rejects the S4U2Proxy step withipa service-mod HTTP/<hostname> --ok-to-auth-as-delegate=trueEVIDENCE_TKT_NOT_FORWARDABLE. - gssproxy must be running on the same host (when
[gssapi] gssproxy = true). - The
krb5:ccachescope must be defined via the admin API (PUT /api/admin/scopes/krb5:ccache) and assigned to the calling client. - At least one HBAC rule must grant the
krb5:ccachescope to the relevant users/groups for the calling client (or FreeIPA’sallow_allrule must be enabled).
Access control
Gossip endpoints are protected by two independent mechanisms at the application layer. A rogue client that can reach the server port gains nothing from these endpoints without the corresponding cryptographic keys.
CMS authentication and encryption (/api/gossip/sync, /api/gossip/wrapping-key)
Every gossip sync payload is a CMS SignedData(EnvelopedData) structure:
- The outer ECDSA P-256 signature is verified against the sender’s pinned
gossip signing key stored in the CRDT. A node whose key is not pinned
receives
401 Unauthorizedregardless of the payload contents. - The inner ML-KEM-768 encryption is addressed to the receiving node’s public key. The encrypted payload is opaque to any party that does not hold the private key.
For /api/gossip/wrapping-key, the requester’s X-Ahdapa-Node-Id must match
a node_id in the admission allowlist (see below); the response is itself an
EnvelopedData blob encrypted to the requester’s ML-KEM-768 key, so the
cluster wrapping key is never transmitted in the clear.
Node admission allowlist (allowed_node_ids)
The [gossip] section of the configuration controls which node_ids are
permitted to exchange CRDT state:
[gossip]
# Static allowlist. Only node_ids listed here may participate.
allowed_node_ids = ["ipa1.example.com", "ipa2.example.com"]
When ipa_topology = true, the allowlist is extended automatically with all
replica hostnames discovered from the IPA replication topology, and updated
every ipa_topology_interval_secs seconds. Entries from the static list and
the topology-derived list are merged; the union fails closed — a node_id absent
from both is denied.
Kerberos authentication (/api/gossip/register-kem)
Self-registration requires a valid Kerberos AP-REQ for the local HTTP
service. Only HTTP/<hostname>@<REALM> service principals are accepted; user
principals are rejected. When gossip.kerberos_realm is set, cross-realm
principals are also rejected. The authenticated hostname is validated against
the submitted node_id and against the allowlist before any key material is
stored.
See Multi-node Cluster and Gossip Protocol for operational details.