The Shift in Thinking
Build a hard shell (firewall), then trust everything inside. Once past the gate, you're family. Single breach → total compromise.
Trust no one. Every request — even from the next container over — must present cryptographic proof. The network stops being a security boundary.
By the end of this lesson you'll have a local stack where a Python script calls a Go API, but the Go API refuses to answer unless the request arrives with two independent credentials: a client TLS certificate issued by our CA (mTLS) and a freshly-signed, audience-scoped JWT from Keycloak. The API also cross-checks that both credentials name the same identity. Remove any piece and you get 401.
What You'll Build (High Level)
- 01Generate a local Certificate Authority, server certs for Keycloak and the API, and a client cert for the Python client.
- 02Boot Keycloak with a pre-imported realm defining two clients: a resource server and a machine client.
- 03Build a Go API that requires both a valid client cert (mTLS) and a JWT, and cross-checks their identities match.
- 04Write a Python client that presents its cert on every TLS handshake and its token on every HTTP call.
- 05Prove the model works by showing that missing either credential (cert OR token) fails.
Core Concepts
Five ideas underpin everything that follows. Click each term for the full briefing.
How It Works
Follow the numbered arrows. Every hop crosses a TLS boundary; every request carries its own credentials.
Notice arrow #4: the API reaches out to Keycloak to fetch public keys. It doesn't have them hardcoded. This is why key rotation "just works" — Keycloak rolls the key, the API re-fetches JWKS, life goes on.
The Python client and the API sit on the same Docker network. In a castle-and-moat world, that shared network would be enough. Here it isn't. Arrow #3 must carry a valid JWT or arrow #3 fails.
Project Structure
Every file we'll walk through in this tutorial, in its place on disk:
zero-trust-demo/
├── docker-compose.yml # Orchestrates all 4 services
├── certs/
│ └── generate-certs.sh # Creates local CA + server certs
├── keycloak/
│ └── realm-export.json # Realm + clients, imported at boot
├── api/
│ ├── Dockerfile # Multi-stage Go build
│ ├── go.mod # Go module + one dependency
│ └── main.go # ~115 lines; the entire API
└── client/
├── Dockerfile # Thin Python image
├── requirements.txt # Just `requests`
└── client.py # The callerMinting a Local CA
Before we can do TLS, we need certificates — three kinds, actually. Server certs for Keycloak and the API (so clients can verify them). And a client cert for the Python client (so the API can verify it, via mTLS). In production you'd get server certs from Let's Encrypt and manage client certs through your internal CA. For local dev we mint all of them ourselves.
Because inside Docker, services reach each other by container name (keycloak, api), not by a public hostname. Let's Encrypt won't issue a cert for the name keycloak. So we become our own CA, issue certs with Subject Alternative Names matching the container names, and trust our own CA inside every container.
They're structurally identical — both are X.509 certs signed by our CA — but they're tagged with different extendedKeyUsage values: serverAuth for server certs, clientAuth for the client cert. Strict TLS validators refuse to let a cert marked serverAuth be presented as a client cert, and vice versa. Same material, different role.
1#!/bin/bash2set -euo pipefail34# ─── Generate a self-signed root CA ───────────────────────5openssl genrsa -out ca.key 40966openssl req -x509 -new -nodes -key ca.key -sha256 \7 -days 365 -out ca.crt -subj "/CN=ZeroTrustLocalCA"89# ─── SERVER cert for Keycloak (tagged serverAuth) ─────────10cat > keycloak.ext <<EOF11basicConstraints=CA:FALSE12extendedKeyUsage=serverAuth13subjectAltName = @alt_names14[alt_names]15DNS.1 = keycloak16DNS.2 = localhost17EOF18openssl genrsa -out keycloak.key 204819openssl req -new -key keycloak.key -out keycloak.csr \20 -subj "/CN=keycloak"21openssl x509 -req -in keycloak.csr -CA ca.crt -CAkey ca.key \22 -CAcreateserial -out keycloak.crt -days 365 \23 -sha256 -extfile keycloak.ext2425# ─── SERVER cert for Go API (same process, different name)─26# ...omitted for brevity: generates api.key + api.crt2728# ─── CLIENT cert for Python (tagged clientAuth) ──────────29cat > client.ext <<EOF30basicConstraints=CA:FALSE31extendedKeyUsage=clientAuth32subjectAltName = @alt_names33[alt_names]34DNS.1 = python-client35EOF36openssl genrsa -out client.key 204837openssl req -new -key client.key -out client.csr \38 -subj "/CN=python-client" # CN must match JWT azp39openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key \40 -CAcreateserial -out client.crt -days 365 \41 -sha256 -extfile client.extLine by Line
set -euo pipefail — bash strict mode. Fail on any error, unset variable, or mid-pipe failure.req -x509 creates a self-signed certificate. A real CA is the trust root — nothing signs it.extendedKeyUsage=serverAuth — this cert is only valid for acting as a TLS server. Can't be repurposed as a client cert.DNS.1 = keycloak is what makes https://keycloak:8443 work inside Docker.keycloak.crt. Any container that trusts ca.crt will now accept this cert.extendedKeyUsage=clientAuth — the critical difference. This cert identifies a caller, not a server. The Go API will only accept certs with this EKU during its mTLS handshake.python-client because the Go API's middleware cross-checks cert.Subject.CommonName against the JWT's azp claim. They must agree or the request is rejected.client.crt. The Python container will mount this alongside its client.key and present both during TLS handshakes with the API.In production you typically use separate CAs for server and client certs. If an attacker compromises the client-cert CA, they can't forge server certs, and vice versa. Limiting blast radius is always worth the extra moving parts.
Orchestration
One file describes four services, their wiring, their volumes, and their startup order. This is the blueprint of the lab.
1services:2 postgres:3 image: postgres:16-alpine4 environment:5 POSTGRES_DB: keycloak6 POSTGRES_USER: keycloak7 POSTGRES_PASSWORD: keycloak_pw8 healthcheck:9 test: ["CMD-SHELL", "pg_isready -U keycloak"]10 networks: [ztnet]1112 keycloak:13 image: quay.io/keycloak/keycloak:24.014 command: start --import-realm --health-enabled=true15 environment:16 KC_DB: postgres17 KC_DB_URL: jdbc:postgresql://postgres:5432/keycloak18 KC_HTTPS_CERTIFICATE_FILE: /etc/x509/keycloak.crt19 KC_HTTPS_CERTIFICATE_KEY_FILE: /etc/x509/keycloak.key20 KC_HTTP_ENABLED: "false"21 KEYCLOAK_ADMIN: admin22 KEYCLOAK_ADMIN_PASSWORD: admin23 ports: ["8443:8443"]24 volumes:25 - ./certs:/etc/x509:ro26 - ./keycloak/realm-export.json:/opt/keycloak/data/import/realm-export.json:ro27 depends_on:28 postgres: { condition: service_healthy }29 healthcheck: # probes /health/ready on mgmt port 900030 test: ["CMD-SHELL", "..."]31 networks: [ztnet]3233 api:34 build: ./api35 environment:36 OIDC_ISSUER: https://keycloak:8443/realms/zerotrust37 OIDC_AUDIENCE: zerotrust-api38 volumes: ["./certs:/certs:ro"]39 depends_on:40 keycloak: { condition: service_healthy }41 networks: [ztnet]4243 client:44 build: ./client45 environment:46 KEYCLOAK_URL: https://keycloak:844347 API_URL: https://api:844448 CLIENT_ID: python-client49 CLIENT_SECRET: change-me-in-realm-export50 CA_CERT: /certs/ca.crt51 # mTLS: client cert/key presented to the Go API52 CLIENT_CERT: /certs/client.crt53 CLIENT_KEY: /certs/client.key54 volumes: ["./certs:/certs:ro"]55 depends_on: [api]56 networks: [ztnet]5758networks:59 ztnet: { driver: bridge }Line by Line
KC_* env vars now.--import-realm tells Keycloak to boot and automatically import any JSON file in /opt/keycloak/data/import/. This is how we ship the tutorial with a pre-configured realm.service_healthy — the API won't start until Keycloak's healthcheck goes green. Prevents the classic "connection refused" race condition.iss claim of every JWT. The API will later check that incoming tokens match this exact string.zerotrust-api will be rejected.https://keycloak:8443 resolves via Docker's embedded DNS. This is why our cert's SAN includes keycloak.CLIENT_CERT and CLIENT_KEY point at the client cert + private key that the Python process will present during TLS handshakes with the API. These files live on the host (generated by the cert script) and are mounted read-only into the container.The Realm
A Keycloak realm is an isolated tenant — users, groups, roles, clients. We ship a pre-configured one so the stack works at first boot.
1{2 "realm": "zerotrust",3 "enabled": true,4 "accessTokenLifespan": 300,5 "clients": [6 {7 "clientId": "zerotrust-api",8 "bearerOnly": true,9 "publicClient": false10 },11 {12 "clientId": "python-client",13 "secret": "change-me-in-realm-export",14 "serviceAccountsEnabled": true,15 "standardFlowEnabled": false,16 "directAccessGrantsEnabled": false,17 "protocolMappers": [18 {19 "name": "audience-zerotrust-api",20 "protocolMapper": "oidc-audience-mapper",21 "config": {22 "included.client.audience": "zerotrust-api",23 "access.token.claim": "true"24 }25 }26 ]27 }28 ]29}Line by Line
zerotrust-api client represents the resource server. bearerOnly: true means "this client never initiates login flows — it only accepts bearer tokens." It's the target of tokens, not an issuer.python-client is the caller. It has a secret it trades for tokens.serviceAccountsEnabled: true flips on the client credentials grant. Keycloak will auto-create a "service account user" attached to this client.aud: "account", which the Go API would reject. This mapper injects zerotrust-api into the aud array so the API's check passes.Forget the audience mapper and you'll see oidc: expected audience "zerotrust-api" in the API logs even though the token is otherwise valid. 90% of first-time Keycloak + go-oidc integrations hit this.
The Resource Server
The heart of the system. ~115 lines of Go that refuse to answer a single question without a signed, unexpired, correctly-addressed JWT.
7A · Imports & Dependencies
1package main23import (4 "context"5 "crypto/tls"6 "crypto/x509"7 "encoding/json"8 "log"9 "net/http"10 "os"11 "strings"12 "time"1314 "github.com/coreos/go-oidc/v3/oidc"15)crypto/tls + crypto/x509 — Go's standard library for TLS. We'll build a custom TLS config that trusts our local CA.7B · The Auth Middleware
This function wraps every protected handler. Every. Single. Request. Runs through here — and now it enforces two credentials, cross-checked.
19func authMiddleware(next http.HandlerFunc) http.HandlerFunc {20 return func(w http.ResponseWriter, r *http.Request) {21 // ── Layer 1: TLS client cert (enforced at handshake; we22 // just inspect its identity here) ───────────────────23 if r.TLS == nil || len(r.TLS.PeerCertificates) == 0 {24 http.Error(w, "mTLS: no client cert", http.StatusUnauthorized)25 return26 }27 clientCN := r.TLS.PeerCertificates[0].Subject.CommonName2829 // ── Layer 2: bearer token ────────────────────────────30 authHeader := r.Header.Get("Authorization")31 if !strings.HasPrefix(authHeader, "Bearer ") {32 http.Error(w, "missing bearer token", http.StatusUnauthorized)33 return34 }35 rawToken := strings.TrimPrefix(authHeader, "Bearer ")3637 token, err := verifier.Verify(r.Context(), rawToken)38 if err != nil {39 log.Printf("token verify failed: %v", err)40 http.Error(w, "invalid token", http.StatusUnauthorized)41 return42 }43 var claims map[string]interface{}44 token.Claims(&claims)4546 // ── Cross-check: cert identity == token identity ────47 tokenAzp, _ := claims["azp"].(string)48 if clientCN != tokenAzp {49 log.Printf("cert/token mismatch: %q vs %q", clientCN, tokenAzp)50 http.Error(w, "identity mismatch", http.StatusUnauthorized)51 return52 }5354 ctx := context.WithValue(r.Context(), "claims", claims)55 next(w, r.WithContext(ctx))56 }57}r.TLS.PeerCertificates[0] to find out which valid client showed up. The existence check is defensive belt-and-suspenders.Authorization: Bearer ... header or 401.verifier.Verify() parses the JWT, fetches JWKS if not cached, checks the RS256 signature, validates iss / aud / exp."invalid token" to the client. Never leak verification details to callers — reconnaissance data for an attacker.python-client) must match the token's azp claim (also python-client). If an attacker steals service A's cert and service B's token, this line catches the mismatch. Without it, the two credentials are just parallel — with it, they're bound.Notice: a request with no cert never reaches this Go code. The rejection happens at the TLS handshake inside Go's standard library (triggered by ClientAuth: RequireAndVerifyClientCert in section 7C). The client sees an SSL error. A request with a cert but no token does reach this code — and gets rejected at line 32. Different layers, different error modes.
7C · Bootstrapping the Verifier & mTLS Listener
This runs once at startup. Two jobs: configure OIDC verification, and configure the TLS listener to require client certs.
70func main() {71 issuer := os.Getenv("OIDC_ISSUER")72 audience := os.Getenv("OIDC_AUDIENCE")7374 caPEM, _ := os.ReadFile(os.Getenv("CA_CERT"))75 caPool := x509.NewCertPool()76 caPool.AppendCertsFromPEM(caPEM)7778 // HTTP client for JWKS fetch — trusts our CA for server-side TLS79 httpClient := &http.Client{80 Transport: &http.Transport{81 TLSClientConfig: &tls.Config{RootCAs: caPool},82 },83 }84 ctx := oidc.ClientContext(context.Background(), httpClient)8586 // Retry OIDC discovery while Keycloak boots (30–60s)87 var provider *oidc.Provider88 for attempt := 1; attempt <= 60; attempt++ {89 provider, err = oidc.NewProvider(ctx, issuer)90 if err == nil { break }91 time.Sleep(3 * time.Second)92 }93 verifier = provider.Verifier(&oidc.Config{ClientID: audience})9495 mux := http.NewServeMux()96 mux.HandleFunc("/api/resource", authMiddleware(protectedHandler))9798 // ══════ mTLS CONFIG — THE NEW PART ══════════════════99 tlsCfg := &tls.Config{100 ClientCAs: caPool, // who we accept client101 // certs from102 ClientAuth: tls.RequireAndVerifyClientCert, // strict: no cert,103 // no connection104 MinVersion: tls.VersionTLS12,105 }106 server := &http.Server{107 Addr: ":8444",108 Handler: mux,109 TLSConfig: tlsCfg,110 }111 log.Fatal(server.ListenAndServeTLS(112 os.Getenv("TLS_CERT"), os.Getenv("TLS_KEY")))113}x509.CertPool. This single pool is used for two purposes below: (1) verifying Keycloak's server cert during JWKS fetch, and (2) verifying the client's cert during mTLS handshakes.go-oidc uses it instead of http.DefaultClient.http.ListenAndServeTLS() directly (which sets up a default TLS config), we build our own tls.Config with two critical fields.ClientCAs: caPool — the set of CAs whose client certs we accept. Reusing caPool because our single local CA signs both server and client certs. In production you'd typically use a separate CA for client certs.tls.RequireAndVerifyClientCert — the strictest of the four ClientAuth modes. No cert = TLS handshake fails = request never reaches our Go code. Weaker options like VerifyClientCertIfGiven would let cert-less clients through to the HTTP layer.MinVersion: tls.VersionTLS12 — refuse to negotiate older, weaker protocols. TLS 1.0 and 1.1 are deprecated and known-broken.http.ListenAndServeTLS() anymore because it doesn't take a tls.Config. Instead we construct an http.Server manually with our config attached, then call its ListenAndServeTLS method.Multi-Stage Build
A textbook Go container: compile in one stage, copy the binary to a tiny runtime image. The final image is ~10MB.
1FROM golang:1.22-alpine AS build2WORKDIR /src3COPY go.mod ./4RUN go mod download || true5COPY . .6RUN go mod tidy && CGO_ENABLED=0 go build -o /api .78FROM alpine:3.199RUN apk add --no-cache ca-certificates10COPY --from=build /api /api11ENTRYPOINT ["/api"]AS build names this stage so we can reference it later. Multi-stage build pattern: the final image never sees the Go toolchain.go.mod first, then download deps. Docker caches layer-by-layer, so if we edit main.go but not go.mod, this layer is reused — rebuilds are fast.CGO_ENABLED=0 produces a static binary that runs on Alpine (which uses musl libc, not glibc). Without this flag, you get cryptic "not found" errors on start.ca-certificates is here so if the API ever needed to reach a public TLS endpoint, it could. For our JWKS fetch we use our custom CA pool, so this is belt-and-suspenders.The Caller
A service account impersonator. Gets a token, calls the API, proves that unauthenticated requests fail.
1import os, sys, time2import requests34KEYCLOAK_URL = os.environ["KEYCLOAK_URL"]5API_URL = os.environ["API_URL"]6CLIENT_ID = os.environ["CLIENT_ID"]7CLIENT_SECRET = os.environ["CLIENT_SECRET"]8CA_CERT = os.environ["CA_CERT"]9CLIENT_CERT = os.environ["CLIENT_CERT"] # mTLS10CLIENT_KEY = os.environ["CLIENT_KEY"] # mTLS1112TOKEN_ENDPOINT = f"{KEYCLOAK_URL}/realms/zerotrust/protocol/openid-connect/token"1314def get_token() -> str:15 # No mTLS to Keycloak — it auths us by client_id/secret16 resp = requests.post(17 TOKEN_ENDPOINT,18 data={19 "grant_type": "client_credentials",20 "client_id": CLIENT_ID,21 "client_secret": CLIENT_SECRET,22 },23 verify=CA_CERT,24 timeout=10,25 )26 resp.raise_for_status()27 return resp.json()["access_token"]2829def call_api(token: str) -> dict:30 # mTLS to the Go API: cert + key presented at handshake,31 # bearer token presented in Authorization header.32 resp = requests.get(33 f"{API_URL}/api/resource",34 headers={"Authorization": f"Bearer {token}"},35 verify=CA_CERT,36 cert=(CLIENT_CERT, CLIENT_KEY), # ← mTLS37 timeout=10,38 )39 resp.raise_for_status()40 return resp.json()4142if __name__ == "__main__":43 token = get_token()44 print(call_api(token))4546 # Negative 1: no cert → TLS handshake fails (SSLError)47 try:48 requests.get(f"{API_URL}/api/resource",49 headers={"Authorization": f"Bearer {token}"},50 verify=CA_CERT, timeout=5) # no cert=51 except requests.exceptions.SSLError:52 print("✓ rejected at TLS handshake")5354 # Negative 2: cert but no token → 401 at middleware55 bad = requests.get(f"{API_URL}/api/resource",56 verify=CA_CERT, cert=(CLIENT_CERT, CLIENT_KEY))57 print("cert, no token →", bad.status_code)client_id + client_secret (what OAuth2 calls "client_secret_basic/post"). Requiring a cert on top would need Keycloak's mTLS-bound tokens feature, a deeper topic. For this tutorial, Keycloak uses one auth method, the Go API uses two.verify=CA_CERT — validates Keycloak's server cert against our CA. Without this, requests would use the system CA store (which doesn't know our local CA) and connection would fail.cert=(CLIENT_CERT, CLIENT_KEY) — this tuple is what makes the call mTLS. requests passes it to urllib3, which passes it to Python's ssl module, which presents the cert during the TLS handshake. Without this parameter, the server's RequireAndVerifyClientCert would refuse the connection.cert=. The TLS handshake fails, and Python raises SSLError. The request never reaches the HTTP layer — the Go API never even sees an HTTP request.401. Two different failure modes, two different layers.requests?
It's the de facto HTTP client for Python. Under the hood it wraps urllib3, which wraps Python's ssl module. The verify= and cert= parameters plumb straight through to OpenSSL's certificate chain validation and client-cert presentation. There's no magic — just well-layered libraries.
Run & Observe
Generate local certificates (one time):
cd certs && chmod +x generate-certs.sh && ./generate-certs.sh && cd ..Build images and start the stack:
docker compose up --buildFirst run takes 60–90s while Keycloak imports the realm. The API will log waiting for keycloak OIDC discovery until Keycloak is ready.
You should see the happy path succeed, then three different failure modes:
# expected output:
Got token: eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIi...
API said: {
'message': 'Zero Trust says hello (mTLS + JWT verified)',
'token_azp': 'python-client',
'tls_client_cn': 'python-client',
...
}
[negative] calling with token but NO client cert...
✓ TLS handshake rejected (expected): SSLError
[negative] calling with cert but NO bearer token...
status: 401 (expect 401)
body: missing bearer token
[negative] calling with cert and FORGED token...
status: 401 (expect 401)Notice that token_azp and tls_client_cn are both python-client in the success response — seeing them match in the output is visible proof that the cert/token cross-check is working.
The Python client fetched a JWT from Keycloak over TLS (authenticating with its client_id/secret), then opened an mTLS connection to the Go API, presenting both its client cert and the JWT. The API verified the cert at the TLS handshake, verified the JWT at the HTTP layer, then confirmed both credentials named the same identity. Then three negative tests proved each layer independently rejects bad requests: no cert = TLS handshake fails; no token = 401 from middleware; bad token = 401 from signature verifier. Defense in depth in ~150 lines of code.
Testing the System
The interesting tests aren't the happy path — they're the failure modes. Each one proves a specific attacker question is answered correctly.
Manual curl test
TOKEN=$(curl -s --cacert certs/ca.crt \
-X POST https://localhost:8443/realms/zerotrust/protocol/openid-connect/token \
-d "grant_type=client_credentials" \
-d "client_id=python-client" \
-d "client_secret=change-me-in-realm-export" \
| python3 -c "import sys,json;print(json.load(sys.stdin)['access_token'])")
curl --cacert certs/ca.crt -H "Authorization: Bearer $TOKEN" \
https://localhost:8444/api/resourceDecode the JWT
Paste the token into jwt.io, or locally:
echo "$TOKEN" | cut -d. -f2 | base64 -d 2>/dev/null | python3 -m json.toolYou'll see claims like iss, aud, exp, azp, sub. The API middleware is checking iss, aud, exp on every request.
Attacker scenarios
Each row tests a specific security property. The column on the right tells you which layer caught the attack — useful when debugging your own stacks.
- No client cert → TLS handshake fails (
SSLError). [TLS LAYER] - Wrong CA's client cert → TLS handshake fails. [TLS LAYER]
- Cert but no bearer token → 401 from middleware. [HTTP LAYER]
- Cert + forged/malformed token → 401 (signature check). [HTTP LAYER]
- Cert + tampered token payload → 401 (signature no longer matches).
- Cert + expired token (wait past
exp) → 401 (claim validation). - Cert + token for wrong audience → 401 (
audcheck). - Cert for A + token for B → 401 (cert/token identity mismatch). [CROSS-CHECK]
- Plain HTTP → connection refused (API only binds TLS).
- Skip
--cacert→ client-side TLS failure (client rejects server).
Testing mTLS with curl
To replicate the happy path from your host terminal:
curl --cacert certs/ca.crt \
--cert certs/client.crt \
--key certs/client.key \
-H "Authorization: Bearer $TOKEN" \
https://localhost:8444/api/resourceDrop --cert / --key and watch curl report alert certificate required. That's the TLS handshake rejection in action.
Glossary
extendedKeyUsage=clientAuth. Identifies a caller, not a server.RequireAndVerifyClientCert.