Kubernetes Deployment

Overview

Kubernetes is the production-grade deployment target for PRECINCT. It runs on any conformant Kubernetes cluster and provides the full set of security boundaries that Compose cannot replicate.

• Works on: EKS, GKE, AKS, kind, k3s, Docker Desktop
• Configuration: Kustomize overlays for environment-specific tuning
• Networking: Full zero-trust with default-deny NetworkPolicies in every namespace

Quick Start

Deploy the full stack to a local Kubernetes cluster (Docker Desktop) and validate it with the E2E demo.

cd POC
make k8s-up        # Deploy to local K8s (Docker Desktop)
make k8s-opensearch-up  # Optional: add OpenSearch + Dashboards extension
make demo-k8s      # Run E2E demo
make k8s-down      # Tear down

Namespace Architecture

PRECINCT uses dedicated Kubernetes namespaces to isolate each functional domain. Each namespace has its own NetworkPolicies, resource quotas, and RBAC bindings.

Security Hardening

The Kubernetes deployment enforces defense-in-depth at every layer of the stack, from the network to the container runtime.

• Default-deny NetworkPolicies in every namespace. Traffic is explicitly allowed only for required inter-service communication paths.
• Gateway pod security: non-root execution (UID 65532), read-only root filesystem, all Linux capabilities dropped, RuntimeDefault seccomp profile.
• SPIFFE mTLS for all inter-service communication. No service can communicate without a valid, attested SPIFFE identity.
• Secret-managed certificates and credentials for the OpenSearch extension. TLS/mTLS material and auth credentials are mounted from Kubernetes Secrets only.
• Pod Security Standards enforced per namespace using the Kubernetes built-in admission controller.
• OPA Gatekeeper for runtime admission control, enforcing organizational policies on every API server request.
• Sigstore policy-controller for image signature verification, ensuring only signed and verified images are admitted to the cluster.

Kustomize Overlays

PRECINCT uses four Kustomize overlays to adapt the base manifests for different environments. Each overlay progressively increases security requirements.

Preview and validate overlays before applying them to the cluster.

kustomize build infra/eks/overlays/local/   # Preview local
kustomize build infra/eks/overlays/local-opensearch/  # Preview local + OpenSearch
kustomize build infra/eks/overlays/prod/    # Preview prod
make k8s-validate                            # Validate all overlays

Admission Control

Two independent admission controllers operate on the Kubernetes API server admission chain, each enforcing a different class of policy.

OPA Gatekeeper v3.16.0

Enforces runtime policy on all Kubernetes resources. Gatekeeper operates as a validating admission webhook, evaluating every create, update, and delete request against Rego constraint templates. It uses namespace exclusion lists to scope enforcement.

Sigstore Policy Controller

Enforces image signature admission. Every container image must be signed by a trusted key before it can be scheduled. The controller uses opt-in namespace labels to determine which namespaces require signature verification.

Both controllers operate independently. Gatekeeper and Sigstore are installed in separate namespaces and evaluate admission requests through distinct webhook configurations.

Immutable Audit Storage

Production deployments use S3 Object Lock in COMPLIANCE mode for tamper-proof audit event retention. Once written, audit records cannot be modified or deleted by any principal, including root, until the retention period expires.

• IRSA (IAM Roles for Service Accounts): The gateway pod authenticates to S3 using a Kubernetes service account bound to an IAM role. No static credentials are stored.
• Hash-chain verification: Each audit event includes a hash of the previous event, forming a tamper-evident chain. Gaps or modifications are detectable by any verifier.
• Machine-readable proof artifacts: Validation scripts generate structured proof artifacts that auditors can independently verify.

Cloud Adaptation

PRECINCT adapts to each cloud provider through Kustomize patches. No changes to the base manifests are required. Each cloud provider has its own identity, storage, and audit integration.

AWS EKS

IRSA for pod identity, EBS for persistent volumes, S3 with Object Lock for immutable audit storage. Use the prod overlay with EKS-specific patches.

GCP GKE

Workload Identity for pod-level IAM binding, GCS for audit storage with retention policies. GKE-specific patches handle node attestation for SPIRE.

Azure AKS

Pod Identity (or Workload Identity Federation) for Azure AD integration, Azure Blob Storage with immutability policies for audit retention. AKS patches adapt the storage classes and identity bindings.

In every case, the adaptation is handled through Kustomize patches layered on top of the base manifests. The core PRECINCT architecture remains identical across providers.

Production Checklist

Complete the following before running PRECINCT in a production environment.

1. Enable SPIFFE mTLS (SPIFFE_MODE=prod)
2. Deploy SPIKE with multiple Keepers (Shamir HA)
3. Configure S3 Object Lock for audit retention
4. Enable image signature verification (Sigstore policy-controller)
5. Set default-deny NetworkPolicies in all namespaces
6. Configure Pod Security Standards per namespace
7. Set up monitoring and alerting (OTel Collector, Phoenix, or your preferred observability stack)
8. Run make k8s-validate to verify all wiring
9. Run make k8s-runtime-campaign for the full validation report

CI/CD Pipeline Integration

PRECINCT's kustomize-based manifests integrate naturally with GitOps pipelines. Here are patterns for the most common tools.

ArgoCD

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: precinct
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/RamXX/agentic_reference_architecture
    targetRevision: main
    path: POC/infra/eks/overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: precinct
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Flux

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: precinct
  namespace: flux-system
spec:
  interval: 5m
  path: ./POC/infra/eks/overlays/prod
  prune: true
  sourceRef:
    kind: GitRepository
    name: precinct

GitHub Actions

- name: Deploy PRECINCT
  run: |
    kustomize build POC/infra/eks/overlays/${{ env.ENVIRONMENT }} | kubectl apply -f -
    kubectl rollout status deployment/gateway -n precinct --timeout=300s

Multi-Tenancy

For organizations running multiple teams or business units on a shared cluster, PRECINCT supports namespace-based tenant isolation with per-tenant policy boundaries.

Namespace Isolation

Deploy separate PRECINCT gateway instances per tenant namespace. Each gateway loads only the OPA policies relevant to that tenant's agents and tools.

precinct-team-alpha/    # Gateway + tools for Team Alpha
precinct-team-beta/     # Gateway + tools for Team Beta
spire/                  # Shared SPIRE infrastructure
spike-system/           # Shared SPIKE secret store

SPIRE Trust Domain Segmentation

Use SPIFFE ID path conventions to encode tenant boundaries. OPA policies then enforce that agents can only access tools within their tenant scope:

# Only allow agents to call tools in their own tenant namespace
allow if {
    tenant := split(input.caller_spiffe_id, "/")[3]
    input.target_namespace == concat("-", ["precinct", tenant])
}

RBAC Patterns

Combine Kubernetes RBAC with PRECINCT's OPA policies for defense-in-depth. K8s RBAC controls who can deploy and configure; OPA controls what agents can do at runtime.

Service Mesh Integration

PRECINCT uses SPIRE for workload identity and mTLS. If your cluster also runs a service mesh (Istio, Linkerd), coordination is needed to avoid double mTLS and identity conflicts.

Istio Coexistence

• Disable Istio sidecar injection for PRECINCT namespaces (spire, spike-system, precinct) using the istio-injection: disabled namespace label.
• Alternatively, use PeerAuthentication PERMISSIVE mode for the PRECINCT namespace to allow both SPIRE mTLS and Istio mTLS traffic.

Linkerd Coexistence

• Use the linkerd.io/inject: disabled annotation on PRECINCT pods to prevent proxy injection. SPIRE handles identity and encryption for all gateway traffic.

Helm Deployment