Remembering Simon Riggs. Always.

About EDB

• Contributing to PostgreSQL since 2004
  • In 2020, acquired 2ndQuadrant, another historical PostgreSQL contributor, co-founded by me
• EDB Postgres AI, a sovereign AI and data platform for agentic enterprises
  • Includes: EDB Postgres Extended, EDB Postgres Advanced Server and EDB Postgres Distributed
• A global company, trusted Postgres partner for governments, banks and telcos worldwide
• The most significant organisation contributing at the intersection of PostgreSQL and Kubernetes
  • Creators of CloudNativePG, Kubernetes Certified Service Provider (KCSP) and CNCF Silver Sponsor

Disclaimer

• This presentation assumes your organisation has already decided to move Postgres to Kubernetes
  • If you're still evaluating that decision, this talk is about the how, not the whether
• This presentation is about running Postgres on Kubernetes with CloudNativePG
  • I am one of the founders and maintainers
  • My opinion on it can't be anything but biased and passionate
  • Some of the concepts also adapt to 1st generation operators
• Don't just believe me: do your own research
  • Pick CloudNativePG and two other operators, and test them
  • Involve your infrastructure/Kubernetes team in the process
  • Choose what makes you confident running it in production
• The facts in this presentation are why I chose it, not a substitute for your own evaluation

Agenda

1. Setting the scene: goal and context
2. Day 0: the building block, then storage and a dedicated physical layer
3. Day 1: deploy: HA, GitOps, RBAC, backup and observability declared, configuration
4. Day 2: operate: verify and grow what Day 0/1 set up, upgrades, lessons from the trenches
5. Closing: portability and sovereignty

The goal

Running Postgres on Kubernetes in a nutshell

Complex topic, scan the QR code for background information

• You probably didn't choose Kubernetes
  • That's usually an organisational decision, and DBAs are rarely part of it
• Your organisation needs to know both technologies, collectively (third parties included)
  • You, as an individual, don't need to know both, just build a T-shaped profile and collaborate
    • DBAs: the CNCF's Certified Kubernetes Administrator (CKA) helps you build the top of your T
• Adopt cloud-native paradigms and avoid lift-and-shift operations
• Use an Operator to run Postgres
  • Advice: don't attempt to run it as a plain Deployment or StatefulSet

What is CloudNativePG?

The most popular operator for Postgres

• Second-generation operator for Postgres
  • Kubernetes-native, no external failover manager, pluggable backup system, no StatefulSet
  • Entirely declarative, focused on the Cluster resource
• Created by EDB, now part of the Cloud Native Computing Foundation (CNCF)
  • Alongside Kubernetes, Prometheus, OpenTelemetry, Keycloak, and SPIFFE/SPIRE
  • CNCF Sandbox project (1st stage), working toward CNCF Incubation (2nd stage of three)
• Vendor-neutral governance, with a growing contributor and user base outside EDB
• Health score: 84/100, "Healthy" (Linux Foundation Insights)
  • Top 10% of open source projects for total commits in the last 12 months

Architecture

How the database architecture changes when moving to CloudNativePG

1. Transition from a database-centric perspective to an infrastructure-wide one
   • Postgres must be seen as any other application, with some special needs as a data workload
   • "Cattle vs Pets" → "Cattle vs Elephant herds" for Postgres: the herd concept, applied to your database
     • Kubernetes enables this with clear separation of responsibility between DBA and platform team
2. Define your architectural building block in a single Kubernetes cluster
   • Physical layer: nodes, availability zones, where things actually run
   • Logical layer: namespaces, services, the Cluster resource, what you actually declare
3. Apply the building block across the different Kubernetes clusters
   • Symmetric architectures with the distributed topology feature of replica clusters
   • Enable hybrid and multi-cloud setups, across regions

Mapping the physical layer

From servers to Kubernetes resources like nodes and AZs

Mapping the logical layer

From Postgres instances to HA clusters serving applications

Logical layer: adding continuous backup

Your architectural building block for CloudNativePG Cluster within a single Kubernetes cluster

• Native support:
  • Standard Volume snapshots, inside the same namespace
• Via CNPG-I plugins:
  • Object store, inside or outside Kubernetes (e.g. Barman Cloud plugin)
  • Local persistent volume, inside the same Kubernetes cluster (e.g. Klio, experimental)

Stretching across Kubernetes clusters and regions

Reuse the same building block to deploy symmetric architectures

Storage in CloudNativePG

• Redundancy comes from Postgres replication, not from the storage layer
• Storage in CloudNativePG is provisioned via:
  • Standard StorageClass resource
  • Container Storage Interface (CSI) drivers
• Network storage (most common in Kubernetes):
  • Attention: Blocks are normally replicated 3 times (write amplification issue)
  • Examples: Rook (Ceph), Longhorn, cloud-provider CSI drivers, enterprise SAN
• Local storage on the node (generally recommended for Postgres)
  • Enables shared-nothing architectures
  • On bare metal Kubernetes worker nodes, use directly attached NVMe drives through LVM
  • Examples: OpenEBS (Local PV), TopoLVM
• The same storage concepts from traditional environments apply
  • Benchmark, benchmark, benchmark!

Securing the Postgres physical layer

Be the guardian of Postgres inside Kubernetes

• What I most often see with customers:
  • The infrastructure team has already provisioned the Kubernetes infrastructure
  • They have no special plan for Postgres (unless you do)
    • Runs on existing worker nodes, shared with other applications
    • Relies on network storage, shared with other applications
  • Shared workloads and storage are OK to start a pilot program
• For long-term production, request a dedicated physical layer for Postgres
  • Nodes dedicated to Postgres through node taints and labels (next slides)
  • Where possible, bare metal nodes with local NVMe drives (best TCO)
  • This might lead to unexpected additional budget requests
    • Longer provisioning time, the earlier you start, the better
    • Especially on-premises (CAPEX)

Shared-Postgres architecture

The herd concept: many Postgres instances, one dedicated pool

• You dedicate nodes (node pool) for the Postgres workload (physical layer)
  • Kubernetes schedules Postgres pods across that pool the way a hypervisor schedules VMs across bare metal
• The building block is your AZ where they exist, otherwise the Kubernetes cluster
  • One dedicated Postgres node per AZ, 3+ nodes minimum per Kubernetes cluster (4+ for safer node upgrades)

Shared-nothing architecture

A node dedicated to a single Cluster instance

• True shared-nothing: a node is dedicated to a single instance of a single Cluster
• Reserved for your most important workloads, not the default - an opt-in tier
• Kubernetes gives you full flexibility to mix both models on a per Cluster basis

Defining the physical layer

Dedicate nodes for Postgres via taints and labels

kubectl taint node ${node} \
  node-role.kubernetes.io/postgres=:NoSchedule

kubectl label node ${node} \
  node-role.kubernetes.io/postgres="" \
  node-role.kubernetes.io/worker-

• These operations will be done by the infrastructure team, not you!
• label identifies the node; taint keeps everything else off it
  • CloudNativePG's Cluster resource will match it with its own nodeSelector and tolerations
• No "noisy neighbour" competing for CPU, memory, and, possibly, the storage

Recap

Day 0

• One building block per AZ, then Kubernetes cluster, replicated via Distributed Topology
• Physical layer: AZs and nodes
• Logical layer: Cluster resource and Kubernetes Services
• Continuous backup:
  • Native snapshots
  • CNPG-I plugins (object storage, PVCs)
• Storage: redundancy from Postgres replication, not storage
• Dedicated physical layer: Shared-Postgres (herd) or Shared-nothing

The Cluster resource

Your building block as a YAML file

• The Cluster resource is the interface between DBAs and infrastructure team
• Smallest unit of configuration for a Postgres cluster in CloudNativePG
  • You declare an HA topology, not a single instance
  • Backups too work at the Cluster level, not the instance
• Convention over configuration, with some strict defaults:
  • No superuser access over the network (NULL password for Postgres)
  • ALTER SYSTEM is disabled
  • TLS by default (mTLS for HA replicas)
  • Read-only container filesystem (PGDATA lives on its own writable volume)
  • Fixed 5432 port (no pragmatic need to change it in Kubernetes)

High availability, the declarative way

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: acdc
spec:
  instances: 3
  affinity:
    nodeSelector:
      node-role.kubernetes.io/postgres: ""
  postgresql:
    synchronous:
      method: any
      number: 1
  storage:
    size: 200Gi

GitOps, and where the DBA's job begins

• What I most often see with customers:
  • Organisations implement Infrastructure as Code and GitOps practices
  • Platform owns getting the YAML applied:
    • Argo CD (git-based)
    • Flux (also works "gitless", straight from released OCI artifacts)
• Your boundary is the Cluster resource and everything inside its spec:
  • That's Postgres domain, reviewed in the pull request
  • Everything above it (i.e. node pools, storage classes, network policies, ...) is platform domain
• Your new interface to production might be a pull request
  • Not a terminal on a database host
• Manual kubectl apply might still work
  • Common for local testing, and often good enough for dev/staging
  • Available during Day 2 troubleshooting
  • GitOps is a discipline, not everyone adopts it - most valuable once you hit production

Role-Based Access Control (RBAC)

What I most often see with customers in terms of DBA permissions

• RBAC: Kubernetes' native model for who can do what
• The DBA is not granted a cluster-admin role
• A dedicated, read-only DBA persona instead:
  • Physical layer: read-only on nodes, PVs, PVCs
  • Logical layer: get/list/watch on common CNPG and K8s-related resources
    • Cluster, Backup, Pooler, Lease, ConfigMaps, Services
    • Scoped get on application Secrets only
    • pods/portforward to open a tunnel
    • no pods/exec/attach (no shell inside the container)
• Write access still happens: just through the pull request
• Day 2: troubleshooting may warrant a separate, temporary, audited elevated role (not the default)

Backup and observability, declared

Implement what you already decided in Day 0

• Backup: just more fields in the Cluster spec
  • Standard interface with Backup and ScheduledBackup resources common to native and CNPG-I plugins
• Logs: structured JSON to stdout, to your existing log stack
  • No log is stored inside the containers' volumes!
• Metrics: built-in Prometheus exporter, nothing separate to install
• Same principle as GitOps and RBAC:
  • Postgres integrates with the rest of the infrastructure

Configuring PostgreSQL

How configuration changes from traditional environments

spec:
  postgresql:
    parameters:
      shared_buffers: "4GB"
      max_connections: "200"
      work_mem: "32MB"

Before:

• DBAs edited postgresql.conf, pg_hba.conf, ... and Linux OS settings directly
  • Usually owning the machine itself

Now:

• You change the Cluster resource, full stop
  • CloudNativePG writes the actual config files and applies them for you, safely

Watching it work: backups and observability

Watch what you built in Day 1, then grow it

• Monitor backups: one more metric on the Day 1 dashboard
• Restore drills: an untested backup is a hypothesis, not a backup
• Same discipline for logs/metrics: unused dashboards and alerts are a hypothesis too
• Grow your metrics: the exporter's defaults are a floor, not the ceiling

Rolling changes safely

Kubernetes upgrades, PostgreSQL upgrades, config changes: one pattern, three examples

• Kubernetes node upgrade
• PostgreSQL minor upgrade
• PostgreSQL major upgrade
  • Offline, in-place upgrade with pg_upgrade
  • Blue/green, online upgrade with logical replication
• Config changes
• Test all of these in staging before your platform team schedules the production window

The roadmap: five eras of Postgres infrastructure