Kubernetes Monitoring: Prometheus, Grafana, Alerts

Short answer: on an SMB Kubernetes cluster, useful monitoring comes down to three decisions, not three hundred dashboards. Install kube-prometheus-stack (the community chart that bundles Prometheus, Alertmanager, Grafana and the kubernetes-mixin rules), size retention at 15 days on a 50 Gi volume — roughly €2 to €5 per month of block storage at OVHcloud or Scaleway — then delete 80% of the alert rules shipped by default, keeping only a dozen that describe a symptom an actual user feels. An alert that triggers no action is a configuration bug, not a precaution.

We regularly take over clusters where Prometheus has been running for a year. The pattern is always the same: a #alerts Slack channel silent for eight months because nobody can stand the notifications any more, a Grafana with forty dashboards of which three have been opened at least once, and a 10-day retention on a disk that overflows every fortnight. Meanwhile, the real incident — a PostgreSQL database saturating its connections, a pod killed by the OOM killer every night at 2 a.m. — goes unnoticed. This article describes the installation we set up by default and, above all, how we triage alerts.

What to measure before picking a tool

Let's break the requirement down. On a cluster serving an SMB's websites and applications, monitoring answers four distinct questions, and they don't share the same tool.

The most expensive confusion is trying to do everything with Prometheus. Prometheus is a low-cardinality time-series database: it stores aggregated numeric metrics, not individual events. Putting a request ID or a user's email address in a label makes memory explode — the Prometheus documentation on metric naming best practices and the instrumentation guide keep repeating it: every label combination creates a distinct series.

Our recommendation for an SMB cluster, in the 3 to 10 node range:

• Metrics: kube-prometheus-stack, self-hosted in the cluster. Short retention, clear aggregation.
• Logs: Loki with an object bucket, or the provider's logging service if the cluster is managed. Logs are bulky and rarely consulted: object storage at €0.01 per GB-month makes them painless.
• Traces: only worth adding once an application has more than two services calling each other. Before that, it's wasted time.

Installing kube-prometheus-stack without getting the values wrong

The kube-prometheus-stack chart is maintained by the prometheus-community project. It bundles the Prometheus Operator, Prometheus, Alertmanager, Grafana, node-exporter, kube-state-metrics and a full set of rules from the kubernetes-mixin project. One command gets you working monitoring — that shouts far too much.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  -f values.yaml

The values.yaml file is where everything is decided. Here is the skeleton we use, adapting storageClassName to the provider (csi-cinder-high-speed at OVHcloud, scw-bssd at Scaleway):

prometheus:
  prometheusSpec:
    retention: 15d
    retentionSize: 40GiB
    scrapeInterval: 30s
    evaluationInterval: 30s
    resources:
      requests:
        cpu: 200m
        memory: 2Gi
      limits:
        memory: 4Gi
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: csi-cinder-high-speed
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 50Gi
    # only pick up explicitly labelled ServiceMonitors
    serviceMonitorSelectorNilUsesHelmValues: false

alertmanager:
  alertmanagerSpec:
    storage:
      volumeClaimTemplate:
        spec:
          storageClassName: csi-cinder-high-speed
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 5Gi

grafana:
  adminPassword: ""        # injected from an external Secret
  persistence:
    enabled: true
    size: 5Gi
  defaultDashboardsTimezone: Europe/Paris

Three settings deserve an explanation, because they cause the majority of monitoring incidents we are called in to fix.

retention and retentionSize together, always

retention: 15d on its own does not protect the disk: if cardinality grows (a new service, new labels), the volume fills up before the 15 days are over and Prometheus stops. retentionSize sets a hard ceiling. We set it to roughly 80% of the volume size — 40 GiB for a 50 Gi PVC — to leave room for blocks being compacted.

For sizing, the rule of thumb in the Prometheus storage documentation still works: space ≈ retention period × samples per second × average sample size (on the order of 1 to 2 bytes after compression). On our 5-node clusters with around thirty instrumented services, we observe 15 to 25 Gi for 15 days in practice. A 50 Gi PVC therefore leaves twice the margin needed, for a few euros a month.

A memory limit, never a CPU limit

Prometheus consumes memory in proportion to the number of active series, with spikes during compaction and heavy dashboard queries. A limits.memory stops it from taking down an entire node; a limits.cpu, by contrast, makes it throttle during compaction and triggers false positives everywhere. It's the same logic as for applications, described in our article on autoscaling Next.js on Kubernetes with HPA and KEDA.

scrapeInterval at 30s, not 15s

Doubling the interval halves both stored volume and CPU load. For SMB websites and applications, detecting saturation in 30 seconds rather than 15 changes nothing: the alert carries a for: 5m anyway to avoid noise. The default 15s is only justified if you are building second-level SLOs.

The eight alerts that deserve a notification

The kubernetes-mixin rule set shipped with the chart contains more than a hundred alerts. Many describe internal control-plane states — unusable on a managed cluster at OVHcloud, Scaleway or EKS, where that control plane isn't yours. The first thing to do after installing is therefore to switch off what doesn't concern you:

defaultRules:
  create: true
  rules:
    etcd: false
    kubeApiserverAvailability: false
    kubeApiserverBurnrate: false
    kubeApiserverSlos: false
    kubeControllerManager: false
    kubeSchedulerAlerting: false
    kubeProxy: false

Then we apply a simple principle, borrowed from the alerting philosophy Google documents in the "Monitoring Distributed Systems" chapter of the SRE Book: only alert on symptoms visible to the user, or on causes that will inevitably become a symptom. Everything else belongs on a dashboard, not in a notification.

Here is the list we keep on an SMB cluster, with the rationale for each one.

On top of this baseline, we always add one application alert per exposed service, the one that single-handedly replaces twenty infrastructure rules: the HTTP 5xx error rate as seen by the ingress or the Gateway. The PrometheusRule looks like this, on metrics exposed by a Gateway API-compatible controller:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: app-availability
  namespace: monitoring
  labels:
    release: monitoring
spec:
  groups:
    - name: app-availability
      rules:
        - alert: HighHttp5xxRate
          expr: |
            sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
            /
            sum(rate(http_requests_total[5m])) by (service)
            > 0.02
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "More than 2% of 5xx errors on {{ $labels.service }}"
            runbook: "https://wiki.internal/runbooks/http-5xx"

Two details make all the difference. The release: monitoring label is essential for the Operator to pick the rule up when the chart is installed under that release name — it's the number one cause of applied PrometheusRule objects that never fire. And the runbook annotation: an alert without an associated procedure will be handled differently by every on-call engineer, and therefore badly.

Routing alerts: two severities, not five

Alertmanager allows arbitrarily complex routing trees. On an SMB cluster, two levels are enough, and any intermediate severity ends up ignored.

alertmanager:
  config:
    route:
      receiver: chat-info
      group_by: ['alertname', 'namespace']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 12h
      routes:
        - matchers: [severity="critical"]
          receiver: oncall
          repeat_interval: 1h
    inhibit_rules:
      - source_matchers: [alertname="KubeNodeNotReady"]
        target_matchers: [severity="warning"]
        equal: ['node']
    receivers:
      - name: chat-info
        webhook_configs:
          - url: https://chat.example/hooks/monitoring
      - name: oncall
        webhook_configs:
          - url: https://oncall.example/hooks/critical

• critical → a notification that wakes someone up, with a runbook. A user-facing symptom, or an imminent outage.
• warning → a message in a channel read during business hours, reviewed once a week.

The inhibit_rules avoid the classic scenario: a node goes down, and thirty "pod not ready" alerts arrive at the same time as the node alert. You only want one. Also consider time_intervals for scheduled jobs that push load up at night — an explicit silence window beats an alert the team learns to ignore.

One point that often gets forgotten: test the notification path, not just the rule. A revoked webhook URL behaves exactly like a healthy cluster. We send a dummy Watchdog alert (shipped with the chart, it is always firing) to a dead man's switch service: if the ping stops arriving, it's the monitoring itself that has gone down. Same reasoning as for a backup that is never restored — a topic we cover in detail in our guide to Kubernetes backups with Velero.

What it actually costs

This is the question that settles the self-hosted versus managed-service debate. The public prices below are those displayed by the providers in autumn 2026; check them before budgeting, they move.

In practice, on the clusters we operate, complete monitoring costs less than €10 excl. VAT per month in storage, and consumes the equivalent of a third of a node. The real cost is elsewhere: the few days of initial tuning, then the monthly hour of alert housekeeping.

SaaS observability offerings bill per host or per ingested volume. They become relevant as soon as you have several clusters, a formalised on-call rota and a need to correlate metrics, logs and traces in the same interface. Below that, kube-prometheus-stack does the job — with the same comparison logic we applied to the choice between PostgreSQL on Kubernetes and a managed database: the price of the service is the operational time it gives you back.

The mistakes we see most often

• Prometheus without a persistent volume. The chart happily starts on emptyDir. Every pod restart wipes the history, and with it any post-incident analysis.
• Grafana as the source of truth. Dashboards created by hand in the UI disappear with the PVC. Storing them in ConfigMaps with the grafana_dashboard: "1" label, versioned in Git, makes them reproducible — exactly like the manifests deployed by the GitLab CI/CD pipeline.
• Unmonitored cardinality. An unnormalised user_id or path label multiplies series by a thousand. We watch prometheus_tsdb_head_series and alert beyond a chosen threshold (for example 1 million series for the resources above).
• No alert on certificates. cert-manager fails silently when a DNS challenge breaks. The alert then comes from your users, three weeks later.
• Alerts without for:. A rule that fires on a single data point produces noise at every 30-second spike. Below for: 5m, ask yourself why.

If you are taking over an existing cluster, the first useful exercise isn't adding dashboards: it's exporting the alerts that fired over the last 90 days and deleting every one that never led to an action. That's typically what we do at the start of an engagement on our infrastructure and managed services work, before adding a single rule. One credible alert channel beats a thousand metrics.

Frequently asked questions

Do you still need Prometheus if the cluster is managed at OVHcloud or Scaleway?

Yes. Providers monitor the control plane (API server, etcd, scheduler) and show you its health, but not your applications: not your 5xx error rates, not your PVCs filling up, not your pods in CrashLoopBackOff. The managed scope stops at the boundary of your namespaces. On the other hand, that does let you disable all the etcd, kubeApiserver*, kubeControllerManager and kubeScheduler rules: there's nothing you can do about them, and they generate false positives.

What retention should you choose for Prometheus?

15 days covers post-incident investigation and week-over-week comparison needs. Beyond that, the need changes in nature: it's about trends (capacity, seasonality), not diagnosis. For that you don't extend local retention — you add a remote_write to Thanos, Mimir or a managed service, with pre-aggregated metrics. Keeping 12 months of raw data at a 30-second interval costs storage and memory without adding anything.

Loki or the provider's logging service?

Loki makes sense if you want everything inside Grafana and a cost tied to object-storage gigabytes. It requires tuning (retention, index, per-tenant limits) and an S3 bucket to manage. Logging services built into managed Kubernetes avoid that operational burden, at the price of an ingestion-based bill that surprises you when an application starts logging in a loop. Either way: cap throughput per namespace, otherwise a single console.log in a loop blows up the budget.

Can you alert on latency without instrumenting the application?

Partly. Ingress controllers and Gateway API implementations expose per-route latency histograms, which is enough for a threshold on the 95th percentile. It's one more argument for moving to the Gateway API as a replacement for ingress-nginx. Knowing where the time went, however, requires traces, and therefore application instrumentation — usually with the OpenTelemetry SDKs.

How many critical alerts per week is normal?

Zero, most weeks. On the SMB clusters we operate, a critical alert is an event, not a routine. If your on-call engineer receives several a week without any corrective action following, the problem isn't the frequency: it's the thresholds. The right health indicator for a monitoring setup isn't the number of alerts, it's the percentage of alerts that triggered an action.

In summary

• kube-prometheus-stack installs Prometheus, Alertmanager, Grafana and the kubernetes-mixin rules in one command; all the work is in the values.yaml and in triaging the rules.
• Always set retention and retentionSize (≈ 80% of the volume), with a 50 Gi PVC for 15 days on a 3 to 10 node cluster.
• A limits.memory on Prometheus, never a limits.cpu: throttling during compaction generates false positives.
• On a managed cluster, disable the control-plane rules (etcd, kubeApiserver*, kubeScheduler, kubeControllerManager): you can't act on them.
• Keep around ten infrastructure alerts plus one 5xx error-rate rule per exposed service, each with a for: and a runbook annotation.
• Two severities are enough: critical that wakes someone up, warning reviewed during business hours. More levels = ignored levels.
• Test the notification path with a dead man's switch on the Watchdog alert: monitoring that has gone down looks just like a healthy cluster.
• Budget less than €10 excl. VAT per month in storage and about a third of a node in resources; the real cost is tuning time.
• Need an outside view on the monitoring of an existing cluster? Talk to our team or contact us.