Skip to content

Upgrading

The chart is updated regularly upstream. The OVES copy in deployment-charts/prometheus-stack is currently pinned to chart version 66.3.1 (Prometheus Operator v0.78.2).

General upgrade workflow

  1. Read the upstream release notes between your current version and the target version. Pay special attention to Major Version sections in the chart README.
  2. Update the vendored chart in deployment-charts/prometheus-stack/ (replace files, refresh Chart.yaml, re-run helm dependency update).

  3. Update the CRDs before the rest of the chart:

    kubectl apply --server-side -f \
  https://raw.githubusercontent.com/prometheus-community/helm-charts/kube-prometheus-stack-66.3.1/charts/kube-prometheus-stack/charts/crds/crds/

    Helm v3 does not upgrade CRDs from the crds/ subchart automatically on a regular helm upgrade. The chart's crds subchart only installs CRDs on first install; you must apply CRD updates yourself.

  4. Run the actual chart upgrade:

    helm upgrade prometheus-stack . -n monitoring

  5. Watch the Operator pod and the Prometheus / Alertmanager StatefulSets. The Operator may roll Prometheus/Alertmanager pods to apply spec changes.

Major version upgrades

Each major chart version (e.g. 65.x → 66.x) has a dedicated section in the chart's README.md describing breaking changes. Always:

  • Read every From X.x to Y.x block between your current and target versions.
  • Take a temporary backup of values.yaml and any custom CRs.
  • Try the upgrade on a staging cluster first.

The current version (66.x) was an upgrade to Prometheus Operator v0.78.1+. Future upgrades will note schema changes (often apiVersion bumps for AlertmanagerConfig etc.).

Pinning the chart version

In deployment-charts/prometheus-stack/Chart.yaml:

name: kube-prometheus-stack
version: 66.3.1                # chart version, pinned here
appVersion: v0.78.2            # Operator version

When you re-vendor a new version, bump these and the extra.oves_doc_version in this docs site's mkdocs.yml.

Subchart pins

Subchart versions are pinned in Chart.yaml with a wildcard patch:

dependencies:
  - name: kube-state-metrics
    version: 5.27.*
  - name: prometheus-node-exporter
    version: 4.42.*
  - name: grafana
    version: 8.6.*

helm dependency update . resolves the latest matching patch.

Image overrides

If you use private registries (global.imageRegistry) and pin component images, remember to bump those tags too. The component images move independently of the chart version:

Component Values key for image tag
Prometheus Operator prometheusOperator.image.tag
Prometheus prometheus.prometheusSpec.image.tag
Alertmanager alertmanager.alertmanagerSpec.image.tag
Grafana grafana.image.tag
kube-state-metrics kube-state-metrics.image.tag
node-exporter prometheus-node-exporter.image.tag

Rollback

If a helm upgrade causes problems:

helm history prometheus-stack -n monitoring
helm rollback prometheus-stack <REVISION> -n monitoring

Rollback restores the chart's manifests but does not rollback CRDs that you applied separately. If you applied a newer CRD schema, applying the older CRs again may fail.

Validating the upgrade

After helm upgrade:

# Pods are Running, no restarts
kubectl -n monitoring get pods -l app.kubernetes.io/instance=prometheus-stack

# Operator picked up the new spec
kubectl -n monitoring describe prometheus | grep -A3 Conditions

# Scrape targets are still UP
# (Prometheus UI > Status > Targets)

# Alertmanager config loaded
# (Alertmanager UI > Status)

A short post-upgrade smoke test PromQL:

up{job=~"apiserver|kubelet|kube-state-metrics|node-exporter"} == 1