Grafana was working. Prometheus was collecting metrics, the dashboards had something to say, and I could reach all of it as long as I remembered to leave a port-forward running in a terminal.
I wanted to give it a proper address: grafana.joshlabs.me. A small enough request, considering the amount of infrastructure already involved in telling me how busy my laptop was.
But before making the dashboard easier to reach, I had a few things to sort out. The monitoring chart was still pinned to an older version. Grafana's administrator password needed to move out of the Helm values. And although I'd already used Cloudflare Tunnel for Linkding, this time I wanted to understand HTTPS on the cluster itself.
I expected the hostname to be the easy part. The upgrade and certificate setup needed more attention.
The version pin had been doing its job
In the previous post, getting monitoring running involved recovering from a Flux pruning incident. With that behind me, I could look at the stack without also wondering where my namespaces had gone.
The chart started at 66.2.2, moved to 66.6.0, and was still there when I began planning Grafana's Ingress. Pinning it had kept the deployment predictable. It had also left the maintenance decision with me.
Before changing the chart, I checked the platform underneath it. The node was already running v1.36.3+k3s1, and my local kubectl was v1.36.1. I could leave those versions in place for the monitoring upgrade.
On the ZBook, I inspected how K3s actually started:
sudo systemctl show k3s --property=ExecStart --no-pager--property=ExecStart asks systemd for the service's startup command, and --no-pager prints it directly. The output showed /usr/local/bin/k3s server --disable=helm-controller. Recording that matters for a future K3s upgrade: the existing startup options are part of the installation, even if I haven't thought about them since setting it up.
I also found state.db and its SQLite WAL/SHM files, which told me to use the SQLite backup procedure for this cluster. K3s documents the database directory and server token as required restore material. Active database writes can affect a filesystem copy. I wouldn't rely on copying just state.db while K3s is running.
After losing Linkding's data in the earlier storage incident, I had another backup to think about: its persistent volume. A Kubernetes datastore backup won't include those files. I had identified what to back up, but still needed to test a restore.
For the monitoring upgrade, I selected kube-prometheus-stack 89.2.2 and kept that exact version in Git. Those numbers refer to the chart, not Grafana itself. This particular chart includes Grafana chart 13.2.1 as a dependency.
A jump from 66.x to 89.x crosses multiple chart releases. My successful rollout is evidence about this lab, not a promise that another cluster can make the same jump without reviewing its values and the intervening upgrade notes.
Flux's HelmRelease already allowed CRD replacement during upgrades. After reconciliation, it reported chart 89.2.2 successfully installed. I checked beyond that status: all monitoring pods were running, Grafana accepted my login, the Kubernetes dashboards populated, and Prometheus remained the configured data source. Linkding and its Ingress still worked too.
I'll keep pinning the chart and checking the applications after each upgrade. A successful HelmRelease status was only the first of those checks.
Giving the password somewhere better to live
The administrator password had been committed as an inline Helm value during the initial setup. Moving the same password into an encrypted file wouldn't undo that exposure, so I replaced it with a new credential as well.
I'd already used SOPS and age for Linkding. Grafana could use the same delivery mechanism: an encrypted Kubernetes Secret manifest in Git, decrypted by Flux and referenced by the workload. The chart supports an existing Secret, so there was no reason to put the replacement password inside the HelmRelease.
The one detail that needed separate configuration was Flux's reconciliation boundary. Linkding's apps Kustomization already knew how to decrypt SOPS files. Monitoring had its own Kustomization, so I enabled decryption there too:
# Under spec in the monitoring-controllers Flux Kustomization
decryption:
provider: sops
secretRef:
name: sops-ageThe encrypted manifest creates grafana-admin-credentials in the monitoring namespace, with admin-user and admin-password keys. The HelmRelease references those names. The actual values stay out of its configuration.
After Flux reconciled the changes, I signed in through the port-forward with the new credentials. That gave me a working authentication check before changing how I reached Grafana.
There is a Grafana-specific wrinkle for later: the initial administrator settings aren't a universal password-rotation mechanism for an existing database. If Grafana retains its database, changing a Secret alone may leave the stored password untouched. Moving configuration into Git doesn't remove the need to understand how the application uses it.
A hostname without another ingress controller
Traefik was already running in K3s. I had practised routing Linkding through it, so Grafana didn't need another controller installation. It needed an Ingress rule pointing at its existing Service.
I wanted Helm to generate that rule. To see what my pinned chart offered, I inspected its values:
helm show values prometheus-community/kube-prometheus-stack \
--version 89.2.2This also exposed a useful detail about chart dependencies. The parent chart's values file doesn't repeat every setting available in its Grafana subchart. For example, the parent didn't list grafana.ingress.pathType, while the bundled Grafana chart defaults it to Prefix. An option being absent from helm show values isn't, by itself, proof that the dependency doesn't support it. I left the default in place.
With Ingress enabled for grafana.joshlabs.me, Flux reconciled the HelmRelease and Helm generated the resource. Traefik picked up the host rule and forwarded requests to Grafana's Service.
For DNS, I added a DNS-only A record in Cloudflare pointing to the ZBook's private LAN address. I'd initially used a hosts-file entry for the Linkding test; using DNS meant I didn't have to repeat that edit on every machine.
Cloudflare was only answering the DNS query here. Traffic from my browser went directly to the ZBook over the LAN. A real domain name doesn't make a private address reachable from the internet, and no Cloudflare Tunnel was involved in Grafana's route.
Opening the new HTTP address brought up Grafana, and its dashboards populated. The port-forward could finally have the afternoon off.
Making HTTPS my responsibility
With the earlier Linkding tunnel, Cloudflare handled the browser-facing certificate. For Grafana, Traefik would present a certificate from a Kubernetes Secret. I chose a self-signed certificate for this local exercise and a 90-day lifetime so that renewal would become something to practise.
I generated the certificate and key with OpenSSL in a private temporary directory. These options set the lifetime and hostname:
-days 90 \
-subj "/CN=grafana.joshlabs.me" \
-addext "subjectAltName=DNS:grafana.joshlabs.me"These are options from the generation command, rather than a standalone command. The Common Name describes the subject. Browsers check the hostname against the Subject Alternative Name. I also marked the certificate for server authentication and explicitly set CA:FALSE.
Inspecting the finished certificate showed exactly what I needed:
notBefore=Sep 9 07:44:33 2026 GMT
notAfter=Dec 8 07:44:33 2026 GMT
X509v3 Subject Alternative Name:
DNS:grafana.joshlabs.meI used descriptive local filenames, grafana-tls.crt and grafana-tls.key. Kubernetes' TLS Secret needs entries named tls.crt and tls.key, but kubectl create secret tls handles that mapping. From the repository root, with GRAFANA_TLS_DIR holding the temporary directory's path, I generated the manifest:
(
umask 077
kubectl create secret tls grafana-tls \
--namespace=monitoring \
--cert="$GRAFANA_TLS_DIR/grafana-tls.crt" \
--key="$GRAFANA_TLS_DIR/grafana-tls.key" \
--dry-run=client \
-o yaml > monitoring/controllers/staging/kube-prometheus-stack/grafana-tls-secret.yaml
)
sops --encrypt --in-place \
monitoring/controllers/staging/kube-prometheus-stack/grafana-tls-secret.yaml--dry-run=client writes the definition without creating a live Secret. The subshell's umask 077 restricts the new file's permissions during the short period before encryption. That matters because the generated values are initially base64, including the private key.
I verified that SOPS reported the file as encrypted and added it to the staging Kustomization beside the encrypted admin Secret. The final Grafana values were compact enough to read without hunting through generated manifests:
# Excerpt from the HelmRelease
spec:
values:
grafana:
admin:
existingSecret: grafana-admin-credentials
userKey: admin-user
passwordKey: admin-password
ingress:
enabled: true
ingressClassName: traefik
hosts:
- grafana.joshlabs.me
path: /
tls:
- secretName: grafana-tls
hosts:
- grafana.joshlabs.meThe tls section references the Secret containing the certificate I'd generated. Flux delivers that Secret, and Helm renders the Ingress for Traefik to serve HTTPS. From Traefik to Grafana's Service, traffic still uses HTTP inside the cluster.
The browser was right to complain
After the push and reconciliation, I tested the endpoint using the certificate I'd just generated:
curl --cacert "$GRAFANA_TLS_DIR/grafana-tls.crt" \
-I https://grafana.joshlabs.me
--cacert supplies the trust material for this request, and -I asks for response headers. Curl returned:
HTTP/2 302
cache-control: no-store
content-type: text/html; charset=utf-8
location: /loginThe browser still showed a certificate warning. I'd explicitly told curl to trust this certificate, but hadn't changed my Mac's trust settings. The request confirmed that certificate validation could pass and that Grafana answered over HTTPS. I still needed to configure browser trust.
Using curl -k would have skipped certificate verification. I wanted to check that this certificate worked for the hostname as well as confirm that Grafana answered on port 443.
I imported the public certificate into my macOS login keychain and added trust constrained to SSL and grafana.joshlabs.me. The private key stayed with the server-side configuration. Because this trusts an individual self-signed certificate, replacing it at renewal also means updating that trust on my Mac. Other devices need their own trust configuration.
That was the part of this exercise I found most useful. Creating a certificate is easy enough to copy from an example. Seeing the same endpoint succeed in one client and warn in another made me think about which identity each client was actually prepared to trust.
Grafana remains a local service on one physical machine, with TLS terminating at Traefik. Its administrator credentials and certificate material arrive through SOPS and Flux. I requested HTTPS explicitly for these checks. Automatic HTTP-to-HTTPS redirection still needs separate configuration.
The certificate expires on 8 December. Before then, I'll replace it through the same encrypted Secret manifest, keep the Ingress reference intact and update the trust on my Mac. Until that maintenance job comes around, I can open the dashboard without finding the right terminal tab.