Operating a hub¶

This page is for the person who runs a hub, on their own host or on their cluster, without having built it. install.sh is the installer you downloaded during installation. On a cluster, you run it on the jump host, and /opt/hub is on the jump host too. <ns> stands for the hub's namespace, as set in the overlay's namespace.yaml, and hub.<your domain> for the hub's host name.

Where things are¶

What	Where
Configuration, host names, authentication, package stack	`/opt/hub/overlay/<name>/`, the overlay
Secrets	Only in `secrets.env` in the overlay directory
Releases, the one that runs and the ones that ran before	`/opt/hub/releases/` and `install.sh status`
User home directories	One volume per user, `home-<user id>`, in `<ns>`
Deployed apps	`HubApp` objects in `<ns>`, with their images in the hub's registry
Sessions	`https://hub.<your domain>/admin`, the hub's admin page, which lists every session with a stop button
Dashboards	`https://hub.<your domain>/grafana/`, for admins only, when the hub has the monitoring stack: usage, app logs and resources, deploys and pipeline runs
Alerts	Mail to the addresses in `alerts_to` in `hub.env`, when the hub has the monitoring stack
Backups	The hub's object store, with `backups=true` in `hub.env`

Admins are the members of the group that the auth patch names in admin-groups. On a hub with the built-in users, the admin is the admin user.

Routine tasks¶

bash install.sh status              # releases here, what each hub runs
bash install.sh upgrade latest      # or a version
bash install.sh rollback
bash install.sh diagnose            # one archive for us

On a single host and on a cluster the commands are the same. On a cluster, run them on the jump host. Where a host runs several hubs, name the hub with --name <name> in upgrade, rollback, diagnose and remove.

Status. status lists the releases downloaded to the host and, for each hub, the release it runs, the one it ran before and its address.

Upgrade. Read the release notes of the version first. upgrade works through these steps:

It fetches the release. The previous release stays under /opt/hub/releases.
If we maintain the overlay for you, it replaces the overlay with the version we publish for that release. Your secrets.env and certs/ stay. An edit you made on the host is kept as a copy under /opt/hub/state/<name>/, and the upgrade says so. Send us such a change, so that it reaches every version.
On a single host, it renders Caddy's configuration again from the host names and the certificate settings.
Where backups are on, it backs up the hub's namespace.
It builds the customer layer of the images for the new version if the hub's registry does not have it yet, or if the files the images are built from changed since the last build. After a rebuild it restarts the deploy service.
It applies the release and runs the smoke test, which deploys a sample app from inside the cluster and removes it again.

Sessions and apps keep running. New sessions start on the new images, and apps are rebuilt on their next deploy. The apps server restarts (about a minute). If the smoke test fails, the release is applied nonetheless: run diagnose and send us the archive.

Roll back. rollback applies the release the hub ran before, with the overlay as it is now, and builds the customer layer for it where needed. On a single host, it first renders Caddy's configuration as that release does. A second rollback returns to the release you rolled back from. The commands rollback, remove and fetch run with the installer of the newest release on the host, so the install.sh you downloaded once does not have to be current. If a release has a bullet starting No rollback in its release notes, it cannot go back to its predecessor. rollback refuses whenever the way back crosses such a release, and so does upgrade with an older version. You then restore from the backup instead.

Change the configuration. Edit the overlay, then run upgrade with the version you already run. The overlay page says what each change costs. A setting needs only an apply. The package stack or a certificate needs an image build that takes minutes, and the upgrade builds it. A change to secrets.env needs one more step, described under Secrets.

Report a problem you cannot place. diagnose writes one archive, hub-diagnose-<name>-<time>.tar.gz, into the current directory. It holds the host's details, the overlay without secrets and certificates, the cluster's objects, the platform's logs, and checks of DNS, HTTPS and the certificates for the host names. It contains host names, user ids and log lines, but no secrets. Send it to us with a description of what you saw.

Manage users¶

Add or remove users, change their groups. You do all of this in your identity provider. A user exists in the hub from their first login, and their groups decide what they see.
Grow a full home volume. New home volumes get the size home_size from hub.env. To grow an existing one, patch its volume claim. On a single host the new size takes effect at the next mount, so stop the user's sessions first:
```
kubectl patch pvc home-<id> -n <ns> -p '{"spec":{"resources":{"requests":{"storage":"40Gi"}}}}'
```
Resolve a home volume that belongs to another user. A home volume is named after the user id with everything but letters and digits replaced, so a.b@example.com and a-b@example.com get the same volume. The volume belongs to the first of them who used it, and a session of the other one ends at once with "this home volume belongs to another account" in its log. To resolve it, have your identity provider give one of the two a user id that differs in more than punctuation and case. If the wrong user claimed the volume, correct the user id in the file .hub-owner in it from a pod that mounts the volume claim, while none of their sessions runs.
Delete a user's data. Stop their sessions, then delete their volume claim. Their apps remain, and you can give each of them to someone else with hub owner <app> <user id>.

Sessions and apps¶

A session hangs or does not start. The admin page lists every session. Stop the session there, and the user can start a new one. When the cause is in the cluster, kubectl describe pod in <ns> shows it. The event "exceeded quota" means the hub's quota (patches/quota.yaml) is used up: raise the quota or stop idle sessions.
A deploy failed. The user sees the build log, and so does the Hub deploys dashboard, per app, when the hub has the monitoring stack. The deploy page lists the common messages and their fixes.
A deploy was interrupted. The build continues and is recorded when it ends. hub history shows it about a minute later.
Roll back, delete or restrict an app. The owner, a maintainer or an admin runs hub rollback from any IDE terminal, and the owner or an admin runs hub delete. To restrict an app, they set access-groups in the app's hub.yaml and deploy it again.
Give every app the same variables. Put them into apps-env/ in the overlay, as Variables for every app describes. They reach every app, API, MCP server and pipeline run as environment variables, as .Renviron does for one app. An app's own variable of the same name wins. New app sessions and the next pipeline run see a change. An API or MCP server sees it after kubectl rollout restart deployment/api-<name> -n <ns>.
A scheduled pipeline run did not happen. kubectl get cronjob -n <ns> pipe-<name> shows the schedule and the last run. A run that starts while another run is in progress does nothing, ends with skipped: another run of this pipeline is in progress in its log, and hub runs lists it as skipped. On a hub with the monitoring stack, a failed run, scheduled or manual, sends an alert mail to the addresses in alerts_to. A skipped run sends none. hub logs <name> shows the reason.

Monitoring and alerts¶

The monitoring stack is optional. The installer's --monitoring option sets it up on a new hub. To add it to a hub that runs:

Copy monitoring/ and patches/monitoring.yaml from the release's example overlay, /opt/hub/releases/<the version you run>/platform/overlays/example/, into your overlay.
In monitoring/grafana.yaml and monitoring/alertmanager-url.yaml, replace hub.example.invalid with the hub's host name.
In kustomization.yaml, uncomment the entry patches/monitoring.yaml.
Set the mail keys smtp_* and alerts_to in hub.env (see the reference) and smtp-password in secrets.env.
Run an upgrade to the version you run.

With the stack, Grafana at https://hub.<your domain>/grafana/ is open to admins. It has ShinyProxy's usage dashboard, the logs and resources of every app, and the Hub deploys dashboard with every build and pipeline run. Alerts go by mail to the addresses in alerts_to, and the Alertmanager dashboard shows the ones currently firing. Every alert mail says what to check first. These are the alerts:

Alert	Meaning
`HubShinyProxyDown`	The hub's ShinyProxy server is down, so nobody can log in.
`HubAppsInstanceDown`	The apps server is down, so deployed apps are unreachable.
`HubComponentDown`	The registry, the deploy service or the operator has no replica. Without the registry no session or app can start, without the deploy service no deploys run, and without the operator no configuration change is applied.
`HubRedisDown`	Sessions and app state are unavailable.
`HubVolumeAlmostFull`	A home, a store or the registry is more than 90 % full.
`HubPipelineRunFailed`, `HubGitBuildFailed`	A pipeline run, or a build from a bound repository, failed. The owner sees why with `hub logs` or `hub git status`. The alert ends when a later run of the same pipeline or app succeeds, or a day after the failure.
`HubHousekeepingJobFailed`	The weekly registry cleanup, a token refresh, or the job that mounts idle homes for the backup failed. The alert ends with the job's next successful run, or a day after the failure.
`HubBackupFailed`, `HubBackupStale`	The last backup failed, or no backup has succeeded for more than a day.

For a quick health check from a shell, check that all pods in <ns> are running, including the two ShinyProxy pods (sp-hub-… and sp-apps-…), and that curl -sI https://hub.<your domain>/login answers 302.

Back up and restore¶

Backups are optional. With backups=true in hub.env, the hub backs up its namespace every day at 02:00 into the S3 bucket configured by object_store_* in hub.env, with the keys from secrets.env. The installer's --bucket option sets all of it on a new hub. On a hub that runs, set the keys and run an upgrade to the version you run. A backup contains the Kubernetes objects and the volume data, and is kept for 30 days. Every upgrade also takes a backup. Volume data is copied only from volumes that a running pod mounts, so the hub mounts idle homes during the backup window to include them. The commands below need the Velero command-line tool, in the version the hub ships, on the host or the jump host:

velero backup get
velero backup create now --include-namespaces <ns> --storage-location <ns> --wait

Restore a whole hub. Recreate the namespace, restore the last backup into it, then apply the release again and run the smoke test:

velero restore create --from-backup <backup> --wait
bash install.sh upgrade <the version you run>

Restore one user's home to look at it without touching the hub. Velero runs one restore at a time, so wait for each command to finish.

Restore the Secrets, ConfigMaps and ServiceAccounts into a scratch namespace:

velero restore create --from-backup <backup> --namespace-mappings <ns>:restore-test \
  --include-resources secrets,configmaps,serviceaccounts --wait

Restore that user's pods, volume claims and volumes. A volume is restored only together with the pod that mounted it, and without the volumes the claim stays pending:
```
velero restore create --from-backup <backup> --namespace-mappings <ns>:restore-test \
  --include-resources pods,persistentvolumeclaims,persistentvolumes \
  --selector hub.linnerprogramming.com/user=<id> --wait
```
The pod comes up with the files. Look at them with kubectl exec -n restore-test.
Delete the scratch namespace when you are done: kubectl delete namespace restore-test.

Disk, reboots and removing a hub¶

Disk. On a single host, check the disk with df -h /, and the volumes with kubectl get pvc -A. The registry is cleaned every Sunday at 03:00, keeping the ten builds in each app's history.
Reboot. After a reboot everything comes back on its own. Sessions restart empty, and files persist.
Remove a hub. bash install.sh remove asks you to type the namespace. It then deletes the namespace with every session, app, home volume and secret, the hub's backup schedule, backup location and object-store keys in the velero namespace, on a single host the Caddy configuration for the hub's names, and the overlay and state under /opt/hub. Backups in the object store, the downloaded releases and the monitoring stack stay. Do not delete the hub with kubectl delete -k: that removes the platform's resource definitions from the cluster, and every hub on it with them.