Running Rancher?

Rancher downstream clusters dropping or stuck? We sort it out.

Rancher is lovely when it works and awkward when it stops. A cattle cluster agent that keeps disconnecting, a project quota that will not release capacity, or an upgrade that paused halfway between versions are all familiar calls. We take over the Rancher management cluster and the downstream clusters together and bring the fleet back into one coherent view.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

rancher-server.log

still failing

What it is

Rancher at a glance

Rancher is a multi cluster Kubernetes management platform. Sheng Liang and the Rancher Labs team released the first version in 2016, and SUSE acquired the company in 2020, making Rancher part of the SUSE Rancher stack alongside RKE, RKE2 and K3s.

The management server runs on a dedicated Kubernetes cluster and connects to downstream clusters through an agent based tunnel. From there it provides a shared UI, central authentication, RBAC projects, fleet GitOps and a shared catalog of Helm charts. Rancher earns its place for teams running more than one cluster, and the cost shows up when agents disconnect, webhooks misbehave or upgrades stall across the fleet.

Type
Multi cluster Kubernetes management platform
First released
2016 initial release by Rancher Labs
License
Apache 2.0, open source with paid SUSE support
Maintained by
SUSE and open source contributors
Typically used for
Managing multiple Kubernetes clusters, on prem plus cloud fleets, centralized RBAC, Fleet based GitOps

The honest version

Rancher pros and cons

No technology is all upside. This is where Rancher earns its place and where it tends to cause trouble.

Where Rancher is strong

  • One UI across many clustersEngineers log into Rancher and see every cluster, every namespace and every workload through a shared console, which beats juggling several kubeconfigs by hand.
  • Centralized authentication and RBACPairing Rancher with OIDC, SAML or Active Directory lets one identity source cover every cluster, and projects give you a layer between user and namespace that plain Kubernetes lacks.
  • Fleet based GitOpsFleet ships as part of Rancher and applies GitRepo resources to many clusters at once, which is useful when you want policy to land on every cluster without wiring Flux or Argo on each.
  • RKE2 and K3s integrationRancher provisions RKE2 and K3s clusters directly, so a new edge or on prem cluster can be one form submission rather than a weekend of kubeadm.
  • Catalog of appsThe built in Helm catalog and curated charts give platform teams a place to hand off deployable components without building a portal.
  • Backed by a vendor that still caresSUSE funds the engineering team, releases are regular, and commercial support exists for teams that need a phone number.

Where Rancher bites

  • The management cluster is a real workloadRancher itself needs a healthy Kubernetes cluster with etcd, ingress and persistent storage. Running it on a toy node and then pointing it at a serious fleet does not end well.
  • Agents can be fragile across networksThe cattle cluster agent tunnels to the management cluster, and NAT, firewall or proxy changes on either side knock it offline. Recovery often needs manual intervention on the downstream node.
  • Upgrades of Rancher itself are high stakesA failed Rancher upgrade can take the management UI offline for every cluster at once. Backups, rehearsal and a documented rollback are not optional.
  • Webhook timeouts cause strange failuresThe validating webhook sits in the admission path for Rancher managed resources, and if the pod is restarting or slow, kubectl apply and GitOps controllers see opaque timeouts.
  • Scaling to hundreds of clusters needs careResource usage on the management cluster grows with fleet size, and the default sizing is not meant for large multi tenant operators.
  • Overlap with Flux and ArgoFleet is Rancher specific, so teams that already standardized on Argo or Flux face a policy choice they would rather avoid.

Sound familiar?

Common Rancher problems we fix

The Rancher problems that bring people to us, and what you have once each one is fixed.

cluster agent disconnected: tunnel closed

A downstream cluster has gone offline in Rancher

A network path between the downstream node and the Rancher server that breaks, a proxy that no longer forwards the tunnel, or a certificate that rotated without the agent noticing.

Agents stay connected, fleet visible

project quota exceeded

A project quota is blocking deploys

A Rancher project with CPU or memory limits lower than the sum of requests, so new workloads fail to schedule even though the cluster has capacity.

Projects sized to real demand

x509: certificate signed by unknown authority

The downstream cluster cannot verify Rancher's certificate

A Rancher server using a private CA that was not distributed to the downstream nodes, or a cert that rotated without agents picking up the new chain.

Trust chain consistent everywhere

rancher-webhook OOM

The Rancher webhook keeps running out of memory

A webhook pod with default resources and a cluster sending more admission requests than it can handle, often after a big GitOps sync lands a thousand resources at once.

Webhook sized for the fleet

helm upgrade rancher: ImagePullBackOff

A Rancher upgrade is stuck pulling images

An upgrade to a version whose images are not in the mirror, or a private registry credential that expired just before the upgrade window.

Upgrades complete on schedule

etcd leader lost on local cluster

The management cluster's etcd is unhealthy

A local cluster where etcd has grown past its quota, lost quorum during a node reboot, or run out of disk space on the data volume.

Management etcd healthy and backed up

RKE2 upgrade drain stalled

A node will not drain during an upgrade

A pod with no PodDisruptionBudget, a volume that will not unmount, or a terminating pod that is waiting for a finalizer that will never fire.

Rolling upgrades complete reliably

webhook admission denied: timed out

kubectl apply is failing on webhook timeout

The rancher-webhook pod is slow or restarting, so admission requests time out and ordinary deploys look broken for reasons that have nothing to do with the workload.

Admission fast and reliable

What we do

Rancher help, from agents to upgrades

One disconnected cluster, or a full Rancher platform review, both work for us.

Rescue
A management cluster that lost its UI, a downstream cluster that will not reconnect, or a webhook that is blocking every deploy. We bring the fleet back into one view.
Upgrades
Rancher server and downstream cluster upgrades rehearsed on a copy, with a documented rollback and a plan for the Helm chart bumps along the way.
Projects and RBAC
Project boundaries, quotas and role bindings aligned so teams get what they need without stepping on other tenants.
Fleet GitOps
GitRepo layout, bundle targeting and secrets handling tuned so a push applies cleanly across clusters without surprises.
Downstream provisioning
RKE2 and K3s cluster templates designed to match your network, storage and ingress conventions rather than fighting them.
Managed Tech
Backups of the Rancher local cluster, agent health monitoring and an operator on call for the odd webhook or etcd incident.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running Rancher alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

Rancher questions

Something else? Call+1 737 727 7327

Why does our Rancher downstream cluster keep disconnecting?

The cattle cluster agent maintains a tunnel from the downstream cluster back to the Rancher server, and that tunnel is sensitive to network path changes. The usual culprits are a proxy that stopped forwarding the websocket, a firewall rule that aged out, or a Rancher server certificate that rotated without the agent getting the new trust bundle. We trace it from the agent pod logs upward rather than restarting and hoping.

How do we upgrade Rancher without taking the fleet offline?

Rancher server upgrades cover the management UI; downstream clusters keep running as long as the control plane stays healthy. The risky window is the Helm upgrade of the rancher chart itself. We back up the local cluster, rehearse the upgrade on a copy, confirm Helm chart compatibility and only then run it live with a rollback documented in advance.

What is a Rancher project and when should we use it?

A project is a Rancher layer above namespaces that lets you group several namespaces under one set of RBAC bindings and resource quotas. For single team clusters it is overhead you do not need; for multi tenant clusters it is the cleanest way to give a team a slice of a cluster without creating a cluster per team.

Should we use Fleet or Argo CD with Rancher?

If your clusters all live in Rancher and you want policy to roll out fleet wide from one Git repository, Fleet is well integrated and reduces moving parts. If you already have Argo CD in production and your workflows depend on it, Argo plus Rancher works and is a common setup. We pick based on what your team already runs and has to maintain.

Why is our Rancher webhook causing kubectl apply to time out?

The validating webhook sits in the admission path for Rancher managed resources. If the rancher-webhook pod is slow or getting OOMKilled, every apply has to wait for its timeout, and ordinary deploys start looking broken. Raising the pod's memory and horizontal scaling it usually calms the fleet down, and the real fix is to work out what is sending so many admission requests.

How much does Rancher help cost?

We do estimates for free. Describe the fleet size and the current trouble, and a quote comes back before any configuration changes.

Free estimate

Rancher making the fleet hard to manage?

Share the Rancher server logs or a quick summary of the clusters, and an operator engineer gets back with a plan and a price.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.