Running k3s?

k3s edge nodes crashing or networking broken? We unstick it.

k3s is Kubernetes trimmed down to a single binary, which is a gift until the flannel overlay stops forwarding, Traefik v2 to v3 breaks an ingress, or the SQLite backend hits its ceiling on a cluster that grew faster than planned. We work the edge nodes, the datastore and the add ons together so the fleet stops being a weekend project.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

k3s-node.log

still broken

What it is

k3s at a glance

k3s is a lightweight Kubernetes distribution originally built by Rancher Labs for edge, IoT and small cluster use. The first release shipped in 2019, the project joined the CNCF as a sandbox project in 2020, and it has since graduated to incubating status with wide adoption on single node clusters and ARM boards.

The design strips out alpha APIs and in tree cloud providers, ships as a single less than 100 MB binary, and defaults to SQLite as the datastore instead of etcd. Flannel is the default CNI and Traefik is the default ingress controller, which makes a new cluster come up in minutes. The flip side is that those defaults are opinionated and have their own failure modes once you push past a handful of nodes.

Type
Lightweight Kubernetes distribution
First released
2019 by Rancher Labs, now a CNCF project
License
Apache 2.0, open source
Maintained by
SUSE and the CNCF k3s community
Typically used for
Edge clusters, IoT devices, CI test clusters, single node Kubernetes, ARM boards

The honest version

k3s pros and cons

No technology is all upside. This is where k3s earns its place and where it tends to cause trouble.

Where k3s is strong

  • One binary, one systemd unitA full Kubernetes control plane and agent in a single file, which turns cluster bring up into a scripted install rather than a day of orchestration.
  • Low resource footprintA k3s node runs comfortably on a Raspberry Pi 4 or a small VPS, so teams can run realistic clusters on hardware they already own.
  • Sensible defaults for small clustersSQLite, flannel and Traefik are preinstalled, so a one node cluster serves ingress to a Helm chart without wiring three add ons by hand.
  • Upstream Kubernetes conformancek3s is CNCF conformant Kubernetes, so kubectl, Helm, Argo, Flux and most operators work the same as on EKS or GKE.
  • Embedded etcd for HAThree node HA clusters can use embedded etcd through a flag, so you do not need an external cluster to get redundancy.
  • Great fit for edge and branch officesInstall from a shell script, point at a server URL, and a remote site is in the fleet without shipping a rack.

Where k3s bites

  • SQLite backend caps outThe SQLite datastore is comfortable up to a few hundred megabytes of state. Clusters that grow past that need a switch to embedded etcd or an external datastore, which is a non trivial change.
  • Traefik defaults bite on upgradeTraefik v2 and v3 use different middleware CRDs, and a k3s version bump that swaps the shipped Traefik version can break IngressRoute resources silently.
  • Flannel is minimalThe default CNI is flannel in vxlan mode, which works but offers no network policy, no observability and no advanced routing. Serious workloads often swap it for Calico or Cilium, and the swap is an exercise.
  • Single server is a single point of failureA one node control plane loses the whole cluster if the node dies. k3s makes HA achievable but does not stop teams running production on one box.
  • Firewall rules matter more than expectedk3s uses iptables, and distributions moving to nftables or running stricter firewalls break pod networking in ways that look like random pod failures.
  • Add ons shipped in tree are opinionatedTraefik, ServiceLB and local-path storage are included and enabled by default. Teams that want ingress-nginx or MetalLB have to disable the built ins and then live with the drift.

Sound familiar?

Common k3s problems we fix

The k3s problems that bring people to us, and what you have once each one is fixed.

flannel: failed to acquire lease

Pod networking stops working across nodes

A flannel pod unable to reach the control plane for lease renewal, or a firewall rule blocking VXLAN UDP 8472 between nodes.

Pods route across the cluster again

SQLite datastore size approaching limit

The datastore is outgrowing SQLite

A cluster whose state has grown past what SQLite can comfortably handle, usually because the resource count and controller churn grew faster than planned.

Datastore moved to etcd cleanly

traefik v2 to v3 middleware schema

Traefik upgrade breaks ingress rules

A k3s version bump that replaced Traefik v2 with v3, leaving IngressRoute middleware definitions that no longer match the new CRD schema.

Ingress works on the new Traefik

k3s-agent: Forbidden on agent config

An agent cannot join the cluster

A node token mismatch after the server was reinstalled, or an agent trying to join with the old token after a key rotation.

Agents join cleanly

iptables legacy vs nftables

The host firewall backend is wrong

A host OS that defaults to nftables while k3s expects iptables legacy mode, leading to pod traffic being silently dropped.

Firewall mode aligned with k3s

k3s-killall.sh executed by accident

A cluster was wiped by the built in uninstall script

k3s-killall.sh and k3s-uninstall.sh are powerful scripts that assume you meant it. A copy pasted command on the wrong host takes a node out of the fleet instantly.

Node reprovisioned, scripts guarded

longhorn volume detached

Persistent volumes keep detaching

A storage backend that assumes a stable network and a cluster that loses nodes at the edge. Longhorn and local-path both need deliberate choices in these setups.

Storage choices matched to the topology

etcd compaction took 12s

Embedded etcd is struggling on edge hardware

A three node HA cluster on small nodes where etcd compaction and fsync latency climb to the point where the control plane feels slow.

Datastore sized for the hardware

What we do

k3s help, from edge rollouts to datastore migration

One failing node, or a fleet of edge clusters you want standardized, both work for us.

Rescue
A k3s cluster that will not form, agents refusing to join, or ingress that stopped routing after an upgrade. We take the pressure off first.
Datastore migration
Moving from SQLite to embedded etcd or an external datastore without losing cluster state, with a rehearsal on a clone.
CNI and ingress
Flannel to Calico or Cilium swaps, Traefik to ingress-nginx moves, and MetalLB or ServiceLB choices worked out for your network.
Edge fleet management
Templated node provisioning, air gapped install bundles and Rancher integration for teams running many small clusters.
Upgrades
Minor version upgrades rehearsed on a copy, add on CRD migrations tested, and a documented rollback for every node class.
Managed Tech
Monitoring of control plane health, scheduled backups of the datastore and an operator to call when a remote node drops at 2am.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running k3s alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

k3s questions

Something else? Call+1 737 727 7327

Should we use SQLite or embedded etcd on k3s?

SQLite is fine for a single server, a small number of workloads and a modest resource count. Once you cross a few hundred megabytes of datastore size, or once you want high availability on the control plane, embedded etcd on three server nodes is the right choice. Migrating later is possible but requires planning, so pick the right backend when the cluster is quiet rather than during an incident.

Why did our Traefik ingress break after a k3s upgrade?

k3s bundles Traefik, and minor version bumps of k3s sometimes bundle a new major version of Traefik. The jump from Traefik v2 to v3 changed the middleware CRD schema, so IngressRoute resources that referenced middlewares the old way stopped matching. The fix is to update the IngressRoute resources to the new schema, or to pin the previous Traefik chart via k3s HelmChartConfig.

Can k3s run in production, or is it only for edge?

Many teams run k3s in production at modest scale, including HA control planes, Longhorn storage and real customer traffic. It scales well for small to mid workloads. For large multi team platforms, upstream Kubernetes or RKE2 usually fit better, because the trimmed defaults in k3s start getting replaced one by one.

How do we switch k3s off Traefik to ingress-nginx?

Start the server with --disable=traefik so the built in chart does not install, then deploy ingress-nginx via Helm. Any existing Traefik IngressRoute resources need to be ported to standard Ingress with the ingress-nginx annotations. We often do this as part of a wider cleanup where the full add on surface gets reviewed, not in isolation.

Why is pod networking broken on some of our k3s nodes?

Three usual culprits. First, the host firewall is using nftables while k3s is using iptables legacy, and the rules do not take effect. Second, UDP 8472 for flannel VXLAN is blocked between nodes. Third, /etc/cni/net.d is missing the k3s generated config because the agent crashed during setup. Each has a clean fix once you know which one you have.

How much does k3s help cost?

Our quotes are no charge. Describe the cluster size and what is misbehaving, and a price goes out before any work begins on your nodes.

Free estimate

k3s cluster giving you grief?

Send over the node logs or a short note on the fleet, and an operator engineer gets back with a plan and a price.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.