Pod networking stops working across nodes
A flannel pod unable to reach the control plane for lease renewal, or a firewall rule blocking VXLAN UDP 8472 between nodes.
Pods route across the cluster again
Running k3s?
k3s is Kubernetes trimmed down to a single binary, which is a gift until the flannel overlay stops forwarding, Traefik v2 to v3 breaks an ingress, or the SQLite backend hits its ceiling on a cluster that grew faster than planned. We work the edge nodes, the datastore and the add ons together so the fleet stops being a weekend project.
What it is
k3s is a lightweight Kubernetes distribution originally built by Rancher Labs for edge, IoT and small cluster use. The first release shipped in 2019, the project joined the CNCF as a sandbox project in 2020, and it has since graduated to incubating status with wide adoption on single node clusters and ARM boards.
The design strips out alpha APIs and in tree cloud providers, ships as a single less than 100 MB binary, and defaults to SQLite as the datastore instead of etcd. Flannel is the default CNI and Traefik is the default ingress controller, which makes a new cluster come up in minutes. The flip side is that those defaults are opinionated and have their own failure modes once you push past a handful of nodes.
The honest version
No technology is all upside. This is where k3s earns its place and where it tends to cause trouble.
Sound familiar?
The k3s problems that bring people to us, and what you have once each one is fixed.
A flannel pod unable to reach the control plane for lease renewal, or a firewall rule blocking VXLAN UDP 8472 between nodes.
Pods route across the cluster again
A cluster whose state has grown past what SQLite can comfortably handle, usually because the resource count and controller churn grew faster than planned.
Datastore moved to etcd cleanly
A k3s version bump that replaced Traefik v2 with v3, leaving IngressRoute middleware definitions that no longer match the new CRD schema.
Ingress works on the new Traefik
A node token mismatch after the server was reinstalled, or an agent trying to join with the old token after a key rotation.
Agents join cleanly
A host OS that defaults to nftables while k3s expects iptables legacy mode, leading to pod traffic being silently dropped.
Firewall mode aligned with k3s
k3s-killall.sh and k3s-uninstall.sh are powerful scripts that assume you meant it. A copy pasted command on the wrong host takes a node out of the fleet instantly.
Node reprovisioned, scripts guarded
A storage backend that assumes a stable network and a cluster that loses nodes at the edge. Longhorn and local-path both need deliberate choices in these setups.
Storage choices matched to the topology
A three node HA cluster on small nodes where etcd compaction and fsync latency climb to the point where the control plane feels slow.
Datastore sized for the hardware
What we do
One failing node, or a fleet of edge clusters you want standardized, both work for us.
How it works
Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.
We fix the cause, not the symptom. If something is down, getting it back up comes first.
Backups, monitoring, updates and access, so the same problem can't quietly come back.
You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.
Often found together
We work on the whole stack, so the problem can be wherever it actually is.
SQLite is fine for a single server, a small number of workloads and a modest resource count. Once you cross a few hundred megabytes of datastore size, or once you want high availability on the control plane, embedded etcd on three server nodes is the right choice. Migrating later is possible but requires planning, so pick the right backend when the cluster is quiet rather than during an incident.
k3s bundles Traefik, and minor version bumps of k3s sometimes bundle a new major version of Traefik. The jump from Traefik v2 to v3 changed the middleware CRD schema, so IngressRoute resources that referenced middlewares the old way stopped matching. The fix is to update the IngressRoute resources to the new schema, or to pin the previous Traefik chart via k3s HelmChartConfig.
Many teams run k3s in production at modest scale, including HA control planes, Longhorn storage and real customer traffic. It scales well for small to mid workloads. For large multi team platforms, upstream Kubernetes or RKE2 usually fit better, because the trimmed defaults in k3s start getting replaced one by one.
Start the server with --disable=traefik so the built in chart does not install, then deploy ingress-nginx via Helm. Any existing Traefik IngressRoute resources need to be ported to standard Ingress with the ingress-nginx annotations. We often do this as part of a wider cleanup where the full add on surface gets reviewed, not in isolation.
Three usual culprits. First, the host firewall is using nftables while k3s is using iptables legacy, and the rules do not take effect. Second, UDP 8472 for flannel VXLAN is blocked between nodes. Third, /etc/cni/net.d is missing the k3s generated config because the agent crashed during setup. Each has a clean fix once you know which one you have.
Our quotes are no charge. Describe the cluster size and what is misbehaving, and a price goes out before any work begins on your nodes.
Free estimate
Send over the node logs or a short note on the fleet, and an operator engineer gets back with a plan and a price.
Your message is with a real person, and we'll get back to you soon.
Rather talk now? Call us: