Running on GitLab?

GitLab pipelines stuck pending for hours? We clear the backlog.

GitLab is three products wearing one name: a Git host, a CI/CD system and a DevSecOps platform, and the hard problems usually sit at the seams. We come in when pipelines stall on pending runners, when the self-managed instance is three majors behind supported versions, or when a PostgreSQL that nobody meant to run in production has started to groan under the weight of the Container Registry.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

gitlab-ci.log

still broken

What it is

GitLab at a glance

GitLab is an open-core DevSecOps platform founded in 2011 by Dmitriy Zaporozhets and Valery Sizov, now listed as GitLab Inc. and used by millions of developers across the hosted GitLab.com service and self-managed Community and Enterprise Editions. The product covers Git hosting, merge requests, CI/CD with GitLab Runners, a Container Registry, Pages, Package Registry, security scanning and a built-in Kubernetes agent.

The CI system runs .gitlab-ci.yml pipelines on runners that register against an instance, which can be shared or project-specific, and which execute jobs on shell, Docker, Kubernetes or custom executors. The upgrade story on self-managed instances is particular: GitLab enforces an upgrade path across minor versions with background migrations between them, and skipping majors is explicitly unsupported.

Type
Git hosting with integrated CI/CD and DevSecOps
First released
2011
License
MIT for the Community Edition core, proprietary for Enterprise features
Maintained by
GitLab Inc.
Typically used for
Source control, pipelines, container registry, security scanning, self-hosted DevOps for regulated environments

The honest version

GitLab pros and cons

No technology is all upside. This is where GitLab earns its place and where it tends to cause trouble.

Where GitLab is strong

  • One tool for the whole delivery chainSource code, merge requests, CI, container registry, package registry, security scans and Pages all live in one place, which removes integration glue that other stacks pay for out of pocket.
  • Self-managed is a first-class productUnlike GitHub Enterprise Server, self-managed GitLab is the same codebase as the hosted service and has a clear upgrade cadence that regulated customers can actually plan around.
  • CI/CD is powerful out of the boxJobs, stages, needs, parent-child pipelines, dynamic pipelines from generated YAML, and merge trains are there without a plugin ecosystem to maintain.
  • Runners are flexibleShell, Docker, Docker Machine, Kubernetes and custom executors mean you can run CI on a laptop, a VM fleet or an autoscaling cluster depending on the cost model that suits you.
  • Compliance and auditabilityApproval rules, protected branches, audit events, SOC 2 reporting and SAST, DAST, dependency and container scanning sit inside the product, which matters in regulated shops.

Where GitLab bites

  • Self-managed is heavy to operateA full Omnibus install drags in PostgreSQL, Redis, Sidekiq, Puma, Workhorse, Gitaly, Praefect and Registry, and keeping all of them healthy is a job on its own. Teams underestimate it until an upgrade hangs.
  • The upgrade path is strictGitLab enforces stops between majors, and each stop runs background migrations that can take hours. A site three majors behind cannot jump forward and will need a planned multi-step migration.
  • CI YAML grows into its own languageRules, needs, includes, workflow and variables precedence combine in non-obvious ways, and large monorepos end up with CI configuration that only one engineer can read.
  • Runner resource costs add upAutoscaling runners on Kubernetes or Docker Machine are either overcommitted and slow, or undercommitted and expensive, and a cost review on a self-hosted CI fleet is overdue in most shops we see.
  • Hosted plan feature splits moveFeatures move between Premium, Ultimate and Free over time, and teams that bought on one plan find a capability they relied on now sits on a higher tier.

Sound familiar?

Common GitLab problems we fix

The GitLab problems that bring people to us, and what you have once each one is fixed.

Job failed: preparing environment

Pipelines fail before any job runs

A runner that lost its Docker socket, a shell executor with a broken PATH, or a Kubernetes executor where the service account no longer has pod creation rights.

Environments prepared and healthy

Runner has not contacted this instance

Runners fall silent and jobs sit pending

A runner VM shut down and never restarted, a token revoked after a security audit, or a network policy that blocked the heartbeat. The pipeline is pending until a runner rejoins.

Runners present and keeping in touch

yaml invalid: should be a string or array

Pipeline YAML stops validating

An edit to .gitlab-ci.yml that looked fine locally and breaks when includes merge it on the server, or a rules clause with a type mismatch after a refactor.

Pipeline YAML valid and well-structured

gitaly: context deadline exceeded

Clones and fetches on large repos time out

A repository with hundreds of thousands of refs or very large files, where Gitaly cannot finish serving within the deadline. Developer clones fail and CI jobs cancel mid-clone.

Repository serves cleanly under load

upgrade path: 15.11 -> 17.3 not supported

Self-managed is dangerously out of date

Instances stopped at an old major with background migrations never completed, and the direct path to the current version is not supported. The upgrade has to be stepped through planned minor versions.

Instance on a supported version

Registry garbage collection failed

Container Registry is filling the disk

A registry that grew past the size anyone sized the host for, with garbage collection either never scheduled or failing on a specific repository that blocks the whole run.

Registry size under control

PG::QueryCanceled in production.log

500 errors on specific project pages

A PostgreSQL query that runs past the statement timeout under load, usually on issue search or merge request widgets. Users see a Whoops page and reload.

Slow queries tuned, timeouts respected

merge train dropped

Merge trains keep breaking

A flaky job on the head of the train cascades into every following MR dropping off, and a team that depends on trains for ordering loses throughput for hours.

Trains stable under real workload

What we do

GitLab help, from rescue to upkeep

Call us for one pending pipeline, or hand us the whole GitLab instance.

Rescue
A GitLab instance returning 500s, pipelines stuck pending for hours, or a runner fleet that has fallen out of touch. We stabilize the instance and then explain what went wrong.
Self-managed upgrades
Multi-step upgrades across the supported path, rehearsed on a copy of the instance, with background migrations completed on each stop before moving on.
CI/CD review
A review of .gitlab-ci.yml across the organization, with rules, needs, includes and workflow cleaned up so pipelines run only when they should and finish faster when they do.
Runner fleet
Autoscaling runners on Kubernetes or Docker Machine, sized against real queue depth, with the before and after cost on paper and tags that actually match the jobs.
Migrations
Moving from GitHub, Bitbucket or a self-hosted stack onto GitLab, or taking a GitLab instance that has outgrown its box onto a properly sized platform.
Managed Tech
Version watch, upgrade rehearsals, runner and registry monitoring, and someone to page when Sidekiq backs up overnight.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running GitLab alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

GitLab questions

Something else? Call+1 737 727 7327

Why are our GitLab pipelines stuck pending?

Three usual causes come up: runners disconnected and nobody noticed, the job requires tags no runner actually offers, and concurrency on a shared runner is saturated. The admin area shows which runners have checked in recently and what they advertise, and the job detail page shows which tags the job asked for. The fix is to match the two, or to bring a runner fleet back online.

How do we safely upgrade our self-managed GitLab?

You follow the official upgrade path across minor versions, finish every background migration before moving to the next stop, take a verified backup at each step, and rehearse the whole sequence on a copy of the instance before touching production. Jumping majors without stopping at the required versions puts the database in a state the application cannot start against.

Should we move from Jenkins to GitLab CI?

For most teams, yes. GitLab CI is version-controlled YAML next to the code rather than a Groovy pipeline on a server with state. The migration is real work: Jenkinsfile patterns do not translate line for line, shared libraries become CI includes, and credentials move into GitLab CI/CD variables with proper masking. We help plan it so the two systems run in parallel until the new side is proven.

Why is our GitLab instance slow or returning 500s?

The common causes on a busy self-managed install are PostgreSQL queries running past statement_timeout on specific pages, Gitaly saturated by huge repositories, and Sidekiq queues backing up because a worker was removed or a job class fails fast. The gitlab-rails production log plus the Sidekiq admin panel usually point at the specific class or query in minutes.

Can you take over our GitLab installation from a previous team?

Yes. We inventory the version, pending migrations, runner fleet and third-party integrations, rotate the admin credentials, make sure backups actually restore, and bring the instance up to a supported version through the correct path. Then we document how it is wired so a new engineer is not reading production.log blind.

How much does GitLab help cost?

The estimate is free. Describe the instance or pipeline, the version and the problem you are seeing, and you get a quote before we start any work.

Free estimate

GitLab giving you trouble?

Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.