JC_

September 14, 2026

Station 11, Part 1: Three Blockers a Dry-Run Almost Missed

Gitlab
DevOps
Postgresql
Site Reliability
Software Engineering

Live Wire: A Solo GitLab Upgrade Log — The plan said “just swap the image, keep the environment variables.” A dry-run found three separate reasons that assumption was wrong, stacked one on top of the next.

Three errors, one dry-run, each one hiding behind the last

This is Part 15 of "Live Wire: A Solo GitLab Upgrade Log" Previous: Two Thousand Stalled Jobs | Series overview: Live Wire: A Solo GitLab Upgrade Log — Series Overview | Next: Station 11, Part 2: 202, Not 400 — The Image That Wouldn't Die, Finally Does

TL;DR

Six days after reaching 19.2.4 — enough to satisfy the “roughly a week of observation” threshold set out in the original plan — Station 11 preparation began: replacing the Container Registry image itself with GitLab’s official fork, to finally resolve the protocol mismatch this series met early on. The plan assumed this was a simple image swap, keeping every existing environment variable in place. A dry-run found three separate, stacked reasons that assumption was wrong — a missing config file, a known GitLab bug around database detection, and a permission model mismatch. A follow-up, more rigorous audit then found that a supposedly-complete permission fix had actually only touched one out of 265,487 directories.

An Assumption Worth Testing Before Trusting

The original plan for this station was simple on paper: swap the registry’s container image, leave the rest of the configuration untouched. That kind of assumption — “this should just be a drop-in replacement” — is exactly the kind worth testing in a dry-run before it gets tested in production instead.

The Fix, in One Sentence

Run the swap against an isolated, read-only test first, let every assumption fail where it’s safe to fail, and only then bring the same fix to the real thing.

Blocker 1: The Image That Doesn’t Read the Old Configuration

configuration error: open /etc/docker/registry/config.yml: no such file or directory

The fork image doesn’t configure itself from REGISTRY_* environment variables the way the upstream image did — it uses an Omnibus-style templating mechanism (/scripts/set-config) that expects an actual config file. The "just swap the image, environment variables unchanged" assumption failed on the very first attempt to start the container.

Blocker 2: A Known Bug Around “Fresh Install” Detection

With a minimal config.yml in place, a second error appeared:

"msg":"database in prefer mode on a fresh install: configured and reachable metadata database required to start"
{"err":"failed to connect to `user=git database=`: /tmp/.s.PGSQL.5432 (/tmp): dial error: dial unix /tmp/.s.PGSQL.5432: connect: no such file or directory"}

This traced to a known GitLab issue, #600955: starting with GitLab 19.0, the registry defaults to “prefer mode” for its metadata database, but an upgrade or image swap doesn’t automatically import existing filesystem metadata into that database — so the registry misreads an environment with real, existing data as a brand-new install. The officially documented fix landed in version v4.40.1-gitlab; the image in use here, v4.41.0-gitlab, was built after that fix shipped. Worth being explicit about the limit of that reasoning, though: a version number being newer than a fix's release is an inference, not a guarantee — it's not something to treat as confirmed without testing it directly, which is exactly what this dry-run was for. Setting database.enabled: false sidestepped the issue for the registry's actual configuration going forward.

Blocker 3: A Container That Runs as a Different User Entirely

{"error":"filesystem: mkdir /registry/docker/registry/lockfiles: read-only file system"}

Switching to a writable mount surfaced the real issue underneath:

{"error":"filesystem: mkdir /registry/docker/registry/lockfiles: permission denied"}

The root cause: registry:2.8 and the existing compose setup never specified a user:, so it had always run as root by default. The fork image, by contrast, has its runtime identity fixed to git:git — uid=1000, gid=1000 — not root. The existing data directory was owned by root:root. Nothing about the container itself was broken; it simply wasn't allowed to touch data it didn't own.

Three errors deep before the real blocker (ownership) showed itself

Correcting an Assumption With a Full Audit, Not a Sample

Before running a recursive ownership fix across a 1.6TB data tree, the sensible move was checking the current state first — and a quick sample suggested most of it was already in reasonable shape. A full, unsampled check told a different story:

find /data/gitlab/shared/registry -type d -not -uid 1000 | wc -l
# 265486

265,487 total directories. Exactly one of them had the correct owner. The sample had been technically accurate about what it looked at — it just hadn’t looked at enough of the tree to represent the whole thing. That gap between “the part I checked looks fine” and “the whole thing is fine” is the entire lesson here.

Testing Permission Requirements Function by Function

Rather than guessing how much of the tree actually needed the ownership fix, an isolated, fully-copied mini registry (not a hard-linked shortcut — a genuine copy) made it possible to test each function’s real requirement directly:

Chown Scope            Start   Read    Delete           Push
docker/registry only     ✅      ✅      ❌ 500 EACCES     ❌
+ repositories/          ✅      ✅      ✅ 202            ❌ 500
Whole tree                ✅      ✅      ✅                ✅ 201

Starting and reading tolerated a shallow fix. Deleting needed more. Pushing needed the entire tree. The conclusion: the full recursive chown was necessary, not optional — anything less would have left delete or push silently broken while everything else looked fine. And the fix itself turned out to be fast once run correctly: a full traversal of 208,469 directories took 5.1 seconds, with the actual chown estimated at 10 to 30 seconds — nowhere close to a real constraint on the maintenance window.

One More Thing the Dry-Run Almost Didn’t Catch: Health Checks

The dry-run’s test environment ran over a plain, unencrypted port with no TLS configured, so it never triggered a health-check-related error. Re-auditing the full compose configuration afterward surfaced a difference the isolated test hadn’t been positioned to catch:

HEALTHCHECK={"Test":["CMD-SHELL","/scripts/healthcheck"],"Interval":30000000000,...}
status=$(curl --retry 6 -s -q -o /dev/null -w '%{http_code}' http://127.0.0.1:${PORT:-5000}/v2/)
[[ '200 401' =~ "$status" ]]

The fork image ships a **built-in HEALTHCHECK — something registry:2.8 never had at all — and its script probes over plain HTTP. The real configuration serves HTTPS on that same port. Tested directly, that combination returns 400 Bad Request (Client sent an HTTP request to an HTTPS server) — meaning the container would report permanently unhealthy, regardless of whether the service is actually working. The plan for cutover day: disable the health check (healthcheck: disable: true) during the swap itself, confirm nothing else in the compose setup depends on that health status (restart: always doesn't act on it, and no other service declares a health-based dependency on the registry), and switch to a correct HTTPS-based probe once the swap settles. Worth noting how this one surfaced: it wasn't found by something breaking — it came from a simple planning question about whether the port could stay unchanged, which pulled a thread into several previously unconsidered details at once.

Under the Hood

+---------------------------+-----------------------------------------------+----------------------------------------------------+
| Finding                     | Root Cause                                      | Resolution                                            |
+---------------------------+-----------------------------------------------+----------------------------------------------------+
| Missing config.yml           | Fork image ignores REGISTRY_* env vars,          | Supplied a minimal config.yml matching the fork's    |
|                             | expects a real config file                       | expected format                                        |
+---------------------------+-----------------------------------------------+----------------------------------------------------+
| Database prefer-mode error   | Known GitLab issue #600955 — misreads an          | Set database.enabled: false, avoiding the fresh-      |
|                             | existing environment as a fresh install           | install misdetection entirely                          |
+---------------------------+-----------------------------------------------+----------------------------------------------------+
| Permission denied             | Fork image runs as git:git (1000:1000),           | Full recursive chown, confirmed necessary via         |
|                             | not root; data was owned by root:root              | function-by-function testing                            |
+---------------------------+-----------------------------------------------+----------------------------------------------------+
| Health check mismatch         | Fork image's built-in HTTP health probe            | Disabled during cutover, switched to a correct         |
|                             | against an HTTPS-only port                        | HTTPS probe once the swap settled                        |
+---------------------------+-----------------------------------------------+----------------------------------------------------+

What Actually Worked

Every one of these four findings surfaced during a dry-run, not during the real cutover — which is the entire point of running one. The permission audit specifically demonstrates something worth generalizing past this one station: a sample that looks clean isn’t the same claim as a full check that’s actually clean, and the gap between the two can be enormous — one correct directory out of 265,487 would have been easy to mistake for “already handled” without looking at all of them.

Where This Still Falls Short

This dry-run used read-only mounts as a deliberate, conservative choice — which meant it couldn’t test anything requiring real write access, including the exact permission depth that later needed its own separate, more thorough check. None of these findings involved any drama or urgency; they were exactly what a dry-run is supposed to produce; the real payoff of this station is what happens next, once every one of these was already resolved before the actual cutover began.

Have you ever had a “simple swap” turn out to have three separate, stacked assumptions hiding behind it — each one only visible once the last one got out of the way?

This is Part 15 of "Live Wire: A Solo GitLab Upgrade Log" Previous: Two Thousand Stalled Jobs | Series overview: Live Wire: A Solo GitLab Upgrade Log — Series Overview | Next: Station 11, Part 2: 202, Not 400 — The Image That Wouldn't Die, Finally Does