September 11, 2026
Station 7–8: The Backup That Looked Broken
Live Wire: A Solo GitLab Upgrade Log — A backup file appeared on disk at a third of its usual size. The obvious read was failure. The actual answer was: it just wasn’t done writing yet.

A file that looked finished, and wasn’t
This is Part 12 of "Live Wire: A Solo GitLab Upgrade Log" Previous: Tuning the Parallelism Knob | Series overview: Live Wire: A Solo GitLab Upgrade Log — Series Overview | Next: Reaching 19.2.4: The Milestone
TL;DR
Between Stations 7 and 8, a daily automatic backup produced a .tar file sitting at 12GB — a third the size of the 38-39GB it had consistently been for the three days before. The obvious read was a failed or stuck backup. The actual process was still running, mid-write, confirmed by checking for the process directly instead of trusting the file on disk. Ruling out two real suspects — competing migrations, a full disk — left one honest answer: every stage had simply gotten 50 to 80% slower, from data volume growing day over day. Nothing was broken. It just needed more time.
A File a Third the Size It Should Have Been
The daily automatic backup ahead of Station 8 had taken roughly two hours for each of the three days prior — consistent, predictable. That day, checking in on it found something that looked wrong immediately: a final .tar file already present — 1788397250_2026_09_03_18.8.11_gitlab_backup.tar — but sitting at 12GB, against a consistent 38-39GB every day before. The intermediate db/ and repositories/ directories hadn't been cleaned up. artifacts.tar.gz was nowhere to be found. The first read was that the backup had failed or gotten stuck.
The Fix, in One Sentence
Don’t trust a file’s presence on disk as proof of completion — check whether the process that’s supposed to produce it is still actually running.
Quick Start: Checking the Process, Not Just the File
ssh Gitlab "bash -ic 'echo \$SUDO_PASS | sudo -S pgrep -af \"tar --exclude\" | grep -v pgrep'"
# empty result = it actually finished
# any result = it's still running
Running this against the “failed” backup returned a result. The tar process was still alive.
What Was Actually Happening
ps aux | grep tar
The packaging process — tar --exclude=lost+found --directory=/home/git/data/backups --create ... — was still running, in state D: uninterruptible disk I/O wait, not dead. The parent rake gitlab:backup:create process was alive too. That 12GB file wasn't a failed or finished artifact — it was a snapshot of a tar archive caught mid-write, at whatever point it happened to be when it was checked.
Ruling Out the Obvious Suspects
Two real hypotheses got checked before accepting “it’s just slow”:
Was a leftover migration competing for I/O? Station 7 had left 27 background migrations behind, two of them touching hundreds of thousands of rows — a reasonable suspect for I/O contention.
SELECT count(*) FROM batched_background_migrations WHERE status = 1;
-- 0
Zero active migrations. Ruled out.
Was the disk simply full?
df -h /data
# 3.0T total, 1.8T used, 1.1T available
1.1TB free. Ruled out.

Ruled out migration contention. Ruled out a full disk. What was left was simply more data than the day before.
The Real Answer: Every Stage Slowed Down, Systemically
Stage 09-01 09-02 09-03 change
Dumping database 104s 105s 176s +68%
Dumping repositories 4063s 2983s 5371s +80%
Dumping artifacts 3413s 2606s 3986s +53%
Every single stage — database, repositories, artifacts — slowed by 50 to 80% compared to the day before, not just one bottlenecked component. A single failing piece would show up as one outlier stage, not a uniform slowdown across all of them. The most honest explanation: repositories/ and artifacts/ grow with every day's commits and CI activity, and that day's dataset was simply larger than the prior day's, compounded by a short-lived I/O spike. Nothing here needed fixing. It needed time.
The Judgment Call, Learned a Second Time
This is the same lesson Station 4’s pg_dump taught earlier in this series, generalized past a single tool: a file existing on disk doesn't mean the job that produces it is done. The reliable check needs three things — the process has actually exited (an empty pgrep result), the intermediate working directories have been cleaned up, and the final file's size has stabilized in the expected range rather than still climbing. Any one of those missing means the job isn't finished yet, file or no file.
Under the Hood
+---------------------------+-----------------------------------------------+----------------------------------------------------+
| Item | What I Did | Why It Mattered |
+---------------------------+-----------------------------------------------+----------------------------------------------------+
| First read | Saw a 12GB file where 38-39GB was expected | Looked exactly like a failed or stuck backup |
+---------------------------+-----------------------------------------------+----------------------------------------------------+
| Real check | pgrep for the tar process instead of trusting | Confirmed the process was alive and still writing — |
| | the file on disk | the file was a mid-write snapshot, not a result |
+---------------------------+-----------------------------------------------+----------------------------------------------------+
| Ruled out | Checked for competing migrations and disk space | Eliminated two real, checkable suspects before |
| | | accepting "it's just slower today" |
+---------------------------+-----------------------------------------------+----------------------------------------------------+
| Real answer | Compared all three stages across three days | Uniform 50-80% slowdown across every stage pointed |
| | | to data growth, not a single broken component |
+---------------------------+-----------------------------------------------+----------------------------------------------------+
What Actually Worked
Checking for the actual process instead of trusting the file on disk is what prevented a false alarm from turning into an unnecessary intervention. Ruling out the real suspects — migration contention, disk space — one at a time, rather than jumping straight to “data must just be growing,” is what made that final conclusion trustworthy instead of a convenient guess.
Where This Still Falls Short
There’s no fix to report here, and it’s worth being direct about that rather than manufacturing one: the diagnosis was “nothing is actually wrong,” and the resolution was waiting. That’s a less satisfying ending than most of this series, but it’s the honest one — and it’s the second time this exact lesson has shown up in this project, which is itself worth noting. A rule learned once might be a fluke. A rule learned twice, in two different tools, is a real principle.
Have you ever chased what looked like a serious failure, only to find the honest answer was “nothing’s broken, it just needs more time”?
This is Part 12 of "Live Wire: A Solo GitLab Upgrade Log" Previous: Tuning the Parallelism Knob | Series overview: Live Wire: A Solo GitLab Upgrade Log — Series Overview | Next: Reaching 19.2.4: The Milestone
GitHub: jasonChen0604
Portfolio: jason-chen-1cb56.web.app
LinkedIn: jason-cj-chen