Public incident analysis. This article is based on GitLab’s published posts about the GitLab.com database outage of 31 January 2017. Oscillate Infotech did not build GitLab, manage its infrastructure, participate in the incident, or investigate the outage, and has no non-public information about it. Confirmed facts come from those official publications. Sections labelled independent analysis are Oscillate Infotech’s operational reading of the documented failure sequence.
Confirmed facts from the company publications
On 31 January 2017, while engineers were restoring database replication after load and lag problems, data was accidentally removed from the primary production database host. The destructive command was intended for the secondary. Around 300 GB had been removed before the command was stopped.
GitLab.com was unavailable for about 18 hours. Some production database changes could not be recovered. GitLab estimated loss on the order of 5,000 projects, 5,000 comments and 700 new user accounts. Git repositories and wikis were stored separately and were not lost.
The regular `pg_dump` backup process was failing: it used PostgreSQL 9.2 tools against a PostgreSQL 9.6 database. Failure notifications were emailed but rejected (DMARC-related). The expected S3 bucket had no usable recent backup. Database-server disk snapshots had not been enabled. The secondary could not be used for recovery because its data had already been wiped while rebuilding replication. Recovery used an LVM snapshot taken roughly six hours before the outage; copying data back took around 18 hours. GitLab identified lack of ownership for regularly testing recovery as a contributing problem.
Configured jobs are not recoverable copies
A calendar of backup jobs can look complete while every recent artefact is missing. Version-skewed tooling that exits with an error, plus alerts that never arrive, produce false confidence until the night a restore is required. The operational metric that matters is the age and integrity of the newest copy you can actually restore—not the existence of a cron entry.
Replicas are not disaster recovery
Independent analysis. Hot-standby replication improves availability when the replica is healthy. It does not protect you when rebuild procedures wipe both sides, or when corruption is copied downstream. Availability design and disaster-recovery design answer different questions; conflating them delays the moment you discover you never had an independent recovery point.
Restore proof needs an owner
Independent analysis. Without a named owner, restore drills slip and runbooks stay theoretical. RPO is the age of the newest restorable copy. RTO is how long that restore takes under real storage limits. Both numbers are fiction until someone restores into an isolated environment, checks the application, times the run, and records the result. Off-site and immutable copies help only if they are reachable and tested.
Storage path decides recovery time
Copying a large snapshot across throttled disks can dominate outage length even when a usable snapshot exists. Planning must include restore bandwidth, not only backup frequency. Staging hosts used as the only practical snapshot location create another dependency under pressure.
Controls tied to this sequence
Independent analysis:
- Clear production versus replica identification on every host and session.
- Privileged-access controls and dual approval for destructive data-directory operations.
- Version-compatible backup tooling and monitored alert delivery.
- Independent off-site copies with protected retention.
- Named ownership for backup success and restore proof.
- Periodic disaster-recovery exercises that include storage speed.
What smaller businesses should test
Restore last week’s backup of one critical system into isolation. Time it. Confirm the application opens. Confirm failure alerts arrive. Confirm replicas are not your only second copy. Write the last successful restore date next to the last backup date.
Our application maintenance and IT support work treats recoverability as something you demonstrate for the business applications we maintain and support. That is operational support and maintenance discipline—not a standalone disaster-recovery, cloud, or hosting product. When estates are rebuilt, data migration planning should include how you reverse or rebuild if cutover fails. Related reading: Application Maintenance: Preventing Downtime, Maintaining Legacy Systems Effectively, and The Data Moved, but the Service Failed.
How we assess application recoverability
In maintenance and support engagements we ask:
- When was the last end-to-end restore of the application database or datastore, by whom, and how long did it take?
- What is today’s restorable RPO, and does the business accept it?
- Where do failure alerts land, and when was delivery last proven?
- Who owns recoverability after database or tooling version changes?
Independently written and illustrated from documented public facts, with no third-party creative assets reproduced.
Sources
- GitLab, Postmortem of database outage of January 31 — primary source for wrong-host deletion (~300 GB), failed recovery paths (pg_dump 9.2 vs 9.6, empty S3 bucket, rejected failure emails, disabled database disk snapshots, secondary wiped during replication rebuild), ~18-hour outage and restore copy time, LVM snapshot ~6 hours before the outage, estimated loss figures, separate storage of Git repositories and wikis, and lack of ownership for regular restore testing.
- GitLab, GitLab.com database incident — contemporaneous incident post confirming the outage timeline and data-loss window while restoration was underway.
