GitLab Server Migration: From CentOS 7 to Ubuntu 24.04, from 15.8.1 to 18.10.0

With CentOS 7 reaching EOL in June 2024, GitLab upgrades were halted. Version 17.7 is the last version supported on CentOS 7, so we decided to migrate to a new server and upgrade to the latest version.

Migration Overview

┌───────────────┐         ┌───────────────┐
│   Existing Server (CentOS 7)   │         │   New Server (Ubuntu 24.04)│
│                         │         │                         │
│   GitLab 15.8.1         │         │   GitLab 18.10.0        │
│   ↓                     │  rsync  │                         │
│   GitLab 17.7.0 ─────│────▶│   GitLab 17.7.0        │
│   (Last version for CentOS)   │         │   ↓                     │
│                         │         │   GitLab 18.10.0        │
│   /data/git-data (135GB)│         │   /data (300GB)         │
└───────────────┘         └───────────────┘

┌───────────────┐         ┌───────────────┐
│   Existing Server (CentOS 7)   │         │   New Server (Ubuntu 24.04)│
│                         │         │                         │
│   GitLab 15.8.1         │         │   GitLab 18.10.0        │
│   ↓                     │  rsync  │                         │
│   GitLab 17.7.0 ─────│────▶│   GitLab 17.7.0        │
│   (Last version for CentOS)   │         │   ↓                     │
│                         │         │   GitLab 18.10.0        │
│   /data/git-data (135GB)│         │   /data (300GB)         │
└───────────────┘         └───────────────┘

Why Do It This Way?

1. GitLab cannot be upgraded to the latest version all at once. - It is necessary to upgrade sequentially through a defined path.

2. Ubuntu 24.04 supports GitLab from version 17.1.0 - cannot install 15.x directly.

3. CentOS 7 only supports up to 17.7 - there are no packages for later versions.

Ultimately, we decided to upgrade to 17.7 on CentOS first, then migrate to Ubuntu to upgrade the rest.

Phase 1: Upgrade from CentOS 7 (15.8.1 -> 17.7.0)

Maintenance Mode & Backup

# Block push
sudo gitlab-ctl deploy-page up

# Create backup
sudo gitlab-backup create

# Backup configuration files
sudo cp /etc/gitlab/gitlab.rb /root/gitlab.rb.backup
sudo cp /etc/gitlab/gitlab-secrets.json /root/gitlab-secrets.json.backup

# Block push
sudo gitlab-ctl deploy-page up

# Create backup
sudo gitlab-backup create

# Backup configuration files
sudo cp /etc/gitlab/gitlab.rb /root/gitlab.rb.backup
sudo cp /etc/gitlab/gitlab-secrets.json /root/gitlab-secrets.json.backup

Upgrade Path

Followed the upgrade path suggested in the official GitLab documentation.

15.8.1 → 15.11.13 → 16.0.8 → 16.3.7 → 16.7.9 → 16.11.10 → 17.0.8 → 17.3.7 → 17.5.0 → 17.7.0

15.8.1 → 15.11.13 → 16.0.8 → 16.3.7 → 16.7.9 → 16.11.10 → 17.0.8 → 17.3.7 → 17.5.0 → 17.7.0

# Execute step by step sequentially
sudo yum install -y gitlab-ce-15.11.13-ce.0.el7
sudo yum install -y gitlab-ce-16.0.8-ce.0.el7
sudo yum install -y gitlab-ce-16.3.7-ce.0.el7
sudo yum install -y gitlab-ce-16.7.9-ce.0.el7
sudo yum install -y gitlab-ce-16.11.10-ce.0.el7
sudo yum install -y gitlab-ce-17.0.8-ce.0.el7
sudo yum install -y gitlab-ce-17.3.7-ce.0.el7
sudo yum install -y gitlab-ce-17.5.0-ce.0.el7
sudo yum install -y gitlab-ce-17.7.0-ce.0.el7

# Just running yum install will automatically execute reconfigure and restart. Each step takes about 5 minutes.

# Execute step by step sequentially
sudo yum install -y gitlab-ce-15.11.13-ce.0.el7
sudo yum install -y gitlab-ce-16.0.8-ce.0.el7
sudo yum install -y gitlab-ce-16.3.7-ce.0.el7
sudo yum install -y gitlab-ce-16.7.9-ce.0.el7
sudo yum install -y gitlab-ce-16.11.10-ce.0.el7
sudo yum install -y gitlab-ce-17.0.8-ce.0.el7
sudo yum install -y gitlab-ce-17.3.7-ce.0.el7
sudo yum install -y gitlab-ce-17.5.0-ce.0.el7
sudo yum install -y gitlab-ce-17.7.0-ce.0.el7

# Just running yum install will automatically execute reconfigure and restart. Each step takes about 5 minutes.

Phase 2: Prepare Ubuntu 24.04 Server

Disk Mount

Mounted an object storage of 300GB capacity to /data.

# Create partition
sudo parted /dev/vdb --script mklabel gpt mkpart primary ext4 0% 100%

# Create filesystem (took quite a long time.)
sudo mkfs.ext4 /dev/vdb1

# Mount
sudo mkdir -p /data
sudo mount /dev/vdb1 /data

# Set permanent mount
echo '/dev/vdb1 /data ext4 defaults 0 2' | sudo tee -a /etc/fstab

# Create partition
sudo parted /dev/vdb --script mklabel gpt mkpart primary ext4 0% 100%

# Create filesystem (took quite a long time.)
sudo mkfs.ext4 /dev/vdb1

# Mount
sudo mkdir -p /data
sudo mount /dev/vdb1 /data

# Set permanent mount
echo '/dev/vdb1 /data ext4 defaults 0 2' | sudo tee -a /etc/fstab

Install GitLab 17.7.0

Only versions 17.1.0 and above can be installed on Ubuntu 24.04. Installed version 17.7.0, which is the same version as CentOS.

# Required packages
sudo apt update
sudo apt install -y curl openssh-server ca-certificates tzdata perl

# Add GitLab repository
curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | sudo bash

# Install the same version
sudo apt install gitlab-ce=17.7.0-ce.0

# Required packages
sudo apt update
sudo apt install -y curl openssh-server ca-certificates tzdata perl

# Add GitLab repository
curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | sudo bash

# Install the same version
sudo apt install gitlab-ce=17.7.0-ce.0

Phase 3: Data Migration

gitlab-backup Failure

Attempted to back up from CentOS, but encountered an issue.

Out of memory, malloc failed (tried to allocate 386278394 bytes)

Out of memory, malloc failed (tried to allocate 386278394 bytes)

The server had no memory available, and the backup of the repository, which amounted to 135GB, failed.
~~(Waited a long time...)~~

It wasn't just a memory issue. Even after adding 8GB of swap memory, ultimately, there was also insufficient disk space, resulting in a failure to create the backup file.

Solution: rsync + pg_dump Method

Instead of backing up, we decided to transfer the data directly.

# After confirming SSH connection from CentOS to Ubuntu

# Transfer configuration files
scp /etc/gitlab/gitlab.rb root@UBUNTU_IP:/data/gitlab-config/
scp /etc/gitlab/gitlab-secrets.json root@UBUNTU_IP:/data/gitlab-config/

# Transfer repository data (135GB, takes about 1 hour)
rsync -avz --progress /data/git-data/ root@UBUNTU_IP:/data/git-data/

# Transfer PostgreSQL data
rsync -avz --progress /var/opt/gitlab/postgresql/ root@UBUNTU_IP:/data/postgresql/

# After confirming SSH connection from CentOS to Ubuntu

# Transfer configuration files
scp /etc/gitlab/gitlab.rb root@UBUNTU_IP:/data/gitlab-config/
scp /etc/gitlab/gitlab-secrets.json root@UBUNTU_IP:/data/gitlab-config/

# Transfer repository data (135GB, takes about 1 hour)
rsync -avz --progress /data/git-data/ root@UBUNTU_IP:/data/git-data/

# Transfer PostgreSQL data
rsync -avz --progress /var/opt/gitlab/postgresql/ root@UBUNTU_IP:/data/postgresql/

DB Data Not Recognized

Copied PostgreSQL data with rsync, but when logging into GitLab failed, I checked and found that the data had not been transferred. A new DB was created during reconfigure.

# Create dump from CentOS
cd /tmp
sudo -u gitlab-psql /opt/gitlab/embedded/bin/pg_dump \
  -h /var/opt/gitlab/postgresql gitlabhq_production > /tmp/gitlab_db.sql

# Transfer to Ubuntu
scp /tmp/gitlab_db.sql root@UBUNTU_IP:/tmp/

# Restore on Ubuntu
sudo gitlab-ctl stop puma
sudo gitlab-ctl stop sidekiq

sudo gitlab-psql -d postgres -c "DROP DATABASE gitlabhq_production;"
sudo gitlab-psql -d postgres -c "CREATE DATABASE gitlabhq_production OWNER gitlab;"
sudo gitlab-psql -d gitlabhq_production < /tmp/gitlab_db.sql

sudo gitlab-ctl start

# Create dump from CentOS
cd /tmp
sudo -u gitlab-psql /opt/gitlab/embedded/bin/pg_dump \
  -h /var/opt/gitlab/postgresql gitlabhq_production > /tmp/gitlab_db.sql

# Transfer to Ubuntu
scp /tmp/gitlab_db.sql root@UBUNTU_IP:/tmp/

# Restore on Ubuntu
sudo gitlab-ctl stop puma
sudo gitlab-ctl stop sidekiq

sudo gitlab-psql -d postgres -c "DROP DATABASE gitlabhq_production;"
sudo gitlab-psql -d postgres -c "CREATE DATABASE gitlabhq_production OWNER gitlab;"
sudo gitlab-psql -d gitlabhq_production < /tmp/gitlab_db.sql

sudo gitlab-ctl start

Make sure to transfer the gitlab-secrets.json containing the decryption key as well.

scp /etc/gitlab/gitlab-secrets.json root@UBUNTU_IP:/etc/gitlab/

# Restart on Ubuntu
sudo gitlab-ctl stop
sudo chmod 600 /etc/gitlab/gitlab-secrets.json
sudo gitlab-ctl reconfigure
sudo gitlab-ctl start

scp /etc/gitlab/gitlab-secrets.json root@UBUNTU_IP:/etc/gitlab/

# Restart on Ubuntu
sudo gitlab-ctl stop
sudo chmod 600 /etc/gitlab/gitlab-secrets.json
sudo gitlab-ctl reconfigure
sudo gitlab-ctl start

gitlab-secrets.json contains the keys to decrypt encrypted data (passwords, 2FA, etc.). Without this, it is impossible to log in to existing accounts.

Phase 4: Upgrade to Latest Version on Ubuntu (17.7.0 -> 18.10.0)

Upgrade Path

#17.7.0 → 17.8.x → 17.11.x → 18.0.x → 18.2.x → 18.5.x → 18.8.x → 18.10.0

sudo apt install gitlab-ce=17.8.7-ce.0
sudo apt install gitlab-ce=17.11.7-ce.0
# Modify gitlab.rb (git_data_dirs → gitaly configuration)
sudo gitlab-ctl reconfigure
sudo apt install gitlab-ce=18.0.2-ce.0
sudo apt install gitlab-ce=18.2.2-ce.0
sudo apt install gitlab-ce=18.5.2-ce.0
sudo apt install gitlab-ce=18.8.6-ce.0
sudo apt install gitlab-ce=18.10.0-ce.0

#17.7.0 → 17.8.x → 17.11.x → 18.0.x → 18.2.x → 18.5.x → 18.8.x → 18.10.0

sudo apt install gitlab-ce=17.8.7-ce.0
sudo apt install gitlab-ce=17.11.7-ce.0
# Modify gitlab.rb (git_data_dirs → gitaly configuration)
sudo gitlab-ctl reconfigure
sudo apt install gitlab-ce=18.0.2-ce.0
sudo apt install gitlab-ce=18.2.2-ce.0
sudo apt install gitlab-ce=18.5.2-ce.0
sudo apt install gitlab-ce=18.8.6-ce.0
sudo apt install gitlab-ce=18.10.0-ce.0

Change git_data_dirs Setting (18.0)

The git_data_dirs setting was removed starting from 18.0.

# Previous (17.x)
git_data_dirs({"default" => {"path" => "/data/git-data"}})

# Changed (18.0+)
gitaly['configuration'] = {
  storage: [
    {
      name: 'default',
      path: '/data/git-data/repositories'
    }
  ]
}

# Previous (17.x)
git_data_dirs({"default" => {"path" => "/data/git-data"}})

# Changed (18.0+)
gitaly['configuration'] = {
  storage: [
    {
      name: 'default',
      path: '/data/git-data/repositories'
    }
  ]
}

Let's Encrypt Certificate Configuration

# /etc/gitlab/gitlab.rb
external_url 'https://git.example.com'
letsencrypt['enable'] = true
letsencrypt['auto_renew'] = true

# /etc/gitlab/gitlab.rb
external_url 'https://git.example.com'
letsencrypt['enable'] = true
letsencrypt['auto_renew'] = true

Initially, the certificate issuance failed, but it was resolved by first getting it to work normally over http and then switching to https.

Final Results

Item	Before	After
OS	CentOS 7 (EOL)	Ubuntu 24.04 LTS
GitLab	15.8.1	18.10.0
Disk	135GB used	300GB allocated
Total Upgrade Steps	-	About 15 steps