Improving GitLab CI efficiency with Wake-on-LAN

Disclaimer: This solution is imperfect but could serve many use cases well.
Limitations discussed below, YMMV 🙂

Running GitLab CI at home, for personal projects?

You may have the same situation as I did: a GitLab coordinator and runner together on a single home NAS. While this setup was never going to be the best performing, it was at first adequate, until increasingly demanding projects lifted build times upwards of 2 hours.

The QNAP TS-451: A great NAS,
but no beastly build machine

Clearly, a NAS, while perfectly specced for file-sharing applications, is not the best device to run demanding build tasks; and especially not the weighty Android tool-chain.

Meanwhile; a monster 4Ghz ‘gaming rig’ lay switched off in another room, unable to assist the overwhelmed NAS. The direct solution to this, would be to install the Gitlab runner on the gaming rig instead, allowing the NAS to be the coordinator only, and let the good times roll, right?

Except for one thing: power consumption.

A typical developers personal project contribution pattern

Most developers don’t contribute to our personal projects daily, or need to run CI builds that frequently. The out-of-the-box GitLab solution would mean running power hungry hardware 24/7, only for it to idle 99% of the time. This is just not desirable on a number of levels.

Enter Wake-on-LAN

This technology allows nearly any modern PC using wired network to be switched on from standby (off) state, when a ‘magic packet‘ is targeted at the adapter’s MAC address.

It seems reasonable that this could be used by GitLab; enabling the coordinator to wake up a runner only when it is needed to fulfil a pending job. There is indeed a GitLab feature request for this; but somewhat inexplicably (given what an obviously good idea it is) the work has not been taken up. Maybe the way GitLab handles runners would require significant rework, making this a less than trivial request to implement?

We can however ‘hack’ a solution to this using only the GitLab YAML config file and a few common Linux tools.

Solution

At a high level:

  1. User makes a Git commit
  2. GitLab initiates project build.
  3. PC wakes up.
  4. PC completes build.
  5. PC shuts down.

Since GitLab’s design (putting plugins and hooks aside) does not allow for execution of custom commands on the coordinator itself, we can use the existing GitLab runner on the NAS to execute the PC wake up (1) and shut down (5) jobs for us. So we can clarify our concept to:

  1. User makes a Git commit
  2. GitLab coordinator (on NAS) triggers build of a project.
  3. GitLab runner 1 (on NAS) wakes up the PC.
  4. GitLab runner 2 (on PC) executes one or more build jobs.
  5. GitLab runner 1 (on NAS) issues command to shut-down the PC.

To ensure that jobs 2 and 4 are directed to the NAS Runner, and jobs 2 and 3 at the PC Runner, we can use GitLab tags (not to be confused with Git tags): Simply; one or more free-text tags can be attributed both to the jobs of a project, and to runners.
A job will then only execute on a runner where at least one of the tags matches.

In our concept list above, 1 is performed by the user and 2 is a built in function of GitLab. The remaining jobs 3, 4 and 5 will be defined as GitLab jobs in our project – in the .gitlab-ci.yml file.

Let’s use the tag Small to match: (3) and (5) to the NAS Runner and the tag Large to match (4), the main build jobs, to the PC Runner.

  • Overall Concept
    • GitLab coordinator not designed to execute any Jobs itself
    • Label ‘Small’ and ‘Large’ runner vs. Jobs
    • Have the NAS ‘Small’ runner execute wake/shutdown tasks for ‘Large’
    • Main build tasks run only on ‘Large’
  • Set up the new Runner
    • Provision new Ubuntu installation
    • Create gitlab-runner user
    • Set-up GitLab Runner
  • Prove the WakeOnLAN concept
    • The wakeonlan Linux command line utility
    • Overcoming the Containerisation of the GitLab
    • Set up SSH key trust relationships