PAB's blog

Gitlab is a wonderful piece of open source software, incredibly pleasant to use to manage development projects. My own instance, installed from source, is updated version after version. Today, I was faced with a weird issue:

1. all downloads of archive (whatever the format .tar.gz, .zip, .tar.bz2) of the files of the projets were failing, more precisely only empty (0 byte) archives were returned
2. when cloning with git clone http://urlofgitlab/group/repo.git, I consistently obtained warning: You appear to have cloned an empty repository.
3. interestingly, cloning the same repository with git clone git@urlofgitlab:group/repo.git worked seamlessly.

After some research, it appeared my Gitlab instance was not using Gitlab-workhorse at all. The magic unicorn was the only one serving the content of the instance without any help from the local workhorse :-)

Some context

It appears that Gitlab-workhorse was developed and added to Gitlab 8 to circumvent some limitations of Unicorn when serving large files (some history here)... and since then big files would not be served anymore by Unicorn.

As a consequence, if the requests are not treated by Gitlab-workhorse, then the git clone over HTTP and download of large archive files would not complete.

How did it happen?

My instance is regularly updated from version to version and pre-dates Gitlab 8. Before Gitlab 8, it was normal to have my reverse proxy/load balancer (Pound) point directly to the Unicorn server. When upgrading to Gitlab 8, I should have changed the setting of the reverse proxy/load balancer to point to Gitlab-workhorse instead of Unicorn. And then it was necessary to properly set Gitlab-workhorse to rely on Unicorn.

How fix it?

Well, 3 steps.

Step 1: fix the link between Gitlab-workhorse and Unicorn Gitlab-workhorse expects to connect to Unicorn through a Unix socket. It is therefore necessary to make sure that Unicorn is set up accordingly in /home/git/gitlab/config/unicorn.rb, with this line active:

listen "/home/git/gitlab/tmp/sockets/gitlab.socket", :backlog => 1024

Step 2: make sure that Gitlab-workhorse is well set to connect to this socket. This can be done by tweaking the parameters in /etc/default/gitlab, with inspiration from /home/git/gitlab/lib/support/init.d/gitlab.default.example.

Step 3: make sure that the reverse proxy correctly points to workhorse. As a default, Gitlab-workhorse uses a socket. In my case, I had to make it use a TCP connection/port so that the reverse proxy could use it. Again, based on the settings found in /home/git/gitlab/lib/support/init.d/gitlab.default.example, I tweaked the /etc/default/gitlab file to read:

gitlab_workhorse_options="-listenUmask 0 -listenNetwork tcp -listenAddr a.b.c.d:8181 -authBackend http://127.0.0.1:8080 -authSocket $socket_path/gitlab.socket -documentRoot $app_root/public"

Last, with the reverse proxy pointing to a.b.c.d:8181 everything worked very fine.

I am relieved to know that my Unicorn is now so efficiently supported by the Workhorse!