Gitlab is a wonderful piece of open source software, incredibly pleasant to use to manage development projects. My own instance, installed from source, is updated version after version. Today, I was faced with a weird issue:
- all downloads of archive (whatever the format .tar.gz, .zip, .tar.bz2) of the files of the projets were failing, more precisely only empty (0 byte) archives were returned
- when cloning with git clone http://urlofgitlab/group/repo.git, I consistently obtained warning: You appear to have cloned an empty repository.
- interestingly, cloning the same repository with git clone git@urlofgitlab:group/repo.git worked seamlessly.
After some research, it appeared my Gitlab instance was not using Gitlab-workhorse at all. The magic unicorn was the only one serving the content of the instance without any help from the local workhorse :-)
It appears that Gitlab-workhorse was developed and added to Gitlab 8 to circumvent some limitations of Unicorn when serving large files (some history here)... and since then big files would not be served anymore by Unicorn.
As a consequence, if the requests are not treated by Gitlab-workhorse, then the git clone over HTTP and download of large archive files would not complete.
How did it happen?
My instance is regularly updated from version to version and pre-dates Gitlab 8. Before Gitlab 8, it was normal to have my reverse proxy/load balancer (Pound) point directly to the Unicorn server. When upgrading to Gitlab 8, I should have changed the setting of the reverse proxy/load balancer to point to Gitlab-workhorse instead of Unicorn. And then it was necessary to properly set Gitlab-workhorse to rely on Unicorn.
How fix it?
Well, 3 steps.
Step 1: fix the link between Gitlab-workhorse and Unicorn
Gitlab-workhorse expects to connect to Unicorn through a Unix socket. It is therefore necessary to make sure that Unicorn is set up accordingly in /home/git/gitlab/config/unicorn.rb, with this line active:
listen "/home/git/gitlab/tmp/sockets/gitlab.socket", :backlog => 1024
Step 2: make sure that Gitlab-workhorse is well set to connect to this socket. This can be done by tweaking the parameters in /etc/default/gitlab, with inspiration from /home/git/gitlab/lib/support/init.d/gitlab.default.example.
Step 3: make sure that the reverse proxy correctly points to workhorse. As a default, Gitlab-workhorse uses a socket. In my case, I had to make it use a TCP connection/port so that the reverse proxy could use it. Again, based on the settings found in /home/git/gitlab/lib/support/init.d/gitlab.default.example, I tweaked the /etc/default/gitlab file to read:
gitlab_workhorse_options="-listenUmask 0 -listenNetwork tcp -listenAddr a.b.c.d:8181 -authBackend http://127.0.0.1:8080 -authSocket $socket_path/gitlab.socket -documentRoot $app_root/public"
Last, with the reverse proxy pointing to a.b.c.d:8181 everything worked very fine.
I am relieved to know that my Unicorn is now so efficiently supported by the Workhorse!