Mot-clé - unicorn

Fil des billets

dimanche 6 mai 2018

HAProxy in front of Unicorn

Unicorn, the "Rack HTTP server for fast clients", is really for "fast clients". Fast clients mean clients on the same local network (or even same machine) with very low latency and near-zero chance of packets loss. Slow clients are the usual web users: far away from the server. Unicorn has been designed to serve fast clients and if you try to use it to serve slow clients then the performance may be dismal. This is why the Unicorn developers recommend to use Nginx in front of Unicorn. To know more on the Unicorn philosophy, visit this page.

I recently tried to use HAProxy in front of Unicorn and was disappointed to see that:

  • the system was slow and unresponsive
  • a lot of 502 Gateway errors popped up for seemingly no reason (and this popped up unconsistently)

I came to the conclusion that the default configuration of HAProxy was not appropriate for Unicorn. After some web digging, I discovered the "http-buffer-request" option.

Here is what the HAProxy 1.8 documentation says about the "http-buffer-request" option :

It is sometimes desirable to wait for the body of an HTTP request before taking a decision. This is what is being done by "balance url_param" for example. The first use case is to buffer requests from slow clients before connecting to the server. Another use case consists in taking the routing decision based on the request body's contents. This option placed in a frontend or backend forces the HTTP processing to wait until either the wholebody is received, or the request buffer is full, or the first chunk is complete in case of chunked encoding. It can have undesired side effects with some applications abusing HTTP by expecting unbuffered transmissions between the frontend and the backend, so this should definitely not be used by default.

It seems to be the best fit with Unicorn's philosophy! Let's activate the option in each backend corresponding to a Unicorn-run application:

backend unicorn-app
	option http-buffer-request
	server unicorn-app

and restart HAProxy:

/etc/init.d/haproxy reload

samedi 22 octobre 2016

Init script for Rails app served by Unicorn with RVM

Here is an init script for a Rails app served by Unicorn with RVM.

# Provides:          APP_NAME, with Unicorn serving
# Required-Start:    $all
# Required-Stop:     $network $local_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Start APP_NAME unicorn at boot
# Description:       Enable APP_NAME at boot time.

set -u
set -e

# Change these to match your app:

UNICORN_OPTS="-D -E $ENV -c $APP_ROOT/config/unicorn.rb"

SET_PATH="cd $APP_ROOT; rvm use $RVM > /dev/null;"


cd $APP_ROOT || exit 1

sig () {
	test -s "$PID" && kill -$1 `cat $PID`

oldsig () {
	test -s $old_pid && kill -$1 `cat $old_pid`

start () {
        echo "Starting $APP_NAME unicorn..."
        sig 0 && echo -e >&2 "\e[31mAlready running" && exit 0
        su - $USER -c "$CMD" > /dev/null
        echo -e "$APP_NAME has \e[32mstarted\e[0m, PID is `cat $PID`"

stop () {
        echo "Stopping $APP_NAME unicorn (signal QUIT)..."
        sig QUIT && echo -e "$APP_NAME has \e[32mstopped" && exit 0
        echo >&2 "Not running"

case ${1-help} in
	echo "Force stopping $APP_NAME unicorn (signal TERM)..."
	sig TERM && echo "$APP_NAME has stopped" &&exit 0
	echo -e >&2 "\e[31mNot running"
	echo "Reloading $APP_NAME unicorn (signal USR2)..."
	sig USR2 && echo -e "$APP_NAME has \e[32mreloaded" && exit 0
	echo -e >&2 "\e[31mCouldn't reload\e[0m, starting instead"
	sig 0 && echo -e "$APP_NAME \e[32mrunning\e[0m with PID `cat $PID`" && exit 0
	echo -e "$APP_NAME is \e[31mnot running!"
 	echo >&2 "Usage: $0 <start|stop|reload|status|force-stop>"
 	exit 0

lundi 17 octobre 2016

Gitlab, empty repository mystery, when a workhorse comes to help a unicorn!

Gitlab is a wonderful piece of open source software, incredibly pleasant to use to manage development projects. My own instance, installed from source, is updated version after version. Today, I was faced with a weird issue:

  1. all downloads of archive (whatever the format .tar.gz, .zip, .tar.bz2) of the files of the projets were failing, more precisely only empty (0 byte) archives were returned
  2. when cloning with git clone http://urlofgitlab/group/repo.git, I consistently obtained warning: You appear to have cloned an empty repository.
  3. interestingly, cloning the same repository with git clone git@urlofgitlab:group/repo.git worked seamlessly.

After some research, it appeared my Gitlab instance was not using Gitlab-workhorse at all. The magic unicorn was the only one serving the content of the instance without any help from the local workhorse :-)

Some context

It appears that Gitlab-workhorse was developed and added to Gitlab 8 to circumvent some limitations of Unicorn when serving large files (some history here)... and since then big files would not be served anymore by Unicorn.

As a consequence, if the requests are not treated by Gitlab-workhorse, then the git clone over HTTP and download of large archive files would not complete.

How did it happen?

My instance is regularly updated from version to version and pre-dates Gitlab 8. Before Gitlab 8, it was normal to have my reverse proxy/load balancer (Pound) point directly to the Unicorn server. When upgrading to Gitlab 8, I should have changed the setting of the reverse proxy/load balancer to point to Gitlab-workhorse instead of Unicorn. And then it was necessary to properly set Gitlab-workhorse to rely on Unicorn.

How fix it?

Well, 3 steps.

Step 1: fix the link between Gitlab-workhorse and Unicorn Gitlab-workhorse expects to connect to Unicorn through a Unix socket. It is therefore necessary to make sure that Unicorn is set up accordingly in /home/git/gitlab/config/unicorn.rb, with this line active:

listen "/home/git/gitlab/tmp/sockets/gitlab.socket", :backlog => 1024

Step 2: make sure that Gitlab-workhorse is well set to connect to this socket. This can be done by tweaking the parameters in /etc/default/gitlab, with inspiration from /home/git/gitlab/lib/support/init.d/gitlab.default.example.

Step 3: make sure that the reverse proxy correctly points to workhorse. As a default, Gitlab-workhorse uses a socket. In my case, I had to make it use a TCP connection/port so that the reverse proxy could use it. Again, based on the settings found in /home/git/gitlab/lib/support/init.d/gitlab.default.example, I tweaked the /etc/default/gitlab file to read:

gitlab_workhorse_options="-listenUmask 0 -listenNetwork tcp -listenAddr a.b.c.d:8181 -authBackend -authSocket $socket_path/gitlab.socket -documentRoot $app_root/public"

Last, with the reverse proxy pointing to a.b.c.d:8181 everything worked very fine.

I am relieved to know that my Unicorn is now so efficiently supported by the Workhorse!