PAB's blog

lundi 18 juin 2018

Gitlab behind a reverse proxy handling https

Par PAB le lundi 18 juin 2018, 14:58

I share below my configuration of GitLab when served behind a reverse proxy (haproxy to name it) that is handling the HTTPS part of the communication (and is also load balancing). <Disclaimer> I am not claiming this is the best configuration or the only possible configuration but I can report that it works well</Disclaimer>.

Infrastructure

The network transaction is established as follows:

User <- HTTPS HTTP/2 -> HAPROXY <- HTTP -> NGINX <- HTTP -> Unicorn/GitLab

I will not detail here the configuration of HAProxy which is pretty classical.

Configuration of GitLab

In gitlab.yml (presumably in /home/git/gitlab/config/ if you installed from source), I have the following:

production: &base
    ## Web server settings (note: host is the FQDN, do not include http://)
    host: gitlab.domain.tld
    port: 443
    https: true

And yes I indicate https true and port 445 even if Gitlab itself does not handle the https part of the communication.

Configuration of Nginx

In the config of Nginx (another layer of reverse proxy), I have:

    proxy_set_header    X-Forwarded-Proto   https;

which differs from the configuration proposed by the Gitlab team. Indeed, it does not convey the protocol that is seen by Nginx but rather forces the header to https as initially the communication was https-secured.

And it works flawlessly!

Node et socket.io derrière Apache en Reverse Proxy

Par PAB le lundi 18 juin 2018, 12:03

Je viens de déployer une application NodeJS basée sur Angular et utilisant socket.io pour communiquer entre le "front-end" et le "back-end". Dans Firefox, je voyais toutefois ce message d'erreur :

Firefox can’t establish a connection to the server at wss://my.app/socket.io/?token=verylongtoken&EIO=3&transport=websocket&sid=verylongsid

et pourtant l'application fonctionnait parfaitement.

Après enquête, il semble que socket.io avait le bon goût de faire un "fallback" sur une méthode AJAX et s'accomodait donc du non établissement du WebSocket.

Pour régler proprement le problème, il fallait :

  • activer le module proxy_wstunnel d'Apache
a2enmod proxy_wstunnel
service apache2 restart
  • puis ajouter ces lignes dans la configuration (attention, le module rewrite d'Apache doit aussi être chargé !)
  RewriteEngine On
  RewriteCond %{REQUEST_URI}  ^/socket.io            [NC]
  RewriteCond %{QUERY_STRING} transport=websocket    [NC]
  RewriteRule /(.*)           ws://localhost:3000/$1 [P,L]

Et hop, tout était fonctionnel désormais !

dimanche 20 mai 2018

One Plus X E1001 and LineageOS

Par PAB le dimanche 20 mai 2018, 06:41

Installing LineageOS on a One Plus X E1001 (not E1003, I believe E1001 is an Asian version whereas E1003 was old in Europe) was a bit painful. This article will not repeat all instructions to install LineageOS on One plus X (which can be found here: https://wiki.lineageos.org/devices/onyx/install) but only cover some unexpected behaviors I need to circumvent to install LineageOS.

First no recent (3+) TWRP version was running fine: all refused to boot (fastboot boot path/to/recovery/file.img) with the following error message:

Remote: dtb not found error

I had to use an old TWRP 2.8.7 version to make it work. Here is the version that worked fine for me: twrp-2.8.7.0-6.0-onyx.img.

Then, the LineageOS zip file could not install because of the following error message:

Comparing TZ version TZ.BF.2.0-2.0.0137 to TZ.BF.2.0-2.0.0134
assert failed: oppo.verify_trustzone("TZ.BF.2.0-2.0.0137") == "1"
E:Installation aborted.

To fix this one, I had to first flash the firmware/bootloader with a more recent version. Installing the following file with TWRP did the trick: OPXBLUpdateOOS3.1.zip.

After that, LineageOS (one of the nightly version of May 2018) installed just fine.

I hope this will help some people facing the "dtb not found" and "assert failed: oppo.verify_trustzone" error messages!

samedi 12 mai 2018

HAProxy et compression Gzip

Par PAB le samedi 12 mai 2018, 18:00

Pour accélérer l'accès à vos sites et services web propulsées par HAProxy, il est facile d'activer la compression automatique des contenus texte comme HTML, CSS, JS (s'ils ne le sont pas déjà par le serveur sous-jacent).

Rien de plus simple, il suffit d'ajouter les 2 lignes suivantes dans la section defaults de la configuration d'HAProxy :

compression algo gzip
compression type text/html text/plain text/xml text/css text/javascript application/javascript application/json text/json

et redémarrer HAProxy par une commande comme

/etc/init.d/haproxy restart

" Avec cette fonctionnalité activée, vous devez désormais voir dans les en-têtes des transmissions entre le serveur web et le navigateur que le contenu est compressé avec gzip (vérification facile avec le module "Développeur" de Firefox par exemple, onglet "Réseau") :

Content-Encoding: gzip

dimanche 6 mai 2018

HAProxy in front of Unicorn

Par PAB le dimanche 6 mai 2018, 07:19

Unicorn, the "Rack HTTP server for fast clients", is really for "fast clients". Fast clients mean clients on the same local network (or even same machine) with very low latency and near-zero chance of packets loss. Slow clients are the usual web users: far away from the server. Unicorn has been designed to serve fast clients and if you try to use it to serve slow clients then the performance may be dismal. This is why the Unicorn developers recommend to use Nginx in front of Unicorn. To know more on the Unicorn philosophy, visit this page.

I recently tried to use HAProxy in front of Unicorn and was disappointed to see that:

  • the system was slow and unresponsive
  • a lot of 502 Gateway errors popped up for seemingly no reason (and this popped up unconsistently)

I came to the conclusion that the default configuration of HAProxy was not appropriate for Unicorn. After some web digging, I discovered the "http-buffer-request" option.

Here is what the HAProxy 1.8 documentation says about the "http-buffer-request" option :

It is sometimes desirable to wait for the body of an HTTP request before taking a decision. This is what is being done by "balance url_param" for example. The first use case is to buffer requests from slow clients before connecting to the server. Another use case consists in taking the routing decision based on the request body's contents. This option placed in a frontend or backend forces the HTTP processing to wait until either the wholebody is received, or the request buffer is full, or the first chunk is complete in case of chunked encoding. It can have undesired side effects with some applications abusing HTTP by expecting unbuffered transmissions between the frontend and the backend, so this should definitely not be used by default.

It seems to be the best fit with Unicorn's philosophy! Let's activate the option in each backend corresponding to a Unicorn-run application:

backend unicorn-app
	option http-buffer-request
	server unicorn-app 1.2.3.4:3000

and restart HAProxy:

/etc/init.d/haproxy reload

mardi 17 avril 2018

Atheros AR9271 and Debian Stretch

Par PAB le mardi 17 avril 2018, 21:31

I was expecting an easy use of my brand new Wifi adapter AWUS036NHA under Debian Stretch but I hit an unexpected bug...

I started with

apt install firmware-atheros

and soon the blue LED on the Alfa device was blinking furiously. So good so far! But when I tried to connect to a Wifi network with Network-Manager the tool was unable to connect. And no matter what I did it failed to connect. This is what I could see in the system logs (dmesg):

[  217.118991] usb 1-3: USB disconnect, device number 5
[  219.137772] usb 1-3: new high-speed USB device number 6 using xhci_hcd
[  219.294368] usb 1-3: New USB device found, idVendor=0cf3, idProduct=9271
[  219.294373] usb 1-3: New USB device strings: Mfr=16, Product=32, SerialNumber=48
[  219.294376] usb 1-3: Product: UB91C
[  219.294378] usb 1-3: Manufacturer: ATHEROS
[  219.294380] usb 1-3: SerialNumber: 12345
[  219.295099] usb 1-3: ath9k_htc: Firmware ath9k_htc/htc_9271-1.4.0.fw requested
[  219.295555] usb 1-3: firmware: direct-loading firmware ath9k_htc/htc_9271-1.4.0.fw
[  219.578229] usb 1-3: ath9k_htc: Transferred FW: ath9k_htc/htc_9271-1.4.0.fw, size: 51008
[  219.829806] ath9k_htc 1-3:1.0: ath9k_htc: HTC initialized with 33 credits
[  220.096217] ath9k_htc 1-3:1.0: ath9k_htc: FW Version: 1.4
[  220.096221] ath9k_htc 1-3:1.0: FW RMW support: On
[  220.096224] ath: EEPROM regdomain: 0x833a
[  220.096225] ath: EEPROM indicates we should expect a country code
[  220.096227] ath: doing EEPROM country->regdmn map search
[  220.096229] ath: country maps to regdmn code: 0x37
[  220.096231] ath: Country alpha2 being used: GB
[  220.096232] ath: Regpair used: 0x37
[  220.101582] ieee80211 phy1: Atheros AR9271 Rev:1
[  220.115019] ath9k_htc 1-3:1.0 wlx00c0ca9751d9: renamed from wlan1
[  220.148280] IPv6: ADDRCONF(NETDEV_UP): wlx00c0ca9751d9: link is not ready
[  220.332215] IPv6: ADDRCONF(NETDEV_UP): wlx00c0ca9751d9: link is not ready
[  220.578623] IPv6: ADDRCONF(NETDEV_UP): wlx00c0ca9751d9: link is not ready
[  220.642308] IPv6: ADDRCONF(NETDEV_UP): wlx00c0ca9751d9: link is not ready
[  255.566737] IPv6: ADDRCONF(NETDEV_UP): wlx00c0ca9751d9: link is not ready
[  259.745782] IPv6: ADDRCONF(NETDEV_UP): wlx00c0ca9751d9: link is not ready
[  259.986747] IPv6: ADDRCONF(NETDEV_UP): wlx00c0ca9751d9: link is not ready
[  265.841754] IPv6: ADDRCONF(NETDEV_UP): wlx00c0ca9751d9: link is not ready
[  276.133407] wlx00c0ca9751d9: authenticate with ec:08:6b:33:71:ba
[  276.391552] wlx00c0ca9751d9: send auth to ec:08:6b:33:71:ba (try 1/3)
[  276.394687] wlx00c0ca9751d9: authenticated
[  281.389219] wlx00c0ca9751d9: aborting authentication with ec:08:6b:33:71:ba by local choice (Reason: 3=DEAUTH_LEAVING)
[  282.900769] wlx00c0ca9751d9: authenticate with ec:08:6b:33:71:ba
[  283.164429] wlx00c0ca9751d9: send auth to ec:08:6b:33:71:ba (try 1/3)
[  283.167350] wlx00c0ca9751d9: authenticated

wlx00c0ca9751d9 is the name the system gave to the adapter and I was intrigued with this error line:

aborting authentication with ec:08:6b:33:71:ba by local choice (Reason: 3=DEAUTH_LEAVING)

After looking up on the web, I discovered that other users of Atheros firmware reported issue when the adapter was given a long name...

So I decided to teach udev to give a shorter name to the wifi device, for example wlan1. And by the way it would be easier to remember!

The first step was to detect the attributes seen by udev on the device. The right command to do so is:

udevadm info -a -p /sys/class/net/wlx00c0ca9751d9

Among many other attributes, I spotted quite precise ones:

ATTRS{manufacturer}=="ATHEROS"
ATTRS{product}=="UB91C"

Then I opened /etc/udev/rules.d/70-persistent-net.rules and added this line:

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTRS{manufacturer}=="ATHEROS", ATTRS{product}=="UB91C", NAME="wlan1"

After restarting udev with

/etc/init.d/udev restart

I unplugged and then plugged back the Wifi adapter and it was recognized as wlan1. And then the Wifi connection could be established.

Complicated but quite effective fix!

- page 1 de 14