dimanche 20 mai 2018

One Plus X E1001 and LineageOS

Installing LineageOS on a One Plus X E1001 (not E1003, I believe E1001 is an Asian version whereas E1003 was old in Europe) was a bit painful. This article will not repeat all instructions to install LineageOS on One plus X (which can be found here: https://wiki.lineageos.org/devices/onyx/install) but only cover some unexpected behaviors I need to circumvent to install LineageOS.

First no recent (3+) TWRP version was running fine: all refused to boot (fastboot boot path/to/recovery/file.img) with the following error message:

Remote: dtb not found error

I had to use an old TWRP 2.8.7 version to make it work. Here is the version that worked fine for me: twrp-

Then, the LineageOS zip file could not install because of the following error message:

Comparing TZ version TZ.BF.2.0-2.0.0137 to TZ.BF.2.0-2.0.0134
assert failed: oppo.verify_trustzone("TZ.BF.2.0-2.0.0137") == "1"
E:Installation aborted.

To fix this one, I had to first flash the firmware/bootloader with a more recent version. Installing the following file with TWRP did the trick: OPXBLUpdateOOS3.1.zip.

After that, LineageOS (one of the nightly version of May 2018) installed just fine.

I hope this will help some people facing the "dtb not found" and "assert failed: oppo.verify_trustzone" error messages!

samedi 12 mai 2018

HAProxy et compression Gzip

Pour accélérer l'accès à vos sites et services web propulsées par HAProxy, il est facile d'activer la compression automatique des contenus texte comme HTML, CSS, JS (s'ils ne le sont pas déjà par le serveur sous-jacent).

Rien de plus simple, il suffit d'ajouter les 2 lignes suivantes dans la section defaults de la configuration d'HAProxy :

compression algo gzip
compression type text/html text/plain text/xml text/css text/javascript application/javascript application/json text/json

et redémarrer HAProxy par une commande comme

/etc/init.d/haproxy restart

" Avec cette fonctionnalité activée, vous devez désormais voir dans les en-têtes des transmissions entre le serveur web et le navigateur que le contenu est compressé avec gzip (vérification facile avec le module "Développeur" de Firefox par exemple, onglet "Réseau") :

Content-Encoding: gzip

dimanche 6 mai 2018

HAProxy in front of Unicorn

Unicorn, the "Rack HTTP server for fast clients", is really for "fast clients". Fast clients mean clients on the same local network (or even same machine) with very low latency and near-zero chance of packets loss. Slow clients are the usual web users: far away from the server. Unicorn has been designed to serve fast clients and if you try to use it to serve slow clients then the performance may be dismal. This is why the Unicorn developers recommend to use Nginx in front of Unicorn. To know more on the Unicorn philosophy, visit this page.

I recently tried to use HAProxy in front of Unicorn and was disappointed to see that:

  • the system was slow and unresponsive
  • a lot of 502 Gateway errors popped up for seemingly no reason (and this popped up unconsistently)

I came to the conclusion that the default configuration of HAProxy was not appropriate for Unicorn. After some web digging, I discovered the "http-buffer-request" option.

Here is what the HAProxy 1.8 documentation says about the "http-buffer-request" option :

It is sometimes desirable to wait for the body of an HTTP request before taking a decision. This is what is being done by "balance url_param" for example. The first use case is to buffer requests from slow clients before connecting to the server. Another use case consists in taking the routing decision based on the request body's contents. This option placed in a frontend or backend forces the HTTP processing to wait until either the wholebody is received, or the request buffer is full, or the first chunk is complete in case of chunked encoding. It can have undesired side effects with some applications abusing HTTP by expecting unbuffered transmissions between the frontend and the backend, so this should definitely not be used by default.

It seems to be the best fit with Unicorn's philosophy! Let's activate the option in each backend corresponding to a Unicorn-run application:

backend unicorn-app
	option http-buffer-request
	server unicorn-app

and restart HAProxy:

/etc/init.d/haproxy reload

mardi 17 avril 2018

Atheros AR9271 and Debian Stretch

I was expecting an easy use of my brand new Wifi adapter AWUS036NHA under Debian Stretch but I hit an unexpected bug...

I started with

apt install firmware-atheros

and soon the blue LED on the Alfa device was blinking furiously. So good so far! But when I tried to connect to a Wifi network with Network-Manager the tool was unable to connect. And no matter what I did it failed to connect. This is what I could see in the system logs (dmesg):

[  217.118991] usb 1-3: USB disconnect, device number 5
[  219.137772] usb 1-3: new high-speed USB device number 6 using xhci_hcd
[  219.294368] usb 1-3: New USB device found, idVendor=0cf3, idProduct=9271
[  219.294373] usb 1-3: New USB device strings: Mfr=16, Product=32, SerialNumber=48
[  219.294376] usb 1-3: Product: UB91C
[  219.294378] usb 1-3: Manufacturer: ATHEROS
[  219.294380] usb 1-3: SerialNumber: 12345
[  219.295099] usb 1-3: ath9k_htc: Firmware ath9k_htc/htc_9271-1.4.0.fw requested
[  219.295555] usb 1-3: firmware: direct-loading firmware ath9k_htc/htc_9271-1.4.0.fw
[  219.578229] usb 1-3: ath9k_htc: Transferred FW: ath9k_htc/htc_9271-1.4.0.fw, size: 51008
[  219.829806] ath9k_htc 1-3:1.0: ath9k_htc: HTC initialized with 33 credits
[  220.096217] ath9k_htc 1-3:1.0: ath9k_htc: FW Version: 1.4
[  220.096221] ath9k_htc 1-3:1.0: FW RMW support: On
[  220.096224] ath: EEPROM regdomain: 0x833a
[  220.096225] ath: EEPROM indicates we should expect a country code
[  220.096227] ath: doing EEPROM country->regdmn map search
[  220.096229] ath: country maps to regdmn code: 0x37
[  220.096231] ath: Country alpha2 being used: GB
[  220.096232] ath: Regpair used: 0x37
[  220.101582] ieee80211 phy1: Atheros AR9271 Rev:1
[  220.115019] ath9k_htc 1-3:1.0 wlx00c0ca9751d9: renamed from wlan1
[  220.148280] IPv6: ADDRCONF(NETDEV_UP): wlx00c0ca9751d9: link is not ready
[  220.332215] IPv6: ADDRCONF(NETDEV_UP): wlx00c0ca9751d9: link is not ready
[  220.578623] IPv6: ADDRCONF(NETDEV_UP): wlx00c0ca9751d9: link is not ready
[  220.642308] IPv6: ADDRCONF(NETDEV_UP): wlx00c0ca9751d9: link is not ready
[  255.566737] IPv6: ADDRCONF(NETDEV_UP): wlx00c0ca9751d9: link is not ready
[  259.745782] IPv6: ADDRCONF(NETDEV_UP): wlx00c0ca9751d9: link is not ready
[  259.986747] IPv6: ADDRCONF(NETDEV_UP): wlx00c0ca9751d9: link is not ready
[  265.841754] IPv6: ADDRCONF(NETDEV_UP): wlx00c0ca9751d9: link is not ready
[  276.133407] wlx00c0ca9751d9: authenticate with ec:08:6b:33:71:ba
[  276.391552] wlx00c0ca9751d9: send auth to ec:08:6b:33:71:ba (try 1/3)
[  276.394687] wlx00c0ca9751d9: authenticated
[  281.389219] wlx00c0ca9751d9: aborting authentication with ec:08:6b:33:71:ba by local choice (Reason: 3=DEAUTH_LEAVING)
[  282.900769] wlx00c0ca9751d9: authenticate with ec:08:6b:33:71:ba
[  283.164429] wlx00c0ca9751d9: send auth to ec:08:6b:33:71:ba (try 1/3)
[  283.167350] wlx00c0ca9751d9: authenticated

wlx00c0ca9751d9 is the name the system gave to the adapter and I was intrigued with this error line:

aborting authentication with ec:08:6b:33:71:ba by local choice (Reason: 3=DEAUTH_LEAVING)

After looking up on the web, I discovered that other users of Atheros firmware reported issue when the adapter was given a long name...

So I decided to teach udev to give a shorter name to the wifi device, for example wlan1. And by the way it would be easier to remember!

The first step was to detect the attributes seen by udev on the device. The right command to do so is:

udevadm info -a -p /sys/class/net/wlx00c0ca9751d9

Among many other attributes, I spotted quite precise ones:


Then I opened /etc/udev/rules.d/70-persistent-net.rules and added this line:

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTRS{manufacturer}=="ATHEROS", ATTRS{product}=="UB91C", NAME="wlan1"

After restarting udev with

/etc/init.d/udev restart

I unplugged and then plugged back the Wifi adapter and it was recognized as wlan1. And then the Wifi connection could be established.

Complicated but quite effective fix!

lundi 12 mars 2018

Utiliser HAProxy pour profiter d'HTTP/2

Les versions 1.8.x d'HAProxy (premier représentant de la branche publié fin 2017) supportent le protocole HTTP/2 pour la communication frontale (section frontend). L'utiliser en amont de votre infrastructure est un moyen facile de rendre ce protocole disponible même si certains de vos serveurs sous-jacents (backend) n'en sont pas encore capables.

Installer HAProxy 1.8.x sur Debian Stretch

La version distribuée officiellement dans Debian Stretch est 1.7.x. Pour installer une version de la branche 1.8.x, il y a au moins 2 alternatives :

  1. compiler l'outil à partir du code source disponible sur le site officiel
  2. utiliser https://haproxy.debian.net/ qui va fournir un dépôt spécifique

Configurer HAProxy

La configuration d'HAProxy se fait par défaut dans /etc/haproxy/haproxy.cfg.

Voici un exemple de fichier de configuration dont les principales sections sont commentées plus bas :

	log /dev/log	local0
	log /dev/log	local1 notice
	chroot /var/lib/haproxy
	stats socket /run/haproxy/admin.sock mode 660 level admin
	stats timeout 30s
	user haproxy
	group haproxy

	# Default SSL material locations
	ca-base /etc/ssl/certs
	crt-base /etc/ssl/private

	# Default ciphers offered by Mozilla here:
	#  https://mozilla.github.io/server-side-tls/ssl-config-generator
    ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
    ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

	log	global
	mode	http
	option	httplog
	option	dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
	errorfile 400 /etc/haproxy/errors/400.http
	errorfile 403 /etc/haproxy/errors/403.http
	errorfile 408 /etc/haproxy/errors/408.http
	errorfile 500 /etc/haproxy/errors/500.http
	errorfile 502 /etc/haproxy/errors/502.http
	errorfile 503 /etc/haproxy/errors/503.http
	errorfile 504 /etc/haproxy/errors/504.http

frontend http-in
	bind :::80 v4v6

	acl host_letsencrypt path_beg /.well-known/acme-challenge/
	use_backend letsencrypt if host_letsencrypt

	redirect scheme https code 301 if !host_letsencrypt
frontend https-in
	bind :::443 v4v6 ssl crt-list /etc/haproxy/crt-list.txt alpn h2,http/1.1
	option forwardfor
	http-request add-header X-Forwarded-Proto https

        # Define hosts
        acl host_alpha hdr_end(host) -i a.fr
	acl host_beta hdr(host) -i precis.b.fr
	acl host_gamma hdr_reg(host) ^d1.c.fr|d2.c.fr$

        # figure out which one to use
        use_backend alpha if host_alpha
	use_backend beta if host_beta
	use_backend gamma if host_gamma

	default_backend iota

backend alpha
        server alpha

backend beta
        server beta

backend gamma
        server gamma

backend iota
        server iota

backend letsencrypt
        server letsencrypt

Quelques commentaires pour bien comprendre...

On écoute d'abord en HTTP (donc sur le port 80, en IPv4 et IPv6) et on redirige tout vers l'HTTPS (redirect scheme https code 301) car c'est une bonne pratique aujourd'hui de n'offrir que du contenu protégé par SSL/TLS.

frontend http-in
	bind :::80 v4v6
	redirect scheme https code 301

Si l'on utilise letsencrypt pour générer ses certificats et qu'ils sont tous générés sur une machine dédiée pour ne pas avoir à se poser de questions sur les différents types de serveurs sous-jacents, on peut rajouter une exception pour les appels utilisés dans l'ACME challenge de letsencrypt et rediriger alors le trafic vers une machine spécifique ici nommée letsencrypt.

frontend http-in
	bind :::80 v4v6

	acl host_letsencrypt path_beg /.well-known/acme-challenge/
	use_backend letsencrypt if host_letsencrypt

	redirect scheme https code 301 if !host_letsencrypt

On écoute bien sûr également sur le port 443 pour gérer les connexions en SSL/TLS :

frontend https-in
	bind :::443 v4v6 ssl crt-list /etc/haproxy/crt-list.txt alpn h2,http/1.1
	option forwardfor
	http-request add-header X-Forwarded-Proto https

Il y a beaucoup de choses intéressantes ici :

  • bind :::443 v4v6 : IPv4 et IPv6 activés
  • bind :::443 v4v6 ssl crt-list /etc/haproxy/crt-list.txt : on active le SSL/TLS et on spécifie un fichier texte qui contient la liste des certificats à charger pour chiffrer les communications
  • bind :::443 v4v6 ssl crt-list /etc/haproxy/crt-list.txt alpn h2,http/1.1 : on active l'HTTP/2 (en plus de l'HTTP/1)
  • option forwardfor : comme on fonctionne en reverse proxy, on demande à HAProxy de passer les en-têtes X-FORWARDED-FOR aux serveurs sous-jacents (utile pour avoir les vraies adresses IP appelantes dans les logs)
  • http-request add-header X-Forwarded-Proto https : on informe les serveurs sous-jacents que la communication est bien chiffrée (et on évite donc une boucle de redirection si le serveur sous-jacent veut forcer le passage en HTTPS, avec ce paramètre il sera déjà satisfait)

On définit ensuite quelques règles ACL pour, selon le nom d'hôte, orienter la connexion vers un backend ou autre. J'ai mis ici plusieurs exemples que j'utilise - il existe des dizaines d'autres filtres y compris sur d'autres critères que le nom d'hôte (on a vu l'URL dans le cas de letsencrypt un peu plus) : filtre sur la fin du nom d'hôte (tout nom d'hôte finissant par a.fr sera redirigé vers le backend alpha), filtre sur le nom d'hôte complet (precis.b.fr sera envoyé vers beta) ou filtre sur le nom d'hôte avec des expressions régulières.

        # Define hosts
        acl host_alpha hdr_end(host) -i a.fr
	acl host_beta hdr(host) -i precis.b.fr
	acl host_gamma hdr_reg(host) ^d1.c.fr|d2.c.fr$

        # figure out which one to use
        use_backend alpha if host_alpha
	use_backend beta if host_beta
	use_backend gamma if host_gamma

On définit également un backend par défaut (iota ici) :

	default_backend iota

Et il reste alors à définir chaque backend :

backend example
        server example

Pour aller plus loin

Les options possibles dans HAProxy sont fort évidemment beaucoup plus nombreuses ! La documentation est très détaillée et claire ici https://www.haproxy.org/#docs.

HAProxy et IPv6

Pour activer IPv6 sur HAProxy, le plus simple est de modifier la directive bind dans votre fichier de configuration (par ex. /etc/haproxy/haproxy.cfg) pour devenir :

bind :::80 v4v6


bind :::443 v4v6

Il y a 3 fois le caractère ":" car "::" correspond à toute adresse IPv4 ou IPv6, et le troisième ":" sépare l'adresse IP du port.

- page 1 de 14