% tar -jx < *.bz

Écrit le vendredi 8 novembre 2024.

Debian's APT accepts the syntax apt-get install 'some-package=1.32' to install an exact version of a package. TIL about apt-get satisfy, which lets you ask for a version range, as in:

apt-get satisfy 'some-package (>=1.30)'

I found it useful for a CI script fragment where I don't care if the executor runs Debian or Ubuntu, I only want to ensure that I'm using a recent enough version of some-package.

The syntax is the same as in the Depends: field of a control file. You pretty much have to use quotes to avoid the shell interpreting brackets or redirection characters. Several packages are separated with commas, as in apt-get satisfy 'foo (>=1.8), bar (> 2.3)'.

Remember to use apt-get in scripts, apt comes with the following note:

The apt(8) commandline is designed as an end-user tool and it may change behavior between versions. While it tries not to break backward compatibility this is not guaranteed either if a change seems beneficial for interactive use.

All features of apt(8) are available in dedicated APT tools like apt-get(8) and apt-cache(8) as well. apt(8) just changes the default value of some options (see apt.conf(5) and specifically the Binary scope). So you should prefer using these commands (potentially with some additional options enabled) in your scripts as they keep backward compatibility as much as possible.

Écrit le dimanche 3 novembre 2024.

I'm using a UK keyboard layout with dead keys to access accents. The inspiration is James Campbell's UK International keyboard, but I deviated a little bit, eg.: AltGr+o becomes œ, AltGr+a is æ, this makes typing « mon cœur » and "UNIX© dæmon" easier.

In xkeyboard-config's default layouts, the "English (UK)" layout puts the grave on the # hash sign and the circumflex on the @ key, which is a choice I don't like. The "English (UK, intl., with dead keys)" layout puts the dead keys on more sensible positions eg. grave on the ` key, but the dead keys are the primary keys, without an AltGr, which I find annoying. Hence making my own layout, this is the final result.

Écrit le dimanche 27 octobre 2024.

For $WORK I had to setup a Linux machine (a Raspberry Pi) so that right after boot, an interactive CLI tool would start, ready to receive input and show information.

Of course if you start a script at boot, its stdin is closed and stdout goes to the Journal. There exists openvt(1) to launch that script in a new VT (or to take over tty1). (But then you have to mind that openvt needs to run as root to open a new VT, so it should wrap sudo to drop privileges, and that get a bit fiddly.)

Then I happened upon the TTYPath setting, which does exactly what I need; and of course it combines perfectly fine with settings for User= or WorkingDirectory. I ended up with the following:

[Unit]
Description=Interactive script started on boot

[Install]
WantedBy=multi-user.target

[Service]
# The magic to have prompts displayed to the user
TTYPath=/dev/tty1
StandardInput=tty
StandardOutput=tty

User=someuser
WorkingDirectory=/opt/software/
ExecStart=/opt/software/bin/start_soft

# Always restart on exit, disable restart counter, delay a little bit
# the restart to avoid spam in case of repeating error
Restart=always
StartLimitIntervalSec=0
RestartSec=5s

It only occured to me to have a look at how getty.service is configured, and indeed it uses the same options:

StandardInput=tty
StandardOutput=tty
TTYPath=/dev/%I
TTYReset=yes          # reset $TTYPath before and after execution
TTYVHangup=yes        # disconnect all clients which have open $TTYPath
TTYVTDisallocate=yes  # deallocate before and after execution; this ensures screen and scrollback buffer is clear

A inconvenience is that if you start on tty1, then messages from systemd or the kernel get mixed up with your program output. If you start on tty2, I tried to add ExecStartPre=chvt 2, but the VT would switch back to tty1 depending on exact boot order.

Écrit le lundi 21 octobre 2024, modifié le dimanche 27 octobre 2024.

Par exercise (pour apprendre un peu de JavaScript), et un petit peu par utilité (devant résoudre des ambiguités autour de l'encodage de données au travail), j'ai créé une page pour explorer quelques jeux de charactères.

Écrit le vendredi 27 septembre 2024.

I want to import a file (maybe generated at run-time, maybe even the name only known at run-time) that does not exist in sys.path. In a couple of previous jobs that has happened with Python bindings generated from protocol descriptions. In the past I've done that by temporarily adding the parent to sys.path then running importlib.import_module, while being annoyed there is no function import_file(name: str, file_path: Path) -> ModuleType. It turns out that importlib.util has all the bits necessary, it's just a matter of putting them together:

import importlib.util
import os
import sys
from types import ModuleType

def import_file(module_name: str, file_path: os.PathLike) -> ModuleType:
    spec = importlib.util.spec_from_file_location(module_name, file_path)
    # spec: ModuleSpec(name='bar', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f30b1d10310>, origin='/tmp/tmp.WpcbUEkm32/foo.py')
    # if spec_from_file_location fails, it returns None rather than raise an exception
    if spec is None:
        raise ModuleNotFoundError(f"Can't import {module_name} from {file_path}")
    if spec.loader is None:
        raise ModuleNotFoundError(f"Can't import {module_name} from {file_path}")
    module = importlib.util.module_from_spec(spec)
    sys.modules[module_name] = module
    spec.loader.exec_module(module)
    return module

I suppose it is optional to add the module to sys.modules. If we do, that will allow future import module_name to find the same module (which might even be useful for the purpose of injecting different code into a 3rd party library…). If we don't, then we get a new copy of the module every time we import it again. Eg. if /tmp/tmp.WpcbUEkm32/foo.py defined a global:

MODULE_GOBAL = 1

def myname():
    return __name__

And I try to imort it twice under the same name, same path, the module globals are disconnected:

foo1 = import_file("foo", "/tmp/tmp.WpcbUEkm32/foo.py")
foo2 = import_file("foo", "/tmp/tmp.WpcbUEkm32/foo.py")

foo1.myname()
# 'foo'
foo2.myname()
# 'foo'

foo1.MODULE_GOBAL += 1
print(foo1.MODULE_GOBAL)
# 2, as expected
print(foo2.MODULE_GOBAL)
# 1, unmodified

Écrit le vendredi 9 août 2024, modifié le lundi 21 octobre 2024.

Python3 has bytes, which are sequences of 8-bit integers, and str, sequences of Unicode code-points. To go from one to the other, you need to encode or decode by giving an explicit encoding. There are many protocols where the markup is ASCII, even if the data is some other encoding that you don't know. If you know that other encoding is ASCII-compatible, it is useful to be able to parse, split etc. the markup, and you just need to pass-through the payload.

An initial search on the Internet brought up an article by Eric S. Raymond that touches on that, and suggests to decode the data as ISO-8859-1, handle it as str, then the payload can be losslessly recovered by reencoding it. The first 256 codepoints of Unicode are exactly the ISO-8859-1 codepoints (a bit more on that further down). As a result, the following is idempotent:

>>> by = bytes(range(0xff))
>>> by2 = bytes(str(by, encoding="iso-8859-1"), encoding="iso-8859-1")
>>> by == by2
True

A few days later I came across that article by Alyssa Coghlan, which mentions the same idea, and also the existence of the errors="surrogateencoding" (PEP 383) error handler (also: codecs in the Python documentation), which is designed to allow exactly what I needed:

>>> by = bytes(range(0xff))
>>> by2 = bytes(str(by, encoding="ascii", errors="surrogateescape"), encoding="ascii", errors="surrogateescape")
>>> by == by2
True

Alyssa Coghlan has some discussion about the merits of each approach, I can't really say that functionally they have any meaningful difference. As she points out, if you do anything but re-encode the non-ASCII compatible parts using the same codec, you risk getting Mobijake (or if you're lucky, a UnicodeDecodeError rather than silently producing garbage).

Performance-wise, let's see:

>>> import timeit
>>> timeit.timeit("""bytes(str(by, encoding="ISO-8859-1"), encoding="ISO-8859-1")""", setup="import random; by = random.randbytes(10_000)")
0.8885893229962676
>>> timeit.timeit("""bytes(str(by, encoding="ascii", errors="surrogateescape"), encoding="ascii", errors="surrogateescape")""", setup="import random; by = random.randbytes(10_000)")
125.00223343299876

That's… a very large difference. ESR's article points out that ISO-8859-1 has some properties that make it some efficient (it maps bytes 0x80—0xff to Unicode code-points of the same numeric value, so there is no translation cost, and the in-memory representation is more efficient). Trying increasing sizes:

>>> for size in 10, 100, 1_000, 2_000, 5_000, 10_000, 20_000:
...     by = random.randbytes(size)
...     duration = timeit.timeit("""bytes(str(by, encoding="ascii", errors="surrogateescape"), encoding="ascii", errors="surrogateescape")""", globals={"by": by}, number=100_000)
...     print(f"{size}\t{duration}")
... 
10	0.0650910490003298
100	0.1047916579991579
1000	0.5472217770002317
2000	1.5103355319952243
5000	5.779067411000142
10000	12.497241530996689
20000	25.78209423399676

That seems to grow faster than O(n); the duration seems to be ×3 when the size goes ×2, is it growing like O(n^1.5)?

In contrast, using the ISO-8859-1 method seems to having a complexity O(n):

>>> for size in 10, 100, 1_000, 2_000, 5_000, 10_000, 20_000, 50_000, 100_000:
...     by = random.randbytes(size)
...     duration = timeit.timeit("""bytes(str(by, encoding="iso-8859-1"), encoding="iso-8859-1")""", globals={"by": by}, number=100_000)
...     print(f"{size}\t{duration}")
... 
10	0.05453772499458864
100	0.037617702000716235
1000	0.05454556500626495
2000	0.05654650100041181
5000	0.06352802200126462
10000	0.0898260960020707
20000	0.23981017799815163
50000	0.4997737009980483
100000	0.9646763860000647

By design of Unicode, the first 256 codepoints are the same as those of ISO 8859-1, and the conversion between Unicode and ISO 8859-1 is implemented in C in CPython. That's why the technique describes above works losslessly, and is so much faster than any mapping to surrogate escapes (even if I still don't understand how that latter approach is so slow).

I recently learned (between $WORK making me investigate character encoding issues, and making a character table explorer as a JavaScript learning exercise) about the C1 controls characters. Wikipedia has a write-up, in short in 1973, during their work on 8-bit encodings for languages beyond English, ECMA and ISO came up with an extra series of control characters in the 0x80--0x9F range, which would allow things like terminal control (the 0x9B Control Sequence Introducer for ANSI escape sequences), switching to a different encoding (eg. SS2/SS2, used by EUC-JP), etc. However, it would still be necessary to represent those in a 7-bit enviroment; so each of those could also be written as 0x1B ␛ followed by an ASCII char (eg. Control Sequence Introducer could be spelled 0x1B [). Both of the following print HI in reverse video with xfce4-terminal and libvte-2.91 v0.70.6, but the second example doesn't work in XTerm version 379. I guess XTerm doesn't (or dropped) support for the C1 controls spelling of the ANSI escape sequences.

$ printf "\\u001b[7m HI \\u001b[m Normal\n"
HI  Normal
$ printf "\\u009B7m HI \\u009Bm Normal\n"
HI  Normal

Anyway, those C1 controls were almost never used, but are reserved in the ISO 8859-X encodings. Windows-1252, presumably noticing that lack of use, assigned printable glyphs at those positions. HTML5 aliases ISO 8859-1 to Windows-1252, I guess because it wouldn't make sense to use control charactes in an HTML document, so it those appear, that must be because the author actually meant Windows-1252 (and if they don't, then ISO 8859-1 and Windows-1252 are identical outside the range of C1 controls.

Écrit le lundi 1 avril 2024.

Trying a new Today I learned section… Today, I learned about set -u, which will fail a bash script if we try to use a variable that's not defined. No more deleting someone's files because a variable was not defined. Demo:

#!/bin/bash
set -u

GREETING=Hello

#oups I misspelt that
echo ${GREETINGS} world

If you run this:

$ ./script
./script: line 7: GREETINGS: unbound variable

That makes testing whether a variable is set slightly more complicated:

set -u
if [[ -z ${GREETINGS} ]]; then
    # oups we never get here, the previous line fails with "unbound variable"
    GREETINGS=$( ....compute greetings ...)
fi

Instead:

set -u
if [[ ! -v GREETINGS ]]; then
    GREETINGS=$( ... compute greetings ... )
fi

Where -v is a Bash Conditional Expression described thusly:

-v varname

True if the shell variable varname is set (has been assigned a value).

I suppose the actually equivalent test would be: \[\[ ! -v GREETINGS ]] || \[\[ -z ${GREETINGS} ]], to check whether the variable is not only set but also non-null.

I've always used set -e and set -o pipefail in shell scripts. The first will exit the entire script if any command (more specifically: any pipeline fails). However by default, quoting the manual: the "exit status of a pipeline is the exit status of the last command in the pipeline", meaning the following succeeds:

$ false | true
$ echo $?
0

But, if "pipefail is enabled, the pipeline's return status is the value of the last (rightmost) command to exit with a non-zero status, or zero if all commands exit successfully", so:

$ set -o pipefail
$ false | true
$ echo $?
1

TL;DR: Use set -euo pipefail at the top of your bash scripts. Julia Evans agrees.

Écrit le samedi 30 mars 2024.

RPi.GPIO is a Python module for controlling GPIO pins on Raspberry Pi. It is old and little maintained (they still use sourceforge.net!), but still useful today. It doesn't have type hints, so I added stubs for RPi.GPIO, now available on PyPI.

Incidentally I learned of distribution name normalisation. Package names can use ASCII letters, numbers, and ., - and _; but the normalised name is lower-cased with any run of punctuation characters replaced with a single dash, so rpi-gpio is the normal form for RPi.GPIO. pip the CLI tool and the PyPI web interface accept any form: pip install RPi.GPIO, or pip install rpi-gpio, or pip install rPi__GpIo will install the same package. pip-compile with use the normalized name when writing a constraints file.

Écrit le samedi 15 décembre 2012.

J'ai voulu avoir de l'IPv6 à la maison. Mon opérateur Internet ne fournit que de l'IPv4, il faut donc tunneller d'une façon ou d'une autre. J'ai une Kimsufi à disposition, avec un accès IPv6 natif, j'ai donc voulu l'utiliser comme point de sortie.

Bien sûr, une solution est d'utiliser un fournisseur de tunnel comme HE ou SixXS, mais où est l'intérêt de ne pas faire les choses soi-même ?

Écrit le samedi 2 juin 2012, modifié le lundi 11 mars 2024.

Avec FreeBSD 9.0 qui est sorti il y a presque six mois, il est grand temps de se mettre à jour. On ne peut pas à la fois prétendre au status de cyborg, et utiliser autre chose que la dernière version de son OS préféré !

Dans un épisode précédent on avait installé son serveur avec ZFS, justement pour pouvoir faire des mises à jour avec une solution de repli si les choses tournent mal. On a deux solutions : faire un snapshot puis mettre à jour, avec la possibilité de faire un rollback plus tard ; ou alors installer dans un nouveau volume et pointer le bootloader sur le nouvel environnement. J'ai exploré la deuxième option. Cette deuxième option permet de ne pas tout perdre s'il faut revenir en arrière. On verra dans la suite si cela devait se révéler utile…