[arch-projects] [netcfg] [PATCH 0/4] Connecting to wireless speedups
Hi, A couple of patches that speeds up connecting to wireless networks. Connecting to my home WPA DHCP, I went from ~12s to ~8s, the meat of which in patch 0004. Henrik Hallberg (4): Include timestamp in DEBUG message Remove extra start_wpa/stop_wpa when not scanning Lower latency in timeout_wait Wait actively in {start,stop}_wpa src/8021x | 17 +++++++++++------ src/connections/wireless | 17 +++++++++-------- src/globals | 5 +++-- 3 files changed, 23 insertions(+), 16 deletions(-) -- 1.7.11.1
--- src/globals | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/globals b/src/globals index af411fe..436334a 100644 --- a/src/globals +++ b/src/globals @@ -28,7 +28,7 @@ function report_notice { } function report_debug { - checkyesno "$NETCFG_DEBUG" && echo "DEBUG: $*" >&2 + checkyesno "$NETCFG_DEBUG" && echo "DEBUG $(date -Ins): $*" >&2 } function report_try { -- 1.7.11.1
On Mon, Jul 2, 2012 at 11:07 PM, Henrik Hallberg <henrik@k2h.se> wrote:
--- src/globals | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/globals b/src/globals index af411fe..436334a 100644 --- a/src/globals +++ b/src/globals @@ -28,7 +28,7 @@ function report_notice { }
function report_debug { - checkyesno "$NETCFG_DEBUG" && echo "DEBUG: $*" >&2 + checkyesno "$NETCFG_DEBUG" && echo "DEBUG $(date -Ins): $*" >&2
Isn't that a bit too verbose? What about: echo "DEBUG ${SECONDS}s: $*"
}
function report_try { -- 1.7.11.1
On Tue, Jul 03, 2012 at 12:58:57AM +0200, Jouke Witteveen wrote:
On Mon, Jul 2, 2012 at 11:07 PM, Henrik Hallberg <henrik@k2h.se> wrote:
function report_debug { - checkyesno "$NETCFG_DEBUG" && echo "DEBUG: $*" >&2 + checkyesno "$NETCFG_DEBUG" && echo "DEBUG $(date -Ins): $*" >&2
Isn't that a bit too verbose? What about: echo "DEBUG ${SECONDS}s: $*"
Absolute time is useful when cross-checking with e.g. logs. When looking for time thiefes among a dozen lines of output over as many seconds, like I did for these patches, fractions of a second were useful. By all means, if I ever were to do this again, it'd be easy to add the above then and there. / H
On Tue, Jul 3, 2012 at 2:35 AM, Henrik Hallberg <henrik@k2h.se> wrote:
On Tue, Jul 03, 2012 at 12:58:57AM +0200, Jouke Witteveen wrote:
On Mon, Jul 2, 2012 at 11:07 PM, Henrik Hallberg <henrik@k2h.se> wrote:
function report_debug { - checkyesno "$NETCFG_DEBUG" && echo "DEBUG: $*" >&2 + checkyesno "$NETCFG_DEBUG" && echo "DEBUG $(date -Ins): $*" >&2
Isn't that a bit too verbose? What about: echo "DEBUG ${SECONDS}s: $*"
Absolute time is useful when cross-checking with e.g. logs. When looking for time thiefes among a dozen lines of output over as many seconds, like I did for these patches, fractions of a second were useful.
By all means, if I ever were to do this again, it'd be easy to add the above then and there.
/ H
I see. In that case I won't include this one. It is just too verbose and as you say, the bare seconds are of too little use. Have you tried systemd? It is able to bundle all related logging. Regards, - Jouke
Unless $SCAN is enabled, an unnecessary start_wpa/stop_wpa pair was run. --- src/connections/wireless | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/src/connections/wireless b/src/connections/wireless index 44a25b4..4384bd1 100644 --- a/src/connections/wireless +++ b/src/connections/wireless @@ -31,14 +31,15 @@ wireless_up() { else WPA_CONF=$(make_wpa_config_file $INTERFACE) fi - report_debug wireless_up start_wpa "$INTERFACE" "$WPA_CONF" "$WPA_DRIVER" "$WPA_OPTS" - if ! start_wpa "$INTERFACE" "$WPA_CONF" "$WPA_DRIVER" "$WPA_OPTS"; then - report_fail "wpa_supplicant did not start, possible configuration error" - return 1 - fi # Scan for network's existence first if checkyesno "${SCAN:-no}"; then + report_debug wireless_up start_wpa "$INTERFACE" "$WPA_CONF" "$WPA_DRIVER" "$WPA_OPTS" + if ! start_wpa "$INTERFACE" "$WPA_CONF" "$WPA_DRIVER" "$WPA_OPTS"; then + report_fail "wpa_supplicant did not start, possible configuration error" + return 1 + fi + report_debug wireless_up scanning local OLDESSID="$ESSID" if [[ -n "$AP" ]]; then @@ -52,10 +53,10 @@ wireless_up() { stop_wpa "$INTERFACE" return 1 fi - fi - report_debug wireless_up stop_wpa "$INTERFACE" - stop_wpa "$INTERFACE" + report_debug wireless_up stop_wpa "$INTERFACE" + stop_wpa "$INTERFACE" + fi # Build configuration file case "$SECURITY" in -- 1.7.11.1
On Mon, Jul 2, 2012 at 11:07 PM, Henrik Hallberg <henrik@k2h.se> wrote:
Unless $SCAN is enabled, an unnecessary start_wpa/stop_wpa pair was run. --- src/connections/wireless | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-)
diff --git a/src/connections/wireless b/src/connections/wireless index 44a25b4..4384bd1 100644 --- a/src/connections/wireless +++ b/src/connections/wireless @@ -31,14 +31,15 @@ wireless_up() { else WPA_CONF=$(make_wpa_config_file $INTERFACE) fi - report_debug wireless_up start_wpa "$INTERFACE" "$WPA_CONF" "$WPA_DRIVER" "$WPA_OPTS" - if ! start_wpa "$INTERFACE" "$WPA_CONF" "$WPA_DRIVER" "$WPA_OPTS"; then - report_fail "wpa_supplicant did not start, possible configuration error" - return 1 - fi
# Scan for network's existence first if checkyesno "${SCAN:-no}"; then + report_debug wireless_up start_wpa "$INTERFACE" "$WPA_CONF" "$WPA_DRIVER" "$WPA_OPTS" + if ! start_wpa "$INTERFACE" "$WPA_CONF" "$WPA_DRIVER" "$WPA_OPTS"; then + report_fail "wpa_supplicant did not start, possible configuration error" + return 1 + fi + report_debug wireless_up scanning local OLDESSID="$ESSID" if [[ -n "$AP" ]]; then @@ -52,10 +53,10 @@ wireless_up() { stop_wpa "$INTERFACE" return 1 fi - fi
- report_debug wireless_up stop_wpa "$INTERFACE" - stop_wpa "$INTERFACE" + report_debug wireless_up stop_wpa "$INTERFACE" + stop_wpa "$INTERFACE" + fi
# Build configuration file case "$SECURITY" in -- 1.7.11.1
These changes look fine to me, but shitty code remains shitty code. I'm not a big fan of the current organization of the code and the distribution across connections/wireless and 8021x.
On Tue, Jul 03, 2012 at 01:16:07AM +0200, Jouke Witteveen wrote:
These changes look fine to me, but shitty code remains shitty code. I'm not a big fan of the current organization of the code and the distribution across connections/wireless and 8021x.
So far, it has seemed better to submit small patches rather than larger ones. I could very well have spent time doing a larger refactoring. It seems to me, however, that I am not yet enough versed in the code and its' style to make an attempt with reasonable chances to pass review. This solves the problem of the extra calls only. A small, easily reviewable patch. The code behaves better with it, than without it. / H
On Tue, Jul 3, 2012 at 3:00 AM, Henrik Hallberg <henrik@k2h.se> wrote:
On Tue, Jul 03, 2012 at 01:16:07AM +0200, Jouke Witteveen wrote:
These changes look fine to me, but shitty code remains shitty code. I'm not a big fan of the current organization of the code and the distribution across connections/wireless and 8021x.
So far, it has seemed better to submit small patches rather than larger ones. I could very well have spent time doing a larger refactoring. It seems to me, however, that I am not yet enough versed in the code and its' style to make an attempt with reasonable chances to pass review.
This solves the problem of the extra calls only. A small, easily reviewable patch. The code behaves better with it, than without it.
/ H
There's some comments in lines 73-75. They were ignored in commit 0d4c3. Reverting said commit would solve this problem too. Any thoughts what would be better? I like the old code, but the commit message suggests that wpa_supplicant doesn't reassociate properly (at least, that it didn't back then). If wpa_supplicant has improved, I'm in favor of restoring the old code, which looks more reasonable to me. Otherwise, the proposed patch will do. - Jouke
--- src/globals | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/globals b/src/globals index 436334a..62cce3e 100644 --- a/src/globals +++ b/src/globals @@ -100,10 +100,11 @@ function checkyesno() { # $2...: condition command function timeout_wait() { local timeout="$1" + (( timeout *= 10 )) shift while ! eval "$*"; do (( timeout-- > 0 )) || return 1 - sleep 1 + sleep 0.1 done return 0 } -- 1.7.11.1
On Mon, Jul 2, 2012 at 11:07 PM, Henrik Hallberg <henrik@k2h.se> wrote:
--- src/globals | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/src/globals b/src/globals index 436334a..62cce3e 100644 --- a/src/globals +++ b/src/globals @@ -100,10 +100,11 @@ function checkyesno() { # $2...: condition command function timeout_wait() { local timeout="$1" + (( timeout *= 10 )) shift while ! eval "$*"; do (( timeout-- > 0 )) || return 1 - sleep 1 + sleep 0.1 done return 0 } -- 1.7.11.1
I don't think this is such a good idea. This saves less than .45 seconds on average per timeout_wait (old expected waste: .5s, new expected waste: .05s, added polling overhead). If you want to do it this way, you should probably decouple (( timeout-- > 0 )) into (( timeout -= .1 )) and (( timeout > 0 )). The gain is usually a lot less than the wait (saving .45s on something that takes 3s isn't all too exciting) and not seldom, netcfg is an interactive script so that half a second doesn't really matter. Why would we try to make a bash script super fast? Note that even with the 1s sleeps, it can be beneficial to call `timeout_wait 1 'whatever'`, as it might return in 0s. This means the other patches still make some sense without this one. That's just what I think. I appreciate your contributions! - Jouke
On Tue, Jul 03, 2012 at 01:08:34AM +0200, Jouke Witteveen wrote:
On Mon, Jul 2, 2012 at 11:07 PM, Henrik Hallberg <henrik@k2h.se> wrote:
--- src/globals | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/src/globals b/src/globals index 436334a..62cce3e 100644 --- a/src/globals +++ b/src/globals @@ -100,10 +100,11 @@ function checkyesno() { # $2...: condition command function timeout_wait() { local timeout="$1" + (( timeout *= 10 )) shift while ! eval "$*"; do (( timeout-- > 0 )) || return 1 - sleep 1 + sleep 0.1 done return 0 } -- 1.7.11.1
I don't think this is such a good idea. This saves less than .45 seconds on average per timeout_wait (old expected waste: .5s, new expected waste: .05s, added polling overhead).
If you assume uniform distribution, yes. However, this is not the case. For the cases which I fixed in patch 4, the first eval (no sleep at all) was sufficient. (Then, of course, the sleep time does not matter). If not, the second always was.
If you want to do it > this way, you should probably decouple (( timeout-- > 0 )) into (( timeout -= .1 )) and (( timeout > 0 )).
I could have sworn that I tested floating point calculations in bash, failed, and found texts claiming that it was not possible without e.g. `bc`. TIL.
The gain is usually a lot less than the wait (saving .45s on something that takes 3s isn't all too exciting) and not seldom, netcfg is an interactive script so that half a second doesn't really matter.
Before patch 2, there were two calls each to {start,stop}_wpa. With this change alone, and the uneven distribution of required time, I typically saved four seconds -- i.e. the lions share of `netcfg -u` wall time.
Why would we try to make a bash script super fast?
Why keep unnescessary waiting times in? I'm happier to boot and have my net connected by the time my shell is. than waiting a little longer. With this and DHCP_OPTIONS='--noarp', it takes four seconds instead of twelve.
Note that even with the 1s sleeps, it can be beneficial to call `timeout_wait 1 'whatever'`, as it might return in 0s. This means the other patches still make some sense without this one.
They do. I divided this set of patches like I did so you could pick and choose. / H
On Tue, Jul 3, 2012 at 2:49 AM, Henrik Hallberg <henrik@k2h.se> wrote:
On Tue, Jul 03, 2012 at 01:08:34AM +0200, Jouke Witteveen wrote:
On Mon, Jul 2, 2012 at 11:07 PM, Henrik Hallberg <henrik@k2h.se> wrote:
--- src/globals | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/src/globals b/src/globals index 436334a..62cce3e 100644 --- a/src/globals +++ b/src/globals @@ -100,10 +100,11 @@ function checkyesno() { # $2...: condition command function timeout_wait() { local timeout="$1" + (( timeout *= 10 )) shift while ! eval "$*"; do (( timeout-- > 0 )) || return 1 - sleep 1 + sleep 0.1 done return 0 } -- 1.7.11.1
I don't think this is such a good idea. This saves less than .45 seconds on average per timeout_wait (old expected waste: .5s, new expected waste: .05s, added polling overhead).
If you assume uniform distribution, yes. However, this is not the case. For the cases which I fixed in patch 4, the first eval (no sleep at all) was sufficient. (Then, of course, the sleep time does not matter). If not, the second always was.
If you want to do it > this way, you should probably decouple (( timeout-- > 0 )) into (( timeout -= .1 )) and (( timeout > 0 )).
I could have sworn that I tested floating point calculations in bash, failed, and found texts claiming that it was not possible without e.g. `bc`. TIL.
Shame on me, you're right of course :-P.
Why would we try to make a bash script super fast?
Why keep unnescessary waiting times in? I'm happier to boot and have my net connected by the time my shell is. than waiting a little longer. With this and DHCP_OPTIONS='--noarp', it takes four seconds instead of twelve.
Well, they're not entirely unnecessary. Because it's a rather harmless, small patch I threw it in, but I hereby declare that I oppose the next suggestion to upgrade to 1/100th second polling ;-).
Check .pid file regularly instead of waiting a second blindly. Saves up to a second of wall time per call. --- src/8021x | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/src/8021x b/src/8021x index d143b30..b78dcf9 100644 --- a/src/8021x +++ b/src/8021x @@ -38,18 +38,23 @@ start_wpa() fi wpa_supplicant -B -P "/run/wpa_supplicant_${INTERFACE}.pid" -i "$INTERFACE" -D "$WPA_DRIVER" "$WPA_CONF" $WPA_OPTS - sleep 1 - if [[ ! -f "/run/wpa_supplicant_${INTERFACE}.pid" ]]; then - return 1 - fi + # wait up to one second until pid file appears + timeout_wait 1 "[[ -f \"/run/wpa_supplicant_${INTERFACE}.pid\" ]]"; + + # Return success if pid file exists, else failure + return $? } stop_wpa() { wpa_cli -p "$WPA_CTRL_PATH" -i "$1" terminate &> /dev/null - sleep 1 # JP: need this else the file tends to disappear after [[ -f ... ]] but before cat... - # see <http://bbs.archlinux.org/viewtopic.php?pid=515667#p515667> + + # sleep up to one second, waiting for pid file to be removed + # JP: need to sleep else the file tends to disappear after [[ -f ... ]] but before cat... + # see <http://bbs.archlinux.org/viewtopic.php?pid=515667#p515667> + timeout_wait 1 "[[ ! -f \"/run/wpa_supplicant_$1.pid\" ]]" + if [[ -f "/run/wpa_supplicant_$1.pid" ]]; then kill "$(< "/run/wpa_supplicant_$1.pid")" &>/dev/null & fi -- 1.7.11.1
On Mon, Jul 2, 2012 at 11:07 PM, Henrik Hallberg <henrik@k2h.se> wrote:
Check .pid file regularly instead of waiting a second blindly. Saves up to a second of wall time per call. --- src/8021x | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-)
diff --git a/src/8021x b/src/8021x index d143b30..b78dcf9 100644 --- a/src/8021x +++ b/src/8021x @@ -38,18 +38,23 @@ start_wpa() fi
wpa_supplicant -B -P "/run/wpa_supplicant_${INTERFACE}.pid" -i "$INTERFACE" -D "$WPA_DRIVER" "$WPA_CONF" $WPA_OPTS - sleep 1
- if [[ ! -f "/run/wpa_supplicant_${INTERFACE}.pid" ]]; then - return 1 - fi + # wait up to one second until pid file appears + timeout_wait 1 "[[ -f \"/run/wpa_supplicant_${INTERFACE}.pid\" ]]"; + + # Return success if pid file exists, else failure
I would ditch the above two lines (a bit superfluous).
+ return $? }
stop_wpa() { wpa_cli -p "$WPA_CTRL_PATH" -i "$1" terminate &> /dev/null - sleep 1 # JP: need this else the file tends to disappear after [[ -f ... ]] but before cat... - # see <http://bbs.archlinux.org/viewtopic.php?pid=515667#p515667> + + # sleep up to one second, waiting for pid file to be removed + # JP: need to sleep else the file tends to disappear after [[ -f ... ]] but before cat... + # see <http://bbs.archlinux.org/viewtopic.php?pid=515667#p515667> + timeout_wait 1 "[[ ! -f \"/run/wpa_supplicant_$1.pid\" ]]" + if [[ -f "/run/wpa_supplicant_$1.pid" ]]; then kill "$(< "/run/wpa_supplicant_$1.pid")" &>/dev/null & fi
This can be merged like: timeout_wait 1 '[[ ! -f "/run/wpa_supplicant_$1.pid" ]]' || kill "$(< "/run/wpa_supplicant_$1.pid")" &>/dev/null &
-- 1.7.11.1
If these changes save 4 seconds, I guess the supplicant is stopped more than necessary. I'd say: if you're gonna need it, start it once (and once only)! I haven't investigated this. Regards, - Jouke
Check .pid file regularly instead of waiting a second blindly. Saves up to a second of wall time per call. --- src/8021x | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/src/8021x b/src/8021x index d143b30..7af6950 100644 --- a/src/8021x +++ b/src/8021x @@ -38,21 +38,21 @@ start_wpa() fi wpa_supplicant -B -P "/run/wpa_supplicant_${INTERFACE}.pid" -i "$INTERFACE" -D "$WPA_DRIVER" "$WPA_CONF" $WPA_OPTS - sleep 1 - if [[ ! -f "/run/wpa_supplicant_${INTERFACE}.pid" ]]; then - return 1 - fi + # wait up to one second until pid file appears + timeout_wait 1 "[[ -f \"/run/wpa_supplicant_${INTERFACE}.pid\" ]]"; + return $? } stop_wpa() { wpa_cli -p "$WPA_CTRL_PATH" -i "$1" terminate &> /dev/null - sleep 1 # JP: need this else the file tends to disappear after [[ -f ... ]] but before cat... - # see <http://bbs.archlinux.org/viewtopic.php?pid=515667#p515667> - if [[ -f "/run/wpa_supplicant_$1.pid" ]]; then + + # sleep up to one second, waiting for pid file to be removed + # JP: need to sleep else the file tends to disappear after [[ -f ... ]] but before cat... + # see <http://bbs.archlinux.org/viewtopic.php?pid=515667#p515667> + timeout_wait 1 "[[ ! -f \"/run/wpa_supplicant_$1.pid\" ]]" || \ kill "$(< "/run/wpa_supplicant_$1.pid")" &>/dev/null & - fi } wpa_reconfigure() { -- 1.7.11.1
Am 02.07.2012 23:07, schrieb Henrik Hallberg:
Hi,
A couple of patches that speeds up connecting to wireless networks. Connecting to my home WPA DHCP, I went from ~12s to ~8s, the meat of which in patch 0004.
Henrik Hallberg (4): Include timestamp in DEBUG message Remove extra start_wpa/stop_wpa when not scanning Lower latency in timeout_wait Wait actively in {start,stop}_wpa
src/8021x | 17 +++++++++++------ src/connections/wireless | 17 +++++++++-------- src/globals | 5 +++-- 3 files changed, 23 insertions(+), 16 deletions(-)
I found that whole method confusing all the time. You first scan for the network's existence, then connect to it or fail. Instead, why not remodel this to use the same logic as net-auto-wireless? Create the right wpa_supplicant configuration, start wpa_s and let wpa_actiond bring up the interface? This seems much better than hardcoded timeouts and wait loops inside shell scripts. And you do not have to write much new code, everything's there.
On Tue, Jul 3, 2012 at 11:04 AM, Thomas Bächler <thomas@archlinux.org> wrote:
Am 02.07.2012 23:07, schrieb Henrik Hallberg:
Hi,
A couple of patches that speeds up connecting to wireless networks. Connecting to my home WPA DHCP, I went from ~12s to ~8s, the meat of which in patch 0004.
Henrik Hallberg (4): Include timestamp in DEBUG message Remove extra start_wpa/stop_wpa when not scanning Lower latency in timeout_wait Wait actively in {start,stop}_wpa
src/8021x | 17 +++++++++++------ src/connections/wireless | 17 +++++++++-------- src/globals | 5 +++-- 3 files changed, 23 insertions(+), 16 deletions(-)
I found that whole method confusing all the time. You first scan for the network's existence, then connect to it or fail.
Instead, why not remodel this to use the same logic as net-auto-wireless? Create the right wpa_supplicant configuration, start wpa_s and let wpa_actiond bring up the interface? This seems much better than hardcoded timeouts and wait loops inside shell scripts. And you do not have to write much new code, everything's there.
In that case, we should depend on and maintain wpa_actiond, which is not a bash script. I think it is possible to rewrite connections/wireless in a more netcfg way, that is: using plain scripting. Ideally, I see netcfg as just that: a collection of scripts to automate networking routines based on simple profile definitions. If you need speed, power, interfaces or what not, you should consider one of the many alternatives (ConnMan, Wicd, NetworkManager, ...). If you need simplicity in an "I know what I'm doing" fashion ("The Arch Way"?), go with netcfg. The current code might not be all that bad, just cluttered and with the wrong decisions or logic here and there. - Jouke
Am 03.07.2012 13:44, schrieb Jouke Witteveen:
Instead, why not remodel this to use the same logic as net-auto-wireless? Create the right wpa_supplicant configuration, start wpa_s and let wpa_actiond bring up the interface? This seems much better than hardcoded timeouts and wait loops inside shell scripts. And you do not have to write much new code, everything's there.
In that case, we should depend on and maintain wpa_actiond, which is not a bash script. I think it is possible to rewrite connections/wireless in a more netcfg way, that is: using plain scripting. Ideally, I see netcfg as just that: a collection of scripts to automate networking routines based on simple profile definitions.
The problem I saw here was this: Wireless connections can go up and down all the time, and you have to act accordingly. What netcfg currently does is this: - Is the network there? No -> Fail - Start the supplicant. - wpa_cli status -> Are we connected? No -> Sleep and Repeat - After a timeout: Fail. It does not handle the case where connecting takes longer than the timeout (for example, you are out of range and then move closer). It does not handle the case where you lose the connection (although dhcpcd re-acquires the lease nowadays, this was not always the case). In short, netcfg may fail in cases where it shouldn't. When writing net-auto-wireless (or its predecessor autowifi), I considered the following: wpa_supplicant knows when the connection state changes and has mechanisms to notify us. Let's use those mechanisms. Sadly, 'wpa_cli -a' was not suited for this, as disconnecting and reconnecting immediately lead to annoying race conditions (which happened with my wireless constantly at the time). wpa_actiond is simply a more clever version of 'wpa_cli -a'. I don't really understand why anyone would prefer manually selecting the wireless profile instead of running net-auto-wireless to automatically choose. But even if you want to do that, you should not have all those race conditions that result from the half-baked connection logic I mentioned in the beginning.
participants (3)
-
Henrik Hallberg
-
Jouke Witteveen
-
Thomas Bächler