[arch-projects] [devtools] [PATCH 1/2] makechrootpkg: Fix unconditionally running namcap
Fixes regression in 49088b0860276c664933c2b3e36a2fef714b7a07 $run_namcap will always be set to "" `if $not_a_var; then ...; fi` is always truthful when $not_a_var is unset or equal to "" and the `then` clause will always be run. I'm not entirely sure why everything had to be moved into functions purely for the sake of being moved into functions, as it goes against the general theme of devtools/makepkg/etc. especially since it doesn't seem to have been thoroughly tested. I'm also not sure why global state variables need to be cloned locally for their sole explicit purpose. But for now this patch implements the minimum necessary work to properly pass the "do I want namcap" variable into prepare_chroot() according to the current logic flow. Note that I have still not thorougly tested makechrootpkg. Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> --- makechrootpkg.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/makechrootpkg.in b/makechrootpkg.in index 664f9f6..b1c9545 100644 --- a/makechrootpkg.in +++ b/makechrootpkg.in @@ -403,7 +403,7 @@ main() { download_sources "$copydir" "$makepkg_user" - prepare_chroot "$copydir" "$USER_HOME" "$keepbuilddir" + prepare_chroot "$copydir" "$USER_HOME" "$keepbuilddir" "$run_namcap" if arch-nspawn "$copydir" \ --bind="$PWD:/startdir" \ -- 2.14.1
Don't use error-prone logic e.g. foo=true; if $foo ... This invokes an extra process as opposed to a simple value comparison, in the process of completely failing to act as expected when the variable is unset because of unrelated bugs. Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> --- makechrootpkg.in | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/makechrootpkg.in b/makechrootpkg.in index b1c9545..90bd9cf 100644 --- a/makechrootpkg.in +++ b/makechrootpkg.in @@ -188,7 +188,7 @@ prepare_chroot() { local keepbuilddir=$3 local run_namcap=$4 - $keepbuilddir || rm -rf "$copydir/build" + [[ $keepbuilddir = true ]] || rm -rf "$copydir/build" local builduser_uid builduser_gid builduser_uid="${SUDO_UID:-$UID}" @@ -225,7 +225,7 @@ EOF declare -f _chrootbuild printf '_chrootbuild "$@" || exit\n' - if $run_namcap; then + if [[ $run_namcap = true ]]; then declare -f _chrootnamcap printf '_chrootnamcap || exit\n' fi -- 2.14.1
On Fri, 01 Sep 2017 18:53:13 -0400, Eli Schwartz wrote:
Don't use error-prone logic e.g. foo=true; if $foo ...
Hey, I'm not to blame for this one :-)
This invokes an extra process as opposed to a simple value comparison,
That's not quite true: $ type true true is a shell builtin $ type false false is a shell builtin Bash won't actually make any syscalls when executing these commands in a conditional. Apparently the code paths within Bash are slightly faster for `[[ $v = true ]]` than for `$v`, but that's negligible compared to if it actually had to invoke an extra process as you said. Time for 1 million iterations: "v=true; if $v; then ..." | 7.1 sec "v=true; if [[ $v = true ]]; then ..." | 6.5 sec "v=/bin/true; if $v; then ..." | > 5 minutes (I got bored and killed it)
in the process of completely failing to act as expected when the variable is unset because of unrelated bugs.
And that's the real reason that this change should be applied. :-) -- Happy hacking, ~ Luke Shumaker
On 09/01/2017 11:56 PM, Luke Shumaker wrote:
This invokes an extra process as opposed to a simple value comparison,
That's not quite true:
$ type true true is a shell builtin $ type false false is a shell builtin
Ah, true enough. :o I maybe should get out of the habit of submitting brand-new patches on the spur of the moment right before going offline for the weekend. Still, implementation-defined behavior... bash includes several things that shouldn't be builtins, though this isn't one of them. e.g. the horrible `kill` implementation which should always be disabled. coreutils/builtins clashes are so annoying. :( -- Eli Schwartz
On Fri, 01 Sep 2017 18:53:12 -0400, Eli Schwartz wrote:
Fixes regression in 49088b0860276c664933c2b3e36a2fef714b7a07
Are you sure it was there? Looking at the code (but without actually testing it) (and being the one responsible for the bug), I think it was actually in 2fd5931a8c67289a8a4acd327b3ce99a5d64c8c7. Maybe `git blame` told you the erroneous line was introduced in 49088b0--it was actually just re-indented there; it originally appeared (un-indented) in 2fd5931.
$run_namcap will always be set to "" `if $not_a_var; then ...; fi` is always truthful when $not_a_var is unset or equal to "" and the `then` clause will always be run.
I'm not entirely sure why everything had to be moved into functions purely for the sake of being moved into functions, as it goes against the general theme of devtools/makepkg/etc. especially since it doesn't seem to have been thoroughly tested.
I'm the one to blame for that. In Parabola (a derivative of Arch), we have a couple of other programs that want to reuse code from makechrootpkg; putting everything into functions makes that possible; we just delete the call to `main "$@"` at the end and name it makechrootpkg.sh somewhere in /lib. However, we don't actually use the makechrootpkg program itself, we only use functions from it. Because of this, most of the functions except for `main` are tested by Parabola's fairly intense test suite for our dev tools, but main(), init_variables(), usage(), and load_vars() aren't. So I admit, Arch doesn't really benefit from 49088b0 (except for whatever arguments you want to make about code clarity); and that it's a change made for the benefit of Parabola. But hey, Jan merged it :-) But also, prepare_chroot() was a function before any code from me/Parabola was merged. The relevant change made by me was in 2fd5931 when I made it take run_namcap as an argument, instead of as a global variable. Which leads us in to...
I'm also not sure why global state variables need to be cloned locally for their sole explicit purpose.
Well, as I explained above, Parabola has another program that calls prepare_chroot(), and having to set some global variable (especially a lower_case one) is a weird calling convention. But also, I don't think that it's a bad idea to slowly get rid of global variables in programs (of course in Bash it's not quite a good distinction because it's dynamically scoped instead of lexically scoped, and even if it's a local variable in a function, in anything that you call it might as well be global). And this is a step toward that. Think of all the globals as just local variables to main()--other functions (except for init_variables(), which is really just "main() part 1") shouldn't be touching them.
But for now this patch implements the minimum necessary work to properly pass the "do I want namcap" variable into prepare_chroot() according to the current logic flow. Note that I have still not thorougly tested makechrootpkg.
I actually think that this mistake originated in Parabola's libretools commit 6eddc77 way back on 2013-09-11, when I made a mistake when resolving merge conflicts when merging devtools commits 7ca4eb82 (2013-05-02) and 4937422 (2013-05-03) into libretools. Jan and I had made many of the same changes, but in subtly different ways.
Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> --- makechrootpkg.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/makechrootpkg.in b/makechrootpkg.in index 664f9f6..b1c9545 100644 --- a/makechrootpkg.in +++ b/makechrootpkg.in @@ -403,7 +403,7 @@ main() {
download_sources "$copydir" "$makepkg_user"
- prepare_chroot "$copydir" "$USER_HOME" "$keepbuilddir" + prepare_chroot "$copydir" "$USER_HOME" "$keepbuilddir" "$run_namcap"
if arch-nspawn "$copydir" \ --bind="$PWD:/startdir" \ -- 2.14.1
Not that I'm in a position within Arch for it to carry too much weight, but: LGTM. -- Happy hacking, ~ Luke Shumaker
On 09/01/2017 11:34 PM, Luke Shumaker wrote:
Are you sure it was there? Looking at the code (but without actually testing it) (and being the one responsible for the bug), I think it was actually in 2fd5931a8c67289a8a4acd327b3ce99a5d64c8c7.
Maybe `git blame` told you the erroneous line was introduced in 49088b0--it was actually just re-indented there; it originally appeared (un-indented) in 2fd5931.
I think you're right, there. It's all the same basic cause though, turning everything into multiply localized state variables passed around functions that are only ever useful if trying to turn makechrootpkg into a lib.
So I admit, Arch doesn't really benefit from 49088b0 (except for whatever arguments you want to make about code clarity); and that it's a change made for the benefit of Parabola. But hey, Jan merged it :-)
FWIW I happen to feel that the code clarity dropped after your patchset. :p
Well, as I explained above, Parabola has another program that calls prepare_chroot(), and having to set some global variable (especially a lower_case one) is a weird calling convention.
But also, I don't think that it's a bad idea to slowly get rid of global variables in programs (of course in Bash it's not quite a good distinction because it's dynamically scoped instead of lexically scoped, and even if it's a local variable in a function, in anything that you call it might as well be global). And this is a step toward that. Think of all the globals as just local variables to main()--other functions (except for init_variables(), which is really just "main() part 1") shouldn't be touching them.
"it's not quite a good distinction" indeed. And init_variables() is most emphatically *not* main() part 1, whether it is meant to be or not in practice it doesn't even attempt to be. Global variables aren't automatically a sin. Imperative programming isn't automatically a sin either. When used properly it can be both very nice-looking and descriptive of what is going on. Figuring out where the state is at any given time can only ever be harder when scoping it (complexity is hard...), but usually that is a negligible cost offset by the reusability of functions that are going to be reused a lot. Which brings us back to "this is only useful as a lib". Also the existence of this bug testifies that you weren't able to follow what it was doing. :p And if it were so advantageous to stick everything inside a main() function in bash scripts, maybe the rest of devtools should be converted as well. Thing is, that doesn't make a whole lot of sense in either makechrootpkg or any other tool if you aren't actually doing much other than sticking the entire contents of a script into a single function and then calling the function. I kind of feel that makechrootpkg had struck a pretty good balance already. I mean, I guess an argument could be made that on a strictly "make this more reusable for parabola" level prepare_chroot() should be able to scope run_namcap, which I guess was neatly accomplished (and broken) in the commit you mentioned... but tacking on a bunch of questionable changes and claiming that it was done for code readability purposes is IMHO fallacious. Also main() functions in bash scripts are stupid. As are functions whose only purpose is to set a bunch of global (!!!) variables immediately after the function definition. -- Eli Schwartz
On Sun, 03 Sep 2017 03:12:19 -0400, Eli Schwartz wrote:
But also, I don't think that it's a bad idea to slowly get rid of global variables in programs (of course in Bash it's not quite a good distinction because it's dynamically scoped instead of lexically scoped, and even if it's a local variable in a function, in anything that you call it might as well be global). And this is a step toward that. Think of all the globals as just local variables to main()--other functions (except for init_variables(), which is really just "main() part 1") shouldn't be touching them.
"it's not quite a good distinction" indeed.
And init_variables() is most emphatically *not* main() part 1, whether it is meant to be or not in practice it doesn't even attempt to be.
From the commit that put the code in init_variables() and main() into functions instead of floating around in the top-level: It [may] make sense to move init_variables() down into the main() function, instead of having it as a separate function up top (if this done, then the `-g` flag passed to `declare` in init_variables() can be dropped). However, in interest of keeping the `diff -w` small, and merges/rebases simpler, this isn't done here. If you cut/paste the code from init_variables() into the top of main(), it works fine. The sole reason that it isn't part of main() is that if Jan didn't apply that patch, then it meant that all of my work on commits after that would be harder to merge/rebase to new versions of devtools.
Global variables aren't automatically a sin. Imperative programming isn't automatically a sin either. When used properly it can be both very nice-looking and descriptive of what is going on.
Oh, definitely. But many of the functions that were already there before I made any changes weren't really functions, they were just labels serving as comments. That is understandable when you look at the history and see them as incrementally improving the legibility of the codebase from relative mess that it was circa early 2013. I was just taking it a step farther; I made it so that if a function was touching global state, then it was explicit.
Figuring out where the state is at any given time can only ever be harder when scoping it (complexity is hard...), but usually that is a negligible cost offset by the reusability of functions that are going to be reused a lot. Which brings us back to "this is only useful as a lib".
Splitting things in to functions (real functions, that stand alone from the calling code) increases readability because when reading that function, you see the signature, and know the inputs, the outputs, and you don't have to know everything that's going on in the entire file, you can read that one function and understand the function. When it starts dealing with global state, well then you start to lose that, as you have to start worrying about the entire codebase again.
Also the existence of this bug testifies that you weren't able to follow what it was doing. :p
Oh, I followed what it was doing. The code in libremakepkg calls prepare_chroot() correctly :P It was just a simple mistake of missing an argument when I called it. As I said, I hadn't generally been looking at main() as closely as I do the other functions (as on Parabola it never gets called), so it slipped through the cracks.
And if it were so advantageous to stick everything inside a main() function in bash scripts, maybe the rest of devtools should be converted as well. Thing is, that doesn't make a whole lot of sense in either makechrootpkg or any other tool if you aren't actually doing much other than sticking the entire contents of a script into a single function and then calling the function.
I kind of feel that makechrootpkg had struck a pretty good balance already.
I mean, I guess an argument could be made that on a strictly "make this more reusable for parabola" level prepare_chroot() should be able to scope run_namcap, which I guess was neatly accomplished (and broken) in the commit you mentioned... but tacking on a bunch of questionable changes and claiming that it was done for code readability purposes is IMHO fallacious.
Who claimed it was done for readability? The commit message said makechrootpkg: Have functions be more function-y. Rather than them simply being named blocks of code with braces around them. That is: have them take things via arguments rather than global variables. I made it known that I was backporting changes made in Parabola, but I never gave any justification for this change more than what is said in the commit message. There's an argument to be made that making them more function-y improves readability (it's explicit what affects that bit of code), though I didn't make that argument. Of course, there was the original motivating argument that them being more function-y makes them easier to call from places other than (what I indented to become) main(). Though I did not discuss that motivation when contributing it upstream.
Also main() functions in bash scripts are stupid.
Eh, it's similar to the idiom of wrapping everything in { ... }, except that it isn't confusing to people who haven't seen that idiom before. For reference, this serves the purpose of making it so that Bash doesn't begin executing anything until the entire script has been parsed. This is important for scripts that aren't real files (maybe piped from curl, a terrible idea, but a thing people do), but regular programs benefit because if the file is edited (via an upgrade) while the program is running, then it probably hasn't had the entire thing read yet, and that can lead to real weirdness.
As are functions whose only purpose is to set a bunch of global (!!!) variables immediately after the function definition.
Eh, I'd have moved it to be at the top of main(), but I was trying to avoid moving code around in the file. -- Happy hacking, ~ Luke Shumaker
Eli seems miffed that init_variables() is a separate function from main(), so fix that. Luke Shumaker (1): makechrootpkg: move init_variables() to be part of main() makechrootpkg.in | 44 ++++++++++++++++++++------------------------ 1 file changed, 20 insertions(+), 24 deletions(-) -- 2.14.1
The reason it wasn't moved before was just to keep the diffs (with --ignore-all-space) smaller, to make merging and rebasing work easier. Moving code around in a file tends to make that difficult. But, readability wise, it belongs in main(). --- makechrootpkg.in | 44 ++++++++++++++++++++------------------------ 1 file changed, 20 insertions(+), 24 deletions(-) diff --git a/makechrootpkg.in b/makechrootpkg.in index 160ec9a..05826b2 100644 --- a/makechrootpkg.in +++ b/makechrootpkg.in @@ -15,29 +15,6 @@ m4_include(lib/archroot.sh) shopt -s nullglob -init_variables() { - default_makepkg_args=(--syncdeps --noconfirm --log --holdver --skipinteg) - makepkg_args=("${default_makepkg_args[@]}") - keepbuilddir=false - update_first=false - clean_first=false - run_namcap=false - temp_chroot=false - chrootdir= - passeddir= - makepkg_user= - declare -ga install_pkgs - declare -gi ret=0 - - bindmounts_ro=() - bindmounts_rw=() - - copy=$USER - [[ -n ${SUDO_USER:-} ]] && copy=$SUDO_USER - [[ -z "$copy" || $copy = root ]] && copy=copy - src_owner=${SUDO_USER:-$USER} -} - usage() { echo "Usage: ${0##*/} [options] -r <chrootdir> [--] [makepkg args]" echo ' Run this script in a PKGBUILD dir to build a package inside a' @@ -325,7 +302,26 @@ move_products() { # }}} main() { - init_variables + default_makepkg_args=(--syncdeps --noconfirm --log --holdver --skipinteg) + makepkg_args=("${default_makepkg_args[@]}") + keepbuilddir=false + update_first=false + clean_first=false + run_namcap=false + temp_chroot=false + chrootdir= + passeddir= + makepkg_user= + declare -a install_pkgs + declare -i ret=0 + + bindmounts_ro=() + bindmounts_rw=() + + copy=$USER + [[ -n ${SUDO_USER:-} ]] && copy=$SUDO_USER + [[ -z "$copy" || $copy = root ]] && copy=copy + src_owner=${SUDO_USER:-$USER} while getopts 'hcur:I:l:nTD:d:U:' arg; do case "$arg" in -- 2.14.1
Fixes regression in 2fd5931a8c67289a8a4acd327b3ce99a5d64c8c7 $run_namcap will always be set to "" `if $not_a_var; then ...; fi` is always truthful when $not_a_var is unset or equal to "" and the `then` clause will always be run. I'm not sure why global state variables need to be cloned locally for their sole explicit purpose. But for now this patch implements the minimum necessary work to properly pass the "do I want namcap" variable into prepare_chroot() according to the current logic flow. Note that I have still not thorougly tested makechrootpkg. Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> --- v2: somewhat more accurate commit message makechrootpkg.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/makechrootpkg.in b/makechrootpkg.in index ef3f2ec..fe9410d 100644 --- a/makechrootpkg.in +++ b/makechrootpkg.in @@ -408,7 +408,7 @@ main() { download_sources "$copydir" "$makepkg_user" - prepare_chroot "$copydir" "$USER_HOME" "$keepbuilddir" + prepare_chroot "$copydir" "$USER_HOME" "$keepbuilddir" "$run_namcap" if arch-nspawn "$copydir" \ --bind="$PWD:/startdir" \ -- 2.14.1
Don't use error-prone logic e.g. foo=true; if $foo ... This completely fails to act as expected when the variable is unset because of unrelated bugs. While this merely causes the default behavior to be "false" rather than "true" in such cases, it is better to fail to enable explicitly requested behavior (which will be noticed by the user) than to simply upgrade to this behavior for free (which may not seem to have any obvious cause). Signed-off-by: Eli Schwartz <eschwartz@archlinux.org> --- v2: somewhat more accurate commit message makechrootpkg.in | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/makechrootpkg.in b/makechrootpkg.in index fe9410d..ebea171 100644 --- a/makechrootpkg.in +++ b/makechrootpkg.in @@ -188,7 +188,7 @@ prepare_chroot() { local keepbuilddir=$3 local run_namcap=$4 - $keepbuilddir || rm -rf "$copydir/build" + [[ $keepbuilddir = true ]] || rm -rf "$copydir/build" local builduser_uid builduser_gid builduser_uid="${SUDO_UID:-$UID}" @@ -230,7 +230,7 @@ EOF declare -f _chrootbuild printf '_chrootbuild "$@" || exit\n' - if $run_namcap; then + if [[ $run_namcap = true ]]; then declare -f _chrootnamcap printf '_chrootnamcap || exit\n' fi -- 2.14.1
participants (3)
-
Eli Schwartz
-
Luke Shumaker
-
Luke Shumaker