[arch-general] Bashification of initscripts for moderate speedup
I have spent some time over the last month or so rewriting the Arch initscripts in idiomatic bash -- since they rely on bash-specific features, trying to keep them fairly POSIX is rather a waste of time and a net performance loss -- using bash-style conditionals is about 30% faster when nothing needs to be touched on the filesystem and about 200% easier to read, and you can use bash regexps in place of most trivial uses of sed, grep, and awk when mangling parameters or variables -- the fewer short-lived processes we fork the better. You can browse changes I have made in the git repo @ http://git.fnordovax.org/arch-initscripts/log/?h=bashification and you can checkout my changes to play around with from git://fnordovax.org/~victor/arch-initscripts There is a gigantic whitespace fixup patch at the end, all it does is fix all the whitespace errors I introduced while hacking and reformat everything to a consistent indentation style (emacs shell-script mode, if you are interested). I have tested these changes on a VM and on my local laptop. I inlcuded a PKGBUILD that will build an initscripts-git package, but be sure to backup and restore your rc.*local, inittab, and rc.conf files because they will get nuked the first time you install it. Questions, comments, flames, etc. welcome. I can also spam the list with patches if y'all really want. -- Victor Lowther LPIC2 UCP RHCE
On Sun, Jun 27, 2010 at 11:10 PM, Victor Lowther <victor.lowther@gmail.com> wrote:
Questions, comments, flames, etc. welcome.
why go this way instead of the other? (clarification why go deeper into bash instead of trying to posix-ify the scripts) -- Caleb Cushing http://xenoterracide.blogspot.com
On Mon, Jun 28, 2010 at 7:42 AM, Caleb Cushing <xenoterracide@gmail.com> wrote:
On Sun, Jun 27, 2010 at 11:10 PM, Victor Lowther <victor.lowther@gmail.com> wrote:
Questions, comments, flames, etc. welcome.
why go this way instead of the other? (clarification why go deeper into bash instead of trying to posix-ify the scripts)
Because we are never going to eliminate arrays from rc.conf. -Dan
On Mon 28 Jun 2010 08:04 -0500, Dan McGee wrote:
On Mon, Jun 28, 2010 at 7:42 AM, Caleb Cushing <xenoterracide@gmail.com> wrote:
On Sun, Jun 27, 2010 at 11:10 PM, Victor Lowther <victor.lowther@gmail.com> wrote:
Questions, comments, flames, etc. welcome.
why go this way instead of the other? (clarification why go deeper into bash instead of trying to posix-ify the scripts)
Because we are never going to eliminate arrays from rc.conf.
Well, it may still be beneficial to some. The scripts could be used with a different style rc.conf in other environments.
On Jun 28, 2010, at 8:59 AM, Loui Chang <louipc.ist@gmail.com> wrote:
On Mon 28 Jun 2010 08:04 -0500, Dan McGee wrote:
On Mon, Jun 28, 2010 at 7:42 AM, Caleb Cushing <xenoterracide@gmail.com
wrote: On Sun, Jun 27, 2010 at 11:10 PM, Victor Lowther <victor.lowther@gmail.com> wrote:
Questions, comments, flames, etc. welcome.
why go this way instead of the other? (clarification why go deeper into bash instead of trying to posix-ify the scripts)
Because we are never going to eliminate arrays from rc.conf.
Well, it may still be beneficial to some. The scripts could be used with a different style rc.conf in other environments.
Then it will not be Arch.
On Mon 28 Jun 2010 09:13 -0500, Victor Lowther wrote:
On Jun 28, 2010, at 8:59 AM, Loui Chang <louipc.ist@gmail.com> wrote:
On Mon 28 Jun 2010 08:04 -0500, Dan McGee wrote:
On Mon, Jun 28, 2010 at 7:42 AM, Caleb Cushing <xenoterracide@gmail.com> wrote:
On Sun, Jun 27, 2010 at 11:10 PM, Victor Lowther <victor.lowther@gmail.com> wrote:
Questions, comments, flames, etc. welcome.
why go this way instead of the other? (clarification why go deeper into bash instead of trying to posix-ify the scripts)
Because we are never going to eliminate arrays from rc.conf.
Well, it may still be beneficial to some. The scripts could be used with a different style rc.conf in other environments.
Then it will not be Arch.
Depends on what you define as Arch. I've heard that Arch is what you make of it. Hehe. I don't know if you could disqualify it from a difference of one file.
On Jun 28, 2010, at 9:49 AM, Loui Chang <louipc.ist@gmail.com> wrote:
On Mon 28 Jun 2010 09:13 -0500, Victor Lowther wrote:
On Jun 28, 2010, at 8:59 AM, Loui Chang <louipc.ist@gmail.com> wrote:
On Mon 28 Jun 2010 08:04 -0500, Dan McGee wrote:
On Mon, Jun 28, 2010 at 7:42 AM, Caleb Cushing <xenoterracide@gmail.com> wrote:
On Sun, Jun 27, 2010 at 11:10 PM, Victor Lowther <victor.lowther@gmail.com> wrote:
Questions, comments, flames, etc. welcome.
why go this way instead of the other? (clarification why go deeper into bash instead of trying to posix-ify the scripts)
Because we are never going to eliminate arrays from rc.conf.
Well, it may still be beneficial to some. The scripts could be used with a different style rc.conf in other environments.
Then it will not be Arch.
Depends on what you define as Arch. I've heard that Arch is what you make of it. Hehe. I don't know if you could disqualify it from a difference of one file.
For me, part of it is that bash is used pretty ubiquitously as the configuration and scripting language of choice. Changing that to posix sh in one of the main config files would be a big shift.
Actually I see the point of doing this. Arch is a modern distribution with the newest software around so why stuck with shell constructs which are probably dozens of years old? Lukas
Le lundi 28 juin 2010 à 17:55 +0200, Lukáš Jirkovský a écrit :
Actually I see the point of doing this. Arch is a modern distribution with the newest software around so why stuck with shell constructs which are probably dozens of years old?
Lukas
Yes. I definitely agree. We have to pray that a dev is interested in the patch now ;-) @Victor: You last commit that is supposed to fix whitespace mess in fact create a mess with whitespace. Also it changes all the modelines for vim: for example from # vim: set ft=sh sw=2 ts=2 et: to # vim: set ft=sh sw=4 ts=4 et: in fact, I would suggest that you revert your commit 59c7059e78491988c5d8d495c731fb66fe8f7224. It does not fix anything. check for yourself with git checkout bashification git diff master but git diff master HEAD~1 is clearly better
On Jun 29, 2010, at 5:55 AM, solsTiCe d'Hiver <solstice.dhiver@gmail.com> wrote:
Le lundi 28 juin 2010 à 17:55 +0200, Lukáš Jirkovský a écrit :
Actually I see the point of doing this. Arch is a modern distribution with the newest software around so why stuck with shell constructs which are probably dozens of years old?
Lukas
Yes. I definitely agree. We have to pray that a dev is interested in the patch now ;-)
Thomas seemed moderatly interested.
@Victor: You last commit that is supposed to fix whitespace mess in fact create a mess with whitespace. Also it changes all the modelines for vim: for example from # vim: set ft=sh sw=2 ts=2 et: to # vim: set ft=sh sw=4 ts=4 et:
When it comes to whitespace, I do whatever indent-region and whitespace-cleanup tell me to.
in fact, I would suggest that you revert your commit 59c7059e78491988c5d8d495c731fb66fe8f7224. It does not fix anything.
check for yourself with git checkout bashification git diff master but git diff master HEAD~1 is clearly better
I have uploaded some shiny bootchart runs from booting with and without bashification on my laptop to http://bugs.archlinux.org/20006
On Tue, Jun 29, 2010 at 8:20 AM, Victor Lowther <victor.lowther@gmail.com> wrote:
On Jun 29, 2010, at 5:55 AM, solsTiCe d'Hiver <solstice.dhiver@gmail.com> wrote:
Le lundi 28 juin 2010 à 17:55 +0200, Lukáš Jirkovský a écrit :
Actually I see the point of doing this. Arch is a modern distribution with the newest software around so why stuck with shell constructs which are probably dozens of years old?
Lukas
Yes. I definitely agree. We have to pray that a dev is interested in the patch now ;-)
Thomas seemed moderatly interested.
@Victor: You last commit that is supposed to fix whitespace mess in fact create a mess with whitespace. Also it changes all the modelines for vim: for example from # vim: set ft=sh sw=2 ts=2 et: to # vim: set ft=sh sw=4 ts=4 et:
When it comes to whitespace, I do whatever indent-region and whitespace-cleanup tell me to.
Which is change the modelines? No thanks. http://wiki.archlinux.org/index.php/DeveloperWiki:Bash_Coding_Style -Dan
On Jun 29, 2010, at 9:27 AM, Dan McGee <dpmcgee@gmail.com> wrote:
On Tue, Jun 29, 2010 at 8:20 AM, Victor Lowther <victor.lowther@gmail.com> wrote:
On Jun 29, 2010, at 5:55 AM, solsTiCe d'Hiver <solstice.dhiver@gmail.com
wrote:
Le lundi 28 juin 2010 à 17:55 +0200, Lukáš Jirkovský a écrit :
Actually I see the point of doing this. Arch is a modern distribution with the newest software around so why stuck with shell constructs which are probably dozens of years old?
Lukas
Yes. I definitely agree. We have to pray that a dev is interested in the patch now ;-)
Thomas seemed moderatly interested.
@Victor: You last commit that is supposed to fix whitespace mess in fact create a mess with whitespace. Also it changes all the modelines for vim: for example from # vim: set ft=sh sw=2 ts=2 et: to # vim: set ft=sh sw=4 ts=4 et:
When it comes to whitespace, I do whatever indent-region and whitespace-cleanup tell me to.
Which is change the modelines? No thanks.
http://wiki.archlinux.org/index.php/DeveloperWiki:Bash_Coding_Style
I know we are getting into holy war territory here, but: Overall, they are sane, except for using tabs vs. spaces (at least with spaces it is never ambiguous how far you wanted a given indent to be in a world where tab does not always map to a set number of spaces), and 8 character indentation (leads to excessive horizontal scrolling in any moderatly complex nested flow control situation. I prefer 4 space indentation, it tends to align nicely with most flow control statements), and using the source keyword instead of . to source files (personal preference from also maintaining a package written in posix sh). We might want to steal some content from http://mywiki.wooledge.org/BashGuide/Practices to update the Arch bash coding page.
-Dan
On 06/29/2010 11:44 AM, Victor Lowther wrote:
We might want to steal some content from http://mywiki.wooledge.org/BashGuide/Practices to update the Arch bash coding page.
Regardless which way the bashification issue goes, I agree with Victor, the above contains a very good rule-set for bash coding and style. -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com
On Tue 29 Jun 2010 09:27 -0500, Dan McGee wrote:
Which is change the modelines? No thanks.
http://wiki.archlinux.org/index.php/DeveloperWiki:Bash_Coding_Style
Neat. Where does the 132 columns come from?
On Tue, Jun 29, 2010 at 12:47 PM, Loui Chang <louipc.ist@gmail.com> wrote:
On Tue 29 Jun 2010 09:27 -0500, Dan McGee wrote:
Which is change the modelines? No thanks.
http://wiki.archlinux.org/index.php/DeveloperWiki:Bash_Coding_Style
Neat. Where does the 132 columns come from?
Back in the day of real physical terminals, they were typically made to display 80 columns on the screen. At some point, "wide terminals" came out, supporting 132 columns. I have no idea where the number came from, but both 80 and 132 are standard column widths used for normal and wide terminals.
On Tue, Jun 29, 2010 at 1:57 PM, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
I have no idea where the number came from, but both 80 and 132 are standard column widths used for normal and wide terminals.
should be 78 columns I do believe that's the standard for kernel/git it allows for email comments more easily apparently. -- Caleb Cushing http://xenoterracide.blogspot.com
On Wed, Jun 30, 2010 at 10:00 AM, Caleb Cushing <xenoterracide@gmail.com> wrote:
On Tue, Jun 29, 2010 at 1:57 PM, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
I have no idea where the number came from, but both 80 and 132 are standard column widths used for normal and wide terminals.
should be 78 columns I do believe that's the standard for kernel/git it allows for email comments more easily apparently.
No, physical terminals went to 80 columns. The 78 number was likely related to editor chrome, such as line numbering... I'm not sure of the details. I believe email typically has 78 width due to the fact that "> " is expected for replies.
Excerpts from Aaron Griffin's message of 2010-06-30 17:55:40 +0200:
On Wed, Jun 30, 2010 at 10:00 AM, Caleb Cushing <xenoterracide@gmail.com> wrote:
On Tue, Jun 29, 2010 at 1:57 PM, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
I have no idea where the number came from, but both 80 and 132 are standard column widths used for normal and wide terminals.
should be 78 columns I do believe that's the standard for kernel/git it allows for email comments more easily apparently.
No, physical terminals went to 80 columns. The 78 number was likely related to editor chrome, such as line numbering... I'm not sure of the details.
I believe email typically has 78 width due to the fact that "> " is expected for replies.
I also read about 72, and it's my current setting for vim when editing mails, but I have no idea where that comes from... -- Regards, Philipp -- "Wir stehen selbst enttäuscht und sehn betroffen / Den Vorhang zu und alle Fragen offen." Bertolt Brecht, Der gute Mensch von Sezuan
On Wed 30 Jun 2010 18:48 +0200, Philipp Überbacher wrote:
Excerpts from Aaron Griffin's message of 2010-06-30 17:55:40 +0200:
On Wed, Jun 30, 2010 at 10:00 AM, Caleb Cushing <xenoterracide@gmail.com> wrote:
On Tue, Jun 29, 2010 at 1:57 PM, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
I have no idea where the number came from, but both 80 and 132 are standard column widths used for normal and wide terminals.
should be 78 columns I do believe that's the standard for kernel/git it allows for email comments more easily apparently.
No, physical terminals went to 80 columns. The 78 number was likely related to editor chrome, such as line numbering... I'm not sure of the details.
I believe email typically has 78 width due to the fact that "> " is expected for replies.
I also read about 72, and it's my current setting for vim when editing mails, but I have no idea where that comes from...
Yeah I generally use 72 for emails.
On Mon, Jun 28, 2010 at 11:03 AM, Victor Lowther <victor.lowther@gmail.com> wrote:
For me, part of it is that bash is used pretty ubiquitously as the configuration and scripting language of choice. Changing that to posix sh in one of the main config files would be a big shift.
there seems to be a shift to dash as the main system shell in some distro's these days... and as openrc show's it's possible to do things similar to how arch does it (and gentoo used to) without bash. -- Caleb Cushing http://xenoterracide.blogspot.com
On Mon 28 Jun 2010 10:03 -0500, Victor Lowther wrote:
On Jun 28, 2010, at 9:49 AM, Loui Chang <louipc.ist@gmail.com> wrote:
Depends on what you define as Arch. I've heard that Arch is what you make of it. Hehe. I don't know if you could disqualify it from a difference of one file.
For me, part of it is that bash is used pretty ubiquitously as the configuration and scripting language of choice. Changing that to posix sh in one of the main config files would be a big shift.
Ah. For me Arch is much more than bash configs or scripts. They could all become POSIX compatible and I'd still see it as Arch. If Arch started to require python, or more I might start to see it differently - it would be harder to maintain a slimmer system.
On Jun 28, 2010, at 7:42 AM, Caleb Cushing <xenoterracide@gmail.com> wrote:
On Sun, Jun 27, 2010 at 11:10 PM, Victor Lowther <victor.lowther@gmail.com> wrote:
Questions, comments, flames, etc. welcome.
why go this way instead of the other? (clarification why go deeper into bash instead of trying to posix-ify the scripts)
Because arrays and associative arrays are very useful, using native bash costructs instead of posix gives you a decent speed boost, and the main source of slowdowns in the boot sequence is I/O based -- mounting filesystems and launching programs. Serious boot speed optimization is a matter of optimizing I/O.
-- Caleb Cushing
On 06/28/10 09:35, Victor Lowther wrote:
On Jun 28, 2010, at 7:42 AM, Caleb Cushing <xenoterracide@gmail.com> wrote:
On Sun, Jun 27, 2010 at 11:10 PM, Victor Lowther <victor.lowther@gmail.com> wrote:
Questions, comments, flames, etc. welcome.
why go this way instead of the other? (clarification why go deeper into bash instead of trying to posix-ify the scripts)
...and the main source of slowdowns in the boot sequence is I/O based -- mounting filesystems and launching programs.
IIRC, Busybox shell can get notable speed boost by incorporating versions of tools like sed into the same busybox executable, such that it often doesn't have to fork and load other short-lived programs. (I believe it doesn't do bash-array-syntax.) -Isaac
On Mon, Jun 28, 2010 at 12:36 PM, Isaac Dupree <ml@isaac.cedarswampstudios.org> wrote:
IIRC, Busybox shell can get notable speed boost by incorporating versions of tools like sed into the same busybox executable, such that it often doesn't have to fork and load other short-lived programs.
I don't think it loads them into it... it provides them... for example running busybox vi will load busybox's version of vi. Doesn't mean this isn't a good thing... symlinking vi to busybox and then running that vi is the same as typing busybox vi. -- Caleb Cushing http://xenoterracide.blogspot.com
On Jun 28, 2010, at 11:36 AM, Isaac Dupree <ml@isaac.cedarswampstudios.org
wrote:
On 06/28/10 09:35, Victor Lowther wrote:
On Jun 28, 2010, at 7:42 AM, Caleb Cushing <xenoterracide@gmail.com> wrote:
On Sun, Jun 27, 2010 at 11:10 PM, Victor Lowther <victor.lowther@gmail.com> wrote:
Questions, comments, flames, etc. welcome.
why go this way instead of the other? (clarification why go deeper into bash instead of trying to posix-ify the scripts)
...and the main source of slowdowns in the boot sequence is I/O based -- mounting filesystems and launching programs.
IIRC, Busybox shell can get notable speed boost by incorporating versions of tools like sed into the same busybox executable, such that it often doesn't have to fork and load other short-lived programs.
You can eliminate a fair number of trivial uses of grep, awk, and sed by using all of the parameter expansion modifiers that bash offers, and by using the native regex support from bash 3 and above. I made these changes wherever it seemed appropriate in my changes.
(I believe it doesn't do bash-array-syntax.)
-Isaac
Am 28.06.2010 05:10, schrieb Victor Lowther:
I have spent some time over the last month or so rewriting the Arch initscripts in idiomatic bash -- since they rely on bash-specific features, trying to keep them fairly POSIX is rather a waste of time and a net performance loss -- using bash-style conditionals is about 30% faster when nothing needs to be touched on the filesystem and about 200% easier to read, and you can use bash regexps in place of most trivial uses of sed, grep, and awk when mangling parameters or variables -- the fewer short-lived processes we fork the better.
You can browse changes I have made in the git repo @ http://git.fnordovax.org/arch-initscripts/log/?h=bashification and you can checkout my changes to play around with from git://fnordovax.org/~victor/arch-initscripts
Okay, those are LOTS of patches. I agree with most of them, but not all. However, reviewing and commenting them like this is a bit tedious - can you use git send-email to send them to this list as a patch series, so I can comment properly?
On Jun 30, 2010, at 12:40 PM, Thomas Bächler <thomas@archlinux.org> wrote:
Am 28.06.2010 05:10, schrieb Victor Lowther:
I have spent some time over the last month or so rewriting the Arch initscripts in idiomatic bash -- since they rely on bash-specific features, trying to keep them fairly POSIX is rather a waste of time and a net performance loss -- using bash-style conditionals is about 30% faster when nothing needs to be touched on the filesystem and about 200% easier to read, and you can use bash regexps in place of most trivial uses of sed, grep, and awk when mangling parameters or variables -- the fewer short-lived processes we fork the better.
You can browse changes I have made in the git repo @ http://git.fnordovax.org/arch-initscripts/log/?h=bashification and you can checkout my changes to play around with from git://fnordovax.org/~victor/arch-initscripts
Okay, those are LOTS of patches. I agree with most of them, but not all. However, reviewing and commenting them like this is a bit tedious - can you use git send-email to send them to this list as a patch series, so I can comment properly?
Sure. I will do so tonight.
2010-06-28 05:10, Victor Lowther:
You can browse changes I have made in the git repo @ http://git.fnordovax.org/arch-initscripts/log/?h=bashification
The feed link appears broken to me. <link rel='alternate' title='Atom feed' href='http://git.fnordovax.orghttp://git.fnordovax.org/arch-initscripts/atom/?h=ba...' type='application/atom+xml'/> -- kujub (whining)
On Jul 2, 2010, at 10:35 AM, "Kurt J. Bosch" <kjb-temp-2009@alpenjodel.de
wrote:
2010-06-28 05:10, Victor Lowther:
You can browse changes I have made in the git repo @ http://git.fnordovax.org/arch-initscripts/log/?h=bashification
The feed link appears broken to me.
<link rel='alternate' title='Atom feed' href='http://git.fnordovax.orghttp://git.fnordovax.org/arch-initscripts/atom/?h=ba...' type='application/atom+xml'/>
Thanks. I might eventually fix it.
-- kujub (whining)
On Sun, 2010-06-27 at 22:10 -0500, Victor Lowther wrote:
I have spent some time over the last month or so rewriting the Arch initscripts in idiomatic bash -- since they rely on bash-specific features, trying to keep them fairly POSIX is rather a waste of time and a net performance loss -- using bash-style conditionals is about 30% faster when nothing needs to be touched on the filesystem and about 200% easier to read, and you can use bash regexps in place of most trivial uses of sed, grep, and awk when mangling parameters or variables -- the fewer short-lived processes we fork the better.
You can browse changes I have made in the git repo @ http://git.fnordovax.org/arch-initscripts/log/?h=bashification and you can checkout my changes to play around with from git://fnordovax.org/~victor/arch-initscripts
OK, the bashification branch has been rebased and updated to fix issues pointed out in the patch-by-patch feedback. Thomas, if you want I will repost the branch to the mailing list. -- Victor Lowther LPIC2 UCP RHCE
Am 03.07.2010 21:07, schrieb Victor Lowther:
On Sun, 2010-06-27 at 22:10 -0500, Victor Lowther wrote:
I have spent some time over the last month or so rewriting the Arch initscripts in idiomatic bash -- since they rely on bash-specific features, trying to keep them fairly POSIX is rather a waste of time and a net performance loss -- using bash-style conditionals is about 30% faster when nothing needs to be touched on the filesystem and about 200% easier to read, and you can use bash regexps in place of most trivial uses of sed, grep, and awk when mangling parameters or variables -- the fewer short-lived processes we fork the better.
You can browse changes I have made in the git repo @ http://git.fnordovax.org/arch-initscripts/log/?h=bashification and you can checkout my changes to play around with from git://fnordovax.org/~victor/arch-initscripts
OK, the bashification branch has been rebased and updated to fix issues pointed out in the patch-by-patch feedback. Thomas, if you want I will repost the branch to the mailing list.
I will finish reviewing the rest of the patches first, when I get the time again. Sorry for being so slow on this.
participants (12)
-
Aaron Griffin
-
Caleb Cushing
-
Dan McGee
-
David C. Rankin
-
Isaac Dupree
-
Kurt J. Bosch
-
Loui Chang
-
Lukáš Jirkovský
-
Philipp Überbacher
-
solsTiCe d'Hiver
-
Thomas Bächler
-
Victor Lowther