[arch-projects] [mkinitcpio][PATCH 0/4] performance improvements
Hi all, Sharing some wisdom from my adventures with geninit[1] -- the attached patches refactor some of the more critical pieces of module discovery: all_modules, checked_modules, and auto_modules. The last patch is just another bit of low hanging froot I thought I'd knock off. Using HOOKS="base udev autodetect sata usbinput filesystems" with gzip compression for a single image: Before: real 0m3.461s user 0m1.815s sys 0m0.748s After: real 0m2.841s user 0m1.592s sys 0m0.671s Mind you, this is on a SATA2 SSD with /tmp on tmpfs so I expect something more pronounced on a rotating disk and/or non tmpfs /tmp. Times for a non-autodetect build are roughly unchanged, as expected. There is a much larger gain to be had here, but it requries touching a large amount of code. The major bottleneck in mkinitcpio is the constant calls to grep to check module existance prior to adding it to the image list. The alternative is to use the method presented in the gentoo wiki, which can be summarized as dropping the files into a temporary directory and creating the cpio archive using find|(bsd)cpio. The existance check then becomes a simple shell test which is several orders of magnitude faster than grep, matches with a higher degree of confidence, and removes the tedium of checking parent directory instance (as order is crucial with gen_init_cpio). I'd like to eventually do this, but it may be a little while before I get to it. Laying it out here now in case anyone wants to beat me to it, or tell me it's a silly idea. regards, dave [1] http://github.com/falconindy/geninit
sort/uniq the modaliases files' contents, before passing _all_ of them to modprobe -a. This cuts back on execution time as well as cleaning up the resulting list of modules. Signed-off-by: Dave Reisner <d@falconindy.com> --- functions | 15 ++++++--------- 1 files changed, 6 insertions(+), 9 deletions(-) diff --git a/functions b/functions index 4cd3136..a89fd92 100644 --- a/functions +++ b/functions @@ -41,16 +41,13 @@ in_array() { auto_modules () { - aliases="$(find /sys/devices/ -name modalias -exec cat {} +)" - mods="" - for a in $aliases; do - m="$(modprobe --set-version ${KERNELVERSION} --resolve-alias "$a")" - [ -n "$m" ] && mods="$mods $m" - done + IFS=$'\n' read -rd '' -a mods < \ + <(find /sys/devices -name modalias -exec sort -zu {} + | + xargs -0 modprobe --set-version "$KERNELVERSION" -aR | + sort -u) - echo "${mods}" | tr '-' '_' - [ -z "${mods}" ] && return 1 - return 0 + printf "%s\n" "${mods[@]//-/_}" + (( ${#mods[*]} )) } all_modules () -- 1.7.5.4
sanitize and print during module discovery. We also strip path names during this process and null terminate path names for safety. Signed-off-by: Dave Reisner <d@falconindy.com> --- functions | 13 +++++++++---- 1 files changed, 9 insertions(+), 4 deletions(-) diff --git a/functions b/functions index a89fd92..c93701d 100644 --- a/functions +++ b/functions @@ -52,11 +52,16 @@ auto_modules () all_modules () { - mods=$(find ${MODULEDIR} -name '*.ko' -or -name '*.ko.gz' 2>/dev/null | grep $@ | sort -u) + local -i count=0 + while read -r -d '' mod; do + (( ++count )) + mod=${mod##*/} + mod="${mod%.ko.*}" + printf '%s\n' "${mod//-/_}" + done < <(find "$MODULEDIR" -name '*.ko' -o -name '*.ko.gz' -print0 2>/dev/null | + grep -Zz "$@" | sort -zu) - echo "${mods}" - [ -z "${mods}" ] && return 1 - return 0 + (( count )) } checked_modules () -- 1.7.5.4
Do a single grep to compare the contents of the module file (if exists) against the results of all_modules for the provided arguments. If there's no module file, this is still just pass through to all_modules. Signed-off-by: Dave Reisner <d@falconindy.com> --- functions | 11 +++-------- 1 files changed, 3 insertions(+), 8 deletions(-) diff --git a/functions b/functions index c93701d..5aa8435 100644 --- a/functions +++ b/functions @@ -66,16 +66,11 @@ all_modules () checked_modules () { - if [ -e "${MODULE_FILE}" ]; then - for mod in $(all_modules $@); do - modname=$(get_module_name "${mod}") - if grep -q "^${modname}$" "${MODULE_FILE}"; then - echo ${modname} - fi - done + if [ -s "${MODULE_FILE}" ]; then + grep -xFf "$MODULE_FILE" <(all_modules "$@") return 1 else - all_modules ${@} + all_modules "$@" fi } -- 1.7.5.4
On 06/03/2011 03:13 PM, Dave Reisner wrote:
Do a single grep to compare the contents of the module file (if exists) against the results of all_modules for the provided arguments. If there's no module file, this is still just pass through to all_modules.
Signed-off-by: Dave Reisner<d@falconindy.com> --- functions | 11 +++-------- 1 files changed, 3 insertions(+), 8 deletions(-)
diff --git a/functions b/functions index c93701d..5aa8435 100644 --- a/functions +++ b/functions @@ -66,16 +66,11 @@ all_modules ()
checked_modules () { - if [ -e "${MODULE_FILE}" ]; then - for mod in $(all_modules $@); do - modname=$(get_module_name "${mod}") - if grep -q "^${modname}$" "${MODULE_FILE}"; then - echo ${modname} - fi - done + if [ -s "${MODULE_FILE}" ]; then + grep -xFf "$MODULE_FILE"<(all_modules "$@") return 1 else - all_modules ${@} + all_modules "$@" fi }
Hi Dave, I guess this commit cause an issue with this used in archiso_pxe_nbd hook: MODULES="nbd $(checked_modules "/drivers/net/" | grep -v -e "/irda/" -e "/phy/" -e "/plip" -e "/ppp" -e "/wimax/" -e "/wireless/") " Excluded paths are present in initramfs image. -- Gerardo Exequiel Pozzi \cos^2\alpha + \sin^2\alpha = 1
Signed-off-by: Dave Reisner <d@falconindy.com> --- mkinitcpio | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/mkinitcpio b/mkinitcpio index 0ebf7c9..d6f3f04 100755 --- a/mkinitcpio +++ b/mkinitcpio @@ -275,9 +275,9 @@ if [ "${HAS_MODULES}" = "y" ]; then install -m 644 -D "${BASEDIR}${mod}" "${TMPDIR}${mod}" done /sbin/depmod -b ${TMPDIR} ${KERNELVERSION} - add_file "${TMPDIR}/lib/modules/${KERNELVERSION}/modules.dep" "/lib/modules/${KERNELVERSION}/modules.dep" - add_file "${TMPDIR}/lib/modules/${KERNELVERSION}/modules.alias" "/lib/modules/${KERNELVERSION}/modules.alias" - add_file "${TMPDIR}/lib/modules/${KERNELVERSION}/modules.symbols" "/lib/modules/${KERNELVERSION}/modules.symbols" + for dmfile in modules.{dep,alias,symbols}; do + add_file "${TMPDIR}/lib/modules/${KERNELVERSION}/$dmfile" "/lib/modules/${KERNELVERSION}/$dmfile" + done fi status=0 -- 1.7.5.4
Am 03.06.2011 20:13, schrieb Dave Reisner:
Signed-off-by: Dave Reisner <d@falconindy.com> --- mkinitcpio | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/mkinitcpio b/mkinitcpio index 0ebf7c9..d6f3f04 100755 --- a/mkinitcpio +++ b/mkinitcpio @@ -275,9 +275,9 @@ if [ "${HAS_MODULES}" = "y" ]; then install -m 644 -D "${BASEDIR}${mod}" "${TMPDIR}${mod}" done /sbin/depmod -b ${TMPDIR} ${KERNELVERSION} - add_file "${TMPDIR}/lib/modules/${KERNELVERSION}/modules.dep" "/lib/modules/${KERNELVERSION}/modules.dep" - add_file "${TMPDIR}/lib/modules/${KERNELVERSION}/modules.alias" "/lib/modules/${KERNELVERSION}/modules.alias" - add_file "${TMPDIR}/lib/modules/${KERNELVERSION}/modules.symbols" "/lib/modules/${KERNELVERSION}/modules.symbols" + for dmfile in modules.{dep,alias,symbols}; do + add_file "${TMPDIR}/lib/modules/${KERNELVERSION}/$dmfile" "/lib/modules/${KERNELVERSION}/$dmfile" + done fi
status=0
Wow, we haven't touched this code in years. This is incomplete, we need the .bin files from depmod, which speed up modprobe operation in initramfs considerably.
On Jun 4, 2011 3:28 AM, "Thomas Bächler" <thomas@archlinux.org> wrote:
Am 03.06.2011 20:13, schrieb Dave Reisner:
Signed-off-by: Dave Reisner <d@falconindy.com> --- mkinitcpio | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/mkinitcpio b/mkinitcpio index 0ebf7c9..d6f3f04 100755 --- a/mkinitcpio +++ b/mkinitcpio @@ -275,9 +275,9 @@ if [ "${HAS_MODULES}" = "y" ]; then install -m 644 -D "${BASEDIR}${mod}" "${TMPDIR}${mod}" done /sbin/depmod -b ${TMPDIR} ${KERNELVERSION} - add_file "${TMPDIR}/lib/modules/${KERNELVERSION}/modules.dep"
"/lib/modules/${KERNELVERSION}/modules.dep"
- add_file "${TMPDIR}/lib/modules/${KERNELVERSION}/modules.alias" "/lib/modules/${KERNELVERSION}/modules.alias" - add_file "${TMPDIR}/lib/modules/${KERNELVERSION}/modules.symbols" "/lib/modules/${KERNELVERSION}/modules.symbols" + for dmfile in modules.{dep,alias,symbols}; do + add_file "${TMPDIR}/lib/modules/${KERNELVERSION}/$dmfile" "/lib/modules/${KERNELVERSION}/$dmfile" + done fi
status=0
Wow, we haven't touched this code in years. This is incomplete, we need the .bin files from depmod, which speed up modprobe operation in initramfs considerably.
Okay. I don't see this being any huge speedup but ill resubmit. D D
participants (4)
-
Dave Reisner
-
dave reisner
-
Gerardo Exequiel Pozzi
-
Thomas Bächler