[pacman-dev] [PATCH 1/2] Revert "makepkg: calculate exact total file size"
btrfs's cow snapshots seem to do very strange things, causing du to not count data still in filesystem buffers. Repackaging the same build of handbrake within a chroot on brynhild, I witnessed (via bash xtrace) du reporting a different $pkgdir size every time the write_pkginfo() function ran. Unfortunately, replacing du with stat has its own slew of problems, mostly due to hard links (e.g. git, with 106 hardlinks to the same file). Working around this is neither fun, nor practical. As it turns out, all we needed here all along was a simple call to sync to flush writes to disk before calling du. This reverts commit b264fb9e9ddcc31dc8782390309421965e507383. Signed-off-by: Dave Reisner <dreisner@archlinux.org> --- configure.ac | 1 + scripts/Makefile.am | 1 + scripts/makepkg.sh.in | 7 ++++++- 3 files changed, 8 insertions(+), 1 deletion(-) diff --git a/configure.ac b/configure.ac index 12a6e65..52f8ad4 100644 --- a/configure.ac +++ b/configure.ac @@ -282,6 +282,7 @@ esac AM_CONDITIONAL([CYGWIN], test "x$host_os_cygwin" = "xyes") AM_CONDITIONAL([DARWIN], test "x$host_os_darwin" = "xyes") +AC_PATH_PROGS([DUPATH], [du], [du], [/usr/bin$PATH_SEPARATOR/bin] ) AC_SUBST(SIZECMD) AC_SUBST(SEDINPLACE) AC_SUBST(STRIP_BINARIES) diff --git a/scripts/Makefile.am b/scripts/Makefile.am index 727de25..328fbff 100644 --- a/scripts/Makefile.am +++ b/scripts/Makefile.am @@ -53,6 +53,7 @@ edit = sed \ -e 's|@BUILDSCRIPT[@]|$(BUILDSCRIPT)|g' \ -e 's|@SIZECMD[@]|$(SIZECMD)|g' \ -e 's|@SEDINPLACE[@]|$(SEDINPLACE)|g' \ + -e 's|@DUPATH[@]|$(DUPATH)|g' \ -e 's|@SCRIPTNAME[@]|$@|g' \ -e 's|@configure_input[@]|Generated from $@.sh.in; do not edit by hand.|g' diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index 89cd118..c2941b2 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -1156,7 +1156,8 @@ write_pkginfo() { else local packager="Unknown Packager" fi - local size="$(find . -print0 | xargs -0 @SIZECMD@ | awk '{ sum += $1 } END { print sum }')" + local size="$(@DUPATH@ -sk)" + size="$(( ${size%%[^0-9]*} * 1024 ))" msg2 "$(gettext "Generating %s file...")" ".PKGINFO" echo "# Generated by makepkg $myver" @@ -1264,6 +1265,10 @@ create_package() { PKGARCH=$CARCH fi + # flush buffers to disk. This avoids problems with du calculating the wrong + # installed package size, particularly on btrfs CoW snapshots. + sync + write_pkginfo $nameofpkg > .PKGINFO local comp_files=('.PKGINFO') -- 1.7.9.4
Use a simple default value parameter expansion instead of using an intermediate local variable to write the packager name in write_pkginfo. Signed-off-by: Dave Reisner <dreisner@archlinux.org> --- scripts/makepkg.sh.in | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in index c2941b2..4be17e5 100644 --- a/scripts/makepkg.sh.in +++ b/scripts/makepkg.sh.in @@ -1151,11 +1151,6 @@ check_license() { write_pkginfo() { local builddate=$(date -u "+%s") - if [[ -n $PACKAGER ]]; then - local packager="$PACKAGER" - else - local packager="Unknown Packager" - fi local size="$(@DUPATH@ -sk)" size="$(( ${size%%[^0-9]*} * 1024 ))" @@ -1171,7 +1166,7 @@ write_pkginfo() { echo "pkgdesc = $pkgdesc" echo "url = $url" echo "builddate = $builddate" - echo "packager = $packager" + echo "packager = ${PACKAGER:-Unknown Packager}" echo "size = $size" echo "arch = $PKGARCH" -- 1.7.9.4
On 15/03/12 06:41, Dave Reisner wrote:
btrfs's cow snapshots seem to do very strange things, causing du to not count data still in filesystem buffers. Repackaging the same build of handbrake within a chroot on brynhild, I witnessed (via bash xtrace) du reporting a different $pkgdir size every time the write_pkginfo() function ran.
Unfortunately, replacing du with stat has its own slew of problems, mostly due to hard links (e.g. git, with 106 hardlinks to the same file). Working around this is neither fun, nor practical.
As it turns out, all we needed here all along was a simple call to sync to flush writes to disk before calling du.
This reverts commit b264fb9e9ddcc31dc8782390309421965e507383.
Signed-off-by: Dave Reisner <dreisner@archlinux.org>
Argh... look at the commit message in 14474a32... That sync results in the Arch buildserver stalling for minutes. Allan
On Thu, Mar 15, 2012 at 07:11:07AM +1000, Allan McRae wrote:
On 15/03/12 06:41, Dave Reisner wrote:
btrfs's cow snapshots seem to do very strange things, causing du to not count data still in filesystem buffers. Repackaging the same build of handbrake within a chroot on brynhild, I witnessed (via bash xtrace) du reporting a different $pkgdir size every time the write_pkginfo() function ran.
Unfortunately, replacing du with stat has its own slew of problems, mostly due to hard links (e.g. git, with 106 hardlinks to the same file). Working around this is neither fun, nor practical.
As it turns out, all we needed here all along was a simple call to sync to flush writes to disk before calling du.
This reverts commit b264fb9e9ddcc31dc8782390309421965e507383.
Signed-off-by: Dave Reisner <dreisner@archlinux.org>
Argh... look at the commit message in 14474a32...
That sync results in the Arch buildserver stalling for minutes.
Allan
Ok fine. Back to the trenches I go... Some careful changes to makepkg and a simple stack trace catches du in the act of being broken. It receives a statbuf with the correct size of /usr/bin/ghb: newfstatat(AT_FDCWD, "usr/bin/ghb", {st_mode=S_IFREG|0755, st_size=16037808, ...}, AT_SYMLINK_NOFOLLOW) = 0 And then it writes the size completely wrong: write(1, "3456\tusr/bin/ghb\n", 17) = 17 I have no idea what's going on, but I'll keep digging. d
On Wed, Mar 14, 2012 at 07:48:06PM -0400, Dave Reisner wrote:
On Thu, Mar 15, 2012 at 07:11:07AM +1000, Allan McRae wrote:
On 15/03/12 06:41, Dave Reisner wrote:
btrfs's cow snapshots seem to do very strange things, causing du to not count data still in filesystem buffers. Repackaging the same build of handbrake within a chroot on brynhild, I witnessed (via bash xtrace) du reporting a different $pkgdir size every time the write_pkginfo() function ran.
Unfortunately, replacing du with stat has its own slew of problems, mostly due to hard links (e.g. git, with 106 hardlinks to the same file). Working around this is neither fun, nor practical.
As it turns out, all we needed here all along was a simple call to sync to flush writes to disk before calling du.
This reverts commit b264fb9e9ddcc31dc8782390309421965e507383.
Signed-off-by: Dave Reisner <dreisner@archlinux.org>
Argh... look at the commit message in 14474a32...
That sync results in the Arch buildserver stalling for minutes.
Allan
Ok fine. Back to the trenches I go...
Some careful changes to makepkg and a simple stack trace catches du in the act of being broken. It receives a statbuf with the correct size of /usr/bin/ghb:
newfstatat(AT_FDCWD, "usr/bin/ghb", {st_mode=S_IFREG|0755, st_size=16037808, ...}, AT_SYMLINK_NOFOLLOW) = 0
And then it writes the size completely wrong:
write(1, "3456\tusr/bin/ghb\n", 17) = 17
I have no idea what's going on, but I'll keep digging.
d
Replying to myself, because I've been having an monologue with myself all goddamn day over this. It seems that while the stat buffer reports the size properly, there's a measureable delay of a half second or so before the st_blocks member is populated. I can stick a 'sleep 1' into makepkg before the du call, and everything works. I hacked up du and added some lovely printf calls (since you can't use gdb for this...) without the delay... sb->st_size=16037808, sb->st_blocks=0 <- ghb ^ that's when we see problems, because du is measuring size based on (st_blkcnt*st_blocksize). With a 1 second delay... sb->st_size=16037808, sb->st_blocks=31328 <- ghb Yay! d
participants (2)
-
Allan McRae
-
Dave Reisner