[pacman-dev] Disk space checking branch "complete"

[Nezmer]

On Wed, Nov 17, 2010 at 08:22:14PM +0200, Nezmer wrote:
> On Wed, Nov 17, 2010 at 08:13:08PM +0200, Nezmer wrote:
> > On Tue, Nov 16, 2010 at 06:06:32PM -0600, Dan McGee wrote:
> > > On Tue, Nov 16, 2010 at 5:36 PM, Allan McRae <allan at archlinux.org> wrote:
> > > > On 17/11/10 03:25, Nezmer wrote:
> > > >>
> > > >> On Tue, Nov 16, 2010 at 09:03:37PM +1000, Allan McRae wrote:
> > > >>>
> > > >>> On 16/11/10 18:02, Bryce Gibson wrote:
> > > >>>>
> > > >>>> Hi Allan,
> > > >>>> I just wanted to ask, it looks like your patches will make a sync fail
> > > >>>> if it
> > > >>>> finds there's not enough space, is that correct?
> > > >>>> Because I'd suggest a warning may be more appropriate, especially for
> > > >>>> use
> > > >>>> cases like compressed filesystems.
> > > >>>> Cheers
> > > >>>> Bryce
> > > >>>> ps. I noticed that space checking can be completely disabled via
> > > >>>> pacman.conf, which may be seen as a suitable solution for people in
> > > >>>> these
> > > >>>> types of situation.
> > > >>>
> > > >>> Disabling checking in pacman.conf is what I would recommend in these
> > > >>> sort of cases.  These are the sort of things we will only find out
> > > >>> after we get some widespread usage of the feature.  I would be
> > > >>> interested in what the size calculations actually do on compressed
> > > >>> filesystems.
> > > >>>
> > > >>> Allan
> > > >>>
> > > >>>
> > > >>
> > > >> On the topic of FS compression(also FS caching and delayed allocation).
> > > >>
> > > >> The real issue I had was the calculation of install size when building
> > > >> the package. The calculated size was always *very* small. I addressed
> > > >> this issue with 2 patches. The 1st one (adding sync before running du)
> > > >> didn't fix anything. The 2nd patch worked pretty well.
> > > >>
> > > >>
> > > >> http://gitorious.org/pacman-bsd/pacman-bsd/commit/8b367d4441ba85b5548285d987afcfc84c4fcb3e
> > > >>
> > > >> http://gitorious.org/pacman-bsd/pacman-bsd/commit/cf3341fe591293a493bc925e0433a1f696b37d90
> > > >
> > > > I think that initial "sync" is not needed because du does one anyway. The
> > > > second one looks interesting so I have bookmarked it to look at later.
> > > 
> > > We've putzed around with this a few times, haven't we? This basically
> > > "reverts" this one:
> > > 
> > > commit 149839c5391e9a93465f86dbb8d095a0150d755d
> > > Author: Xavier Chantry <shiningxc at gmail.com>
> > > Date:   Mon May 26 23:46:01 2008 +0200
> > > 
> > >     du -b is not available on BSD, use du -k instead.
> > > 
> > >     This fixes FS#10459.
> > > 
> > >     There is apparently no portable ways to get the apparent size of a file,
> > >     like du -b does. So the best compromise seems to get the block size in kB,
> > >     and then convert that to byte so that we keep compatibility.
> > > 
> > >     Signed-off-by: Xavier Chantry <shiningxc at gmail.com>
> > >     Signed-off-by: Dan McGee <dan at archlinux.org>
> > > 
> > > There are three or so file sizes that matter:
> > > 1) du -s : what we do now, sums up actual taken space so would be more
> > > accurate with many < 4K files.
> > > 2) du -sb: what we did before and what is proposed, sums up apparent
> > > size, so does not necessarily best represent installed size (either
> > > sparse files or many <4K files would throw the number off). The 4K
> > > assumption also may not always hold...
> > > 3) du --tell-me-how-many-blocks-but-not-compressed : what seems like
> > > perhaps the ideal? I'm not sure, but this would basically
> > >     for file in tree:
> > >         total += ceil(filesize to 4K)
> > > 
> > > 1 is the most portable; 2 we need different flags all over the place;
> > > 3 we definitely don't seem to be able to use exiting tools but this
> > > would not be an awful one to write.
> > > 
> > > -Dan
> > > 
> > 
> > Is this 3 ?
> > 
> > install_size=0
> > for f in `find $pkgdir -type f`; do
> >   (( s=$(${SIZECMD} ${f})/1024 ))
> >   (( $(${SIZECMD} ${f})%1024 > 0 )) && (( s++ ))
> >   (( $s < 4 )) && (( s=4 ))
> >   (( install_size+=${s} ))
> > done
> > 
> > Unfortunately, this takes around 39 seconds in a 13085 files package.
> > 
> 
> Correction:
> 
> 
> install_size=0
> for f in `find $pkgdir -type f`; do
>   (( s=$(${SIZECMD} ${f})/1024 ))
>   (( $(${SIZECMD} ${f})%1024 )) && (( s++ ))
>   (( ${s}%4 )) && (( s+=${s}%4 ))
>   (( install_size+=${s} ))
> done
> 
> Takes around 40 seconds in a 13085 files package.
> 

Fail

As Dan said, This is too slow to be done in bash. I just want to correct
it for reference.


install_size=0
for f in `find $pkgdir -type f`; do
  (( s=$(${SIZECMD} ${f})/1024 ))
  (( $(${SIZECMD} ${f})%1024 )) && (( s++ ))
  (( ${s}%4 )) && (( s+=4-${s}%4 ))
  (( install_size+=${s} ))
done