[pacman-dev] Pacman Disk Space Issues
Good Evening- In following the posting at: http://bbs.archlinux.org/viewtopic.php?pid=502174#p502174. I was wondering what the status of the disk space issue was and if I could assist in working out a disk space checking function to incorporate into it. According to the afformentioned post, there is some contention as to where the problem lies. Additionally, I would like to be sure I am following the right procedure for submitting changes, as I have not previously done any work for the community, so please forgive any procedural mistakes or dumb questions, thanks. Matt
On Thu, Feb 19, 2009 at 7:06 PM, Matthew Wood <ybotspawn@gmail.com> wrote:
Good Evening-
In following the posting at: http://bbs.archlinux.org/viewtopic.php?pid=502174#p502174. I was wondering what the status of the disk space issue was and if I could assist in working out a disk space checking function to incorporate into it. According to the afformentioned post, there is some contention as to where the problem lies. Additionally, I would like to be sure I am following the right procedure for submitting changes, as I have not previously done any work for the community, so please forgive any procedural mistakes or dumb questions, thanks.
Proper procedure: http://projects.archlinux.org/git/?p=pacman.git;a=blob;f=submitting-patches (not sure if this is fully up to date or not) Bug regarding this issue: http://bugs.archlinux.org/task/11639 Look at where Dan links to git showing the old code - that may be a decent start. Regarding the check - we do get the "size to be installed" at the frontend for display - why can't we check freespace there and just fail before we even download or install anything? Dan?
Aaron Griffin wrote:
Regarding the check - we do get the "size to be installed" at the frontend for display - why can't we check freespace there and just fail before we even download or install anything? Dan? Its very sensible to add this feature - though its not going to prevent all possible issues.
Given the information at hand during package installation, the robust way would be: check available disk space in /var/cache/pacman/pkg/ and / before downloading - give appropriate warnings (new feature?) once downloaded, compare the available disk space in each mount point from /etc/mtab to the compressed package's content (new feature) * still take into account all errors regarding disk space during installation (currently-broken feature) ** * You can't tell where the package's files are going until you've already downloaded the package. If there are separate mount points for different directories in the root, disk space can't be calculated beforehand. Assuming there's only a / mount point will work for most Arch users - but not all. ** On a busy server or desktop, you can't assume that the disk space available at the beginning of the installation isn't going to change independently of pacman's actions. __________ Brendan Hide
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hey, I've personally ran into this before, but don't think that it is worth "fixing" in the current Pacman setup. User error is usually the cause of this, since an admin should always have adequate disk space before installing new packages. Preventing this is difficult, like Brendan said, available disk space may be taken by other causes when Pacman is running. Not to mention, some users like to split the filesystem into a multitude of partitions. However, I do have one unlikely suggestion that *could* combat this without much difficulty. I was thinking of this more to (partially) solve the issue of package updates being cut short by a hard shutdown. Basically, packages would be extracted to /var/lib/pacman/extract (or similair), and if extraction has not errored, files would be moved to their corresponding locations. This would keep the new, possibly incompatible files out of the system until they can almost instantly be moved over, except in the case of /var, or others being on different partitions. This has the drawback that the user must have roughly installed size*2 disk space free, since the old package and new package would both have to be extracted at once (I personally think this is almost a non-issue). The current Pacman implementation may already require that much free disk space, but I'm not sure. An alternative to the above is to create directories in the folders that the package extracts to, such as .pacextract in /bin. This would insure that all mounted partitions get a fast rename/move, mitigating potential data loss and allowing for perfect disk space checking. It could be done so that the .pacextract feature only runs on the bottom 2 or 3 directories (/usr, /usr/local, but not /usr/share/man/man8, which would have .pacextract/man8 in /usr/share/man), or on a predetermined set of directories in pacman.conf. I think my explanation might be a little confusing, so I will write it out in approximate psuedo-shell script. mkdir /usr/bin/.pacextract tar xf xz-4.999.8beta-1.pkg.tar.gz -C /usr/bin/.pacextract (with other flags so that tar only extracts the archive contents of /usr/bin) mv /usr/bin/.pacextract/* /usr/bin && rm -r /usr/bin/.pacextract The simpler (extract to /var/lib/pacman/extract) method does not offer the fast move when multiple partitions are used and perfect disk space utilization checking, but would be easier to code. Anyways, this idea is quite experimental. I do think it would thoroughly solve the disk space without too much difficulty, but I personally wouldn't use it soley for that. Cheers, Teran -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.10 (GNU/Linux) iF4EAREIAAYFAkmevj0ACgkQPsLC06eiyfHnKAD9H0nTW2EU8Z46rjh3PvZD+LT2 mjUUCvhBtuX6xyQANoIA/ipz7RN87qtEJT+N2J295GcqNQ66PUKbppcrM85qh6eL =LgkR -----END PGP SIGNATURE-----
On Fri, Feb 20, 2009 at 8:29 AM, Teran McKinney <sega01@go-beyond.org> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Hey,
I've personally ran into this before, but don't think that it is worth "fixing" in the current Pacman setup. User error is usually the cause of this, since an admin should always have adequate disk space before installing new packages. Preventing this is difficult, like Brendan said, available disk space may be taken by other causes when Pacman is running. Not to mention, some users like to split the filesystem into a multitude of partitions.
However, I do have one unlikely suggestion that *could* combat this without much difficulty. I was thinking of this more to (partially) solve the issue of package updates being cut short by a hard shutdown. Basically, packages would be extracted to /var/lib/pacman/extract (or similair), and if extraction has not errored, files would be moved to their corresponding locations. This would keep the new, possibly incompatible files out of the system until they can almost instantly be moved over, except in the case of /var, or others being on different partitions. This has the drawback that the user must have roughly installed size*2 disk space free, since the old package and new package would both have to be extracted at once (I personally think this is almost a non-issue). The current Pacman implementation may already require that much free disk space, but I'm not sure.
An alternative to the above is to create directories in the folders that the package extracts to, such as .pacextract in /bin. This would insure that all mounted partitions get a fast rename/move, mitigating potential data loss and allowing for perfect disk space checking. It could be done so that the .pacextract feature only runs on the bottom 2 or 3 directories (/usr, /usr/local, but not /usr/share/man/man8, which would have .pacextract/man8 in /usr/share/man), or on a predetermined set of directories in pacman.conf. I think my explanation might be a little confusing, so I will write it out in approximate psuedo-shell script.
mkdir /usr/bin/.pacextract tar xf xz-4.999.8beta-1.pkg.tar.gz -C /usr/bin/.pacextract (with other flags so that tar only extracts the archive contents of /usr/bin) mv /usr/bin/.pacextract/* /usr/bin && rm -r /usr/bin/.pacextract
The simpler (extract to /var/lib/pacman/extract) method does not offer the fast move when multiple partitions are used and perfect disk space utilization checking, but would be easier to code. Anyways, this idea is quite experimental. I do think it would thoroughly solve the disk space without too much difficulty, but I personally wouldn't use it soley for that.
Cheers, Teran
This was done by the freebsd guys who also make libarchive and they abandoned it. I discussed it at length, with the following idea: * packages are extracted to some dir i.e. /var/lib/pacman/extract/$pkgname-$pkgver-$pkgrel/ * for each file to be installed if file in filesystem, move existing file out of the way (rename to *.pactrans.$PID) copy temp file to filesystem if error, remove ALL file, rename ALL .pactrans files back in place This would solve things, but the big problem with this approach is speed. It adds HUGE amounts of disk IO and, according to the people who made libarchive, slows the process down so noticeably, it isn't even worth it. If I were to modify this process, I would use a tmpfs dir for the /extract/ dir, and move existing files THERE (with the .pactrans addition). This would save us all the disk IO, but is going to get all the more complex. This is, in effect, what Dan is talking about when he says "real transactions" - the ability for pacman to abort in the middle of an install and roll everything back successfully. If we implement "real transactions", the disk space issue becomes a moot point.
participants (4)
-
Aaron Griffin
-
Brendan Hide
-
Matthew Wood
-
Teran McKinney