Gaetan Bisson wrote:

If that's easy then it shouldn't be too hard for you to open a bug report on the tracker and submit a patch.

Then what component does the actual extracting? Is it libfetch? That would be an upstream moving target, because it comes from netbsd. Do you guys accept patches against libfetch in this case? clemens

On Fri, Nov 25, 2011 at 12:42, clemens fischer < ino-news@spotteswoode.dnsalias.org> wrote:

clemens fischer wrote:

...

Gaetan Bisson wrote:

...

If that's easy then it shouldn't be too hard for you to open a bug report on the tracker and submit a patch.

Then what component does the actual extracting? Is it libfetch? That would be an upstream moving target, because it comes from netbsd. Do you guys accept patches against libfetch in this case?

Seems libarchive is used to handle packages.

pacman.git lib/libalpm/add.c::perform_extraction() is hard to understand. It should be the place the warning I cited came from. If that is the case, ENOSPC wasn't the cause of it.

I don't understand two things:

- where exactly are files extracted to? If they go to /, how would people use pacman when root is mounted ro (in the case of eg. /.INSTALL)? If I can freely determine where files go via archive_entry_set_pathname(), where's the problem in using the value of TMPDIR as a prefix and move the file into its final position when libalpm declares the transaction ok?

- the file causing the warning was ".INSTALL". It contains upgrade/install scriptlets. How can a problem extracting an important file like this be considered a warning?

clemens

The .install files are contained in the package. It should be in /var/lib/${pkgname}/, you can extract the package file somewhere and look at it that way, or if you compiled the package yourself the .install file should be in the ${pkgdir} in the build folder. As far as the need to keep compile files across a crash, dont' build in /tmp/makepkg and write a simple clean up script in bash or a bash alias.

alias cleandir="rmd src && rmd pkg && rm -vf *.log && rm -vf *.log.*" That way everything will be saved in the build folder where the PKGBUILD is then after the compile and you can purge the folder. I rebuild all my packages, except the tool chain, tailored to my box so I do a lot of compiling and /tmp is mounted on tmpfs. Occassionally I take a look at memory usage while compiling large packages, ie qt, and rarely see any usage close to 1.5 GB of memory usage and thats with x, openbox, firefox, and some game open while I'm waiting for the compile to finish. Might I ask what you were compiling that threw the error? Myra -- Life's fun when your sick and psychotic!

On 26/11/11 02:40, clemens fischer wrote:

Gaetan Bisson wrote:

...

If that's easy then it shouldn't be too hard for you to open a bug report on the tracker and submit a patch.

Then what component does the actual extracting? Is it libfetch? That would be an upstream moving target, because it comes from netbsd. Do you guys accept patches against libfetch in this case?

Why would libfetch extract files.... libarchive extracts the file from an archive but pacman/libalpm tells it where to extract. So it would be a pacman issue. Allan

Hi, Sometimes you simply need more space that the one available in /tmp. In all my systems /tmp is in ram and as some machines have only 4Gb memory the available space in /tmp is about 2Gb only which is sufficient for most of the stuff but some compilations need more. Hector On 25 November 2011 02:28, Allan McRae wrote:

On 25/11/11 09:18, clemens fischer wrote:

...

Hi,

would it be possible to let pacman, libalpm and libfetch honor the environment variable TMPDIR?

I mean, this is stupid.  Many people keep /tmp in RAM, on a tmpfs, and make it big enough, but not too big, as it takes away RAM when getting loaded.

  #define TMP_LOC "TMPDIR"   #include  /* getenv(3) */   #include  /* strncat(3) */   td = getenv(TMP_LOC);   /* cleanup for safety: td = cleanup(td); */   tmpdir = strncat(td, "/", 1);

What can be so difficult about this?

I got this pesty "warning: warning given when extracting .INSTALL (Write failed)" today, and when looking at pacman, libalpm and libfetch with strings(1), none of them seem to check TMPDIR!

File a bug report or at least post to the pacman-dev list.  Then relevant people might actually see your request.

But I do not get it...   what is the point of having /tmp in RAM and then setting TMPDIR elsewhere?

Allan

-- Hector Martínez-Seara Monné mail: hseara@gmail.com Tel: +34656271145 Tel: +358442709253

Allan McRae wrote:

On 25/11/11 09:18, clemens fischer wrote:

...

would it be possible to let pacman, libalpm and libfetch honor the environment variable TMPDIR?

File a bug report or at least post to the pacman-dev list. Then relevant people might actually see your request.

I'm not "relevant" enough to post to any dev lists.

But I do not get it... what is the point of having /tmp in RAM and then setting TMPDIR elsewhere?

This has been answered by Hector Martinez-Seara. Sometimes, there's not enough space in /tmp ... clemens

On Fri, Nov 25, 2011 at 4:32 AM, Mauro Santos wrote:

On 25-11-2011 07:46, C Anthony Risinger wrote:

...

...  however, i would consider it a bug for applications to store *very* large files (exceeding 50-100M or so) in /tmp -- /var/tmp would be more appropriate, even for ephemeral/transient files -- idk

Just out of curiosity, why do you say that? Is it a good practice rule or something like that?

i'm not sure about the best-practice aspect, but it's what i do at least ... reasoning being /var/tmp is more likely to be backed by permanent storage (read: larger), and since it's more "persistent-ish" than /tmp, there's a better chance it'll still be there if my program were to terminate early/crash/etc. i suppose if i have to work with that much data, i'd rather not have to re-extract/re-compute/re-<whatever> if the app chokes early. i don't know if thats a sound reason (FHS doesn't specify relative sizing, or set any size expectations at all), but this tend to hold in practice. in my mind, /tmp is for rapid processing of data in chunks (eg, like a buffer), and should be cleared often by your app as it works -- /var/tmp can hold the entire dataset if need be. ... data must still be throwaway in the end. ultimately, if you *really* want data to stick around, and it *really* is important ... then don't let the only copy of it lie around in a directory called "tmp" :-) ... use your app's /var/lib instead. -- C Anthony

On Fri, Nov 25, 2011 at 10:55:55 -0600, Leonid Isaev wrote:

Actually, what is stupid is keeping /tmp in RAM. It is an important dir, where you might have an valuable info in case of a system crash. I could never understand the logic behind this choice.

Reducing disk i/o. Geert -- geert.hendrickx.be :: geert@hendrickx.be :: PGP: 0xC4BB9E9F This e-mail was composed using 100% recycled spam messages!

On Fri, Nov 25, 2011 at 6:31 PM, Leonid Isaev wrote:

On Fri, 25 Nov 2011 18:07:18 +0100 Geert Hendrickx wrote:

...

On Fri, Nov 25, 2011 at 10:55:55 -0600, Leonid Isaev wrote:

...

Actually, what is stupid is keeping /tmp in RAM. It is an important dir, where you might have an valuable info in case of a system crash. I could never understand the logic behind this choice.

Reducing disk i/o.

      Geert

I find this a very weak excuse, because the normal desktop operation is not I/O bound, and the dafaults must be safest. If you compile a lot/use a lot of DB stuff, just mount /tmp to RAM in fstab but this is a special case.

Note that: 1) FHS says: "Programs must not assume that any files or directories in /tmp are preserved between invocations of the program." 2) the contents of /tmp is deleted by initscripts on boot, so if you want to access stuff in /tmp after an unclean shutdown you somehow have to circumvent that. Given the above, there is no reason not to use tmpfs on /tmp (and plenty of reasons to do so). If extra space is required on /tmp, then the most efficient solution is to add to the available swap space. If you have important data, don't put it in /tmp or /var/tmp as neither has any guarantees about persistence. -t

Leonid Isaev wrote:

On Fri, 25 Nov 2011 18:07:18 +0100 Geert Hendrickx <...> wrote:

...

On Fri, Nov 25, 2011 at 10:55:55 -0600, Leonid Isaev wrote:

...

Actually, what is stupid is keeping /tmp in RAM. It is an important dir, where you might have an valuable info in case of a system crash. I could never understand the logic behind this choice.

/tmp is important, yes, but it is for _temporary_ files. If they should be preserved across a system crash, use another directory for them! Haven't you ever wondered about /var/tmp and the like?

...

Reducing disk i/o.

I find this a very weak excuse, because the normal desktop operation is not I/O bound, and the dafaults must be safest. If you compile a lot/use a lot of DB stuff, just mount /tmp to RAM in fstab but this is a special case.

Well, I do a lot of compiles on my desktop system. I sencerely hope you accept this excuse. I consider pacman installs important enough to warrent the users choice as to where extracted temp files go. clemens

On 28 November 2011 18:04, David Rosenstrauch wrote:

On 11/25/2011 11:55 AM, Leonid Isaev wrote:

...

Actually, what is stupid is keeping /tmp in RAM. It is an important dir, where you might have an valuable info in case of a system crash. I could never understand the logic behind this choice.

Actually, I think it makes a lot of sense. It lets you truly use it as a repository for "temporary" files. Any files written there will get automatically wiped out at the next boot.

I do this routinely on all my boxes. In addition, any files that I download I put in /tmp. If it turns out to be important, I copy it elsewhere. Else, it's automatically gone at the next reboot.

Contrast that with my Windows machine, where I have a "temp" folder on disk that I use for the same purpose. I have to always remember to go back and delete files from there when I'm finished with them.

DR

I concur 100%! Leonid is obviously using the TMP folder for the wrong purpose. It is called that for a reason Silly Goose! If I recall correctly, the rc.sysinit script even deletes the files on the next boot in /tmp/. Frankly for a temporary directory it's faster and saves your hard drive writes. Calvin Morrison

2011/11/29 "Jérôme M. Berger" :

David Rosenstrauch wrote:

...

On 11/25/2011 11:55 AM, Leonid Isaev wrote:

...

Actually, what is stupid is keeping /tmp in RAM. It is an important dir, where you might have an valuable info in case of a system crash. I could never understand the logic behind this choice.

Actually, I think it makes a lot of sense.  It lets you truly use it as a repository for "temporary" files.  Any files written there will get automatically wiped out at the next boot.

       And if your machine only boots very rarely (because it runs continuously or because you hibernate it instead of rebooting) then your "temporary" folder is never cleaned up. The solution that makes the most sense is to have /tmp on a disk and to use tmpwatch [1][2] in a cron job to clean it up regularly.

               Jerome

[1] http://fedorahosted.org/tmpwatch/ [2] http://aur.archlinux.org/packages.php?ID=23510

You should also be able to use /usr/lib/initscripts/arch-tmpfiles, which is what we use at boot, and which should have a similar functionality to tmpwatch (if you are using systemd you'll get a similar functionality for free). -t

On 11/29/2011 05:20 PM, clemens fischer wrote:

With tmpwatch one gets to choose files not accessed or modified for a certain period, and it needs no config file. Arch-tmpfiles, OTOH, would require such a thing.

Then again, a simple "find<some-dirs> -atime +<time-spec> -exec /bin/rm '{}' +" does about the same as tmpwatch.

Use -execdir instead, for security reasons (to protect against race conditions at least a little bit better). Or even better, just use -delete, which is built into find and also does everything to make the command able to delete long-unaccessed directories too. Still, for something this sensitive to mistakes*, I'd be more likely to trust a command made specifically for the purpose. For example, web search found me a tmpwatch man-page that says various things it's careful about: "When changing directories, tmpwatch is very sensitive to possible race conditions and will exit with an error if one is detected. It does not follow symbolic links in the directories it's cleaning (even if a symbolic link is given as its argument), will not switch filesystems, and only removes empty directories and regular files." ... and I think there's more. (disclosure: I don't need it personally, as tmpfs /tmp meets my needs.) -Isaac * at least: shared /tmp is a bit of a security disaster ; programs like X keep socket-type files there ; perhaps more: how many people know all of these parts well enough to write robust generic /tmp-related code?

Leonid Isaev wrote:

On Tue, 29 Nov 2011 19:50:46 +0100 "Jérôme M. Berger" wrote:

...

And if your machine only boots very rarely (because it runs continuously or because you hibernate it instead of rebooting) then your "temporary" folder is never cleaned up. The solution that makes the most sense is to have /tmp on a disk and to use tmpwatch [1][2] in a cron job to clean it up regularly.

Jerome

[1] http://fedorahosted.org/tmpwatch/ [2] http://aur.archlinux.org/packages.php?ID=23510

I am not sure what you mean, but we have uptimes averaging 170 days on the cluster (arch/rhel/ubuntu) and never had a single problem with overfull ext2 /tmp (FS size ~10Gb).

You can afford to waste *10Gb of RAM* with files that are not accessed for over 5 months? Man, you're rich :)

Again, you are thinking pure desktop (even not workstation) -- the most important file in your /tmp is a youtube video. What about various backup solutions which run continuously over the above 5 month period? Or various user data which they put in /tmp? Or data from compilation? Or situations when RAM is a resource?

No, actually I'm thinking mostly server (with an average uptime of 6 months). The long uptime means that cleaning /tmp on boot makes no sense since "booting" happens so rarely. Therefore some other means of cleaning it is required and cron+tmpwatch is the best way to do it (you could of course roll your own script to replace tmpwatch, but why go to the trouble when someone else has already done it and tested it in all the stupid corner cases with race conditions, symlinks and so on). Another issue with force cleaning /tmp on boot is that those twice-a-year reboots are often due to external problems (power outage for example). In that case, it is often useful to still have access to the files you were using 5 min before the reboot even if those files will stop being useful in a day or two and can then safely be removed.

Hibernating is a purely windows concept, doing it on a linux machine is basically looking for trouble, especially because hibernation gives no benefits over shutting down.

I never reboot my laptop unless I just upgraded the kernel and I don't have any issues. Hibernation has two benefits over shutting down: - Waking up is faster (ok, that's a small benefit because booting Arch is very fast, but still...) - When waking up from hibernation, you get your desktop exactly the way you left it, including any open files and tasks in progress. This is especially useful when using a laptop on the move: start working on something in the train, hibernate when you arrive at your station, then wake up when you reach your desk and immediately pick up where you left off). This has nothing to do with windows (although I admit that hibernating is a purely laptop concept: I don't hibernate my desktop PC and even less my servers).

And IMHO putting a simple hook into /etc/pm is much more rational than having yet another daemon.

What other daemon? Cron is already running anyway, so all that is needed is a simple hook in /etc/cron.daily. Jerome -- mailto:jeberger@free.fr http://jeberger.free.fr Jabber: jeberger@jabber.fr

-------- Original Message -------- Subject: Re: [arch-general] pacman/libalpm/libfetch do not honor TMPDIR Date: Thu, 01 Dec 2011 08:35:37 +1300 From: Ross Hamblin Reply-To: rosscoad@gmail.com To: General Discussion about Arch Linux On 01/12/11 07:49, "Jérôme M. Berger" wrote:

Leonid Isaev wrote:

...

Hibernating is a purely windows concept, doing it on a linux machine is basically looking for trouble, especially because hibernation gives no benefits over shutting down.

I never reboot my laptop unless I just upgraded the kernel and I don't have any issues. Hibernation has two benefits over shutting down: - Waking up is faster (ok, that's a small benefit because booting Arch is very fast, but still...) - When waking up from hibernation, you get your desktop exactly the way you left it, including any open files and tasks in progress. This is especially useful when using a laptop on the move: start working on something in the train, hibernate when you arrive at your station, then wake up when you reach your desk and immediately pick up where you left off).

This has nothing to do with windows (although I admit that hibernating is a purely laptop concept: I don't hibernate my desktop PC and even less my servers).

Not entirely a laptop concept, I set my desktop to hibernate if UPS battery runs low. That way when power resumes I continue working from where I was, less downtime for me. Ross.