[arch-dev-public] Punishment needed for not building in a chroot
Hi, We have been through this many times... you should always build in a clean chroot. But there are continuously bugs about packages linking to non-deps. We should never have such bugs. e.g. (FS#17409)
How did that get to linking to jack without jack being in its dependency tree? Poor packaging... The tools are very simple to use and are described in the wiki (http://wiki.archlinux.org/index.php/DeveloperWiki:Building_in_a_Clean_Chroot). There is _no_ excuse not to use them. The are minor changes needed for doing i686 builds on x86_64 and vise versa, but there are plenty of us doing that so help is available. Ask for help if needed. So, we need a creative punishment for those that causes bugs by not building in a clean chroot. It is too early in the morning for me to be creative so I am struggling to come up with ideas besides beatings and removal of commit privileges. Any better ideas? Allan
On Sun, Dec 6, 2009 at 5:53 PM, Allan McRae <allan@archlinux.org> wrote:
Agree. What's more frustrating is that these "missing" dependency are detected by namcap: $ namcap /var/cache/pacman/pkg/mpd-0.15.6-1-i686.pkg.tar.gz mpd E: Dependency detected and not included (jack-audio-connection-kit) from files ['usr/bin/mpd'] so not only a clean chroot is not being used, but namcap is also not used as well.
2009/12/6, Allan McRae <allan@archlinux.org>:
I noticed that mpd packages have been built by Ionut (for x86_64) and Andrea (for i686) I know, and I'm sure, that Ionut always use a clean chroot (I always use a clean chroot too). Andrea, do you use a clean chroot for building your i686 packages? -- Arch Linux Developer http://www.archlinux.org http://www.archlinux.it
On 07/12/2009, Giovanni Scafora <giovanni@archlinux.org> wrote: people which ever use ours development tools. Regards -- Andrea `bash` Scarpino Arch Linux Developer
2009/12/6, Andrea Scarpino <andrea@archlinux.org>:
Don't start a flame against me cause I think I am one of the few people which ever use ours development tools.
Please Andrea, don't get me wrong, this is not a flame against you. I agree with Allan, all of us must always use ours development tools and we must building in a clean chroot. -- Arch Linux Developer http://www.archlinux.org http://www.archlinux.it
2009/12/7 Giovanni Scafora <giovanni@archlinux.org>:
-- Andrea `bash` Scarpino Arch Linux Developer
Andrea Scarpino wrote:
Don't start a flame against me cause I think I am one of the few people which ever use ours development tools.
This was not directed at you only, which is why I never named you in the original email. You just happened to be the latest example of many... Allan
On Sun, Dec 6, 2009 at 17:53, Allan McRae <allan@archlinux.org> wrote:
There is _no_ excuse not to use them.
I don't have enough hard drive space to create a build chroot. That being said, I try to use namcap and also try to resolve any bug reports within the same day it's assigned. Most of the time I don't get any issues.
Daenyth Blank wrote:
Seriously? A chroot takes ~600MB. At build time, even for the most dep heavy application, I doubt you will go much beyond 1GB plus space for the build files. Say 1.5GB for most packages. How do you build packages against [(community-)testing] or not [testing] as required? Or do you not help in the rebuilds for library soname bumps? I am very much of the opinion that if you can not build a package in a clean chroot, then you should not build the package at all. Fixing a bug quickly is no excuse for making the bug in the first place. Allan
Allan McRae wrote:
Is there some obstacle to integrating it further with makepkg? I think everyone will use it when it's as easy to use as makepkg. It doesn't seem like this should be unattainable. It would be great to be able to do a make_arch_chroots and have it make chroots for both architectures and then do a makepkg -c or something, to tell it to make in the chroot. - P
Paul Mattal wrote:
It is as simple as mkarchroot to make the chroot and makechrootpkg to build the package (providing the path to the chroot as an arguement). Making a chroot for the opposite architecture is slightly more difficult, but I can provide patches if needed. Allan
2009/12/6, Allan McRae <allan@archlinux.org>:
mkarchroot to make the chroot and makechrootpkg to build is very sample. I think that patches are not necessary... -- Arch Linux Developer http://www.archlinux.org http://www.archlinux.it
Giovanni Scafora wrote:
I agree that it's pretty simple, but not simple enough to have removed the barrier to everyone doing it. So why not make it easier and get rid of more of the mystery? Let's start at the beginning, with the recommended command for creating chroots: sudo mkarchroot -C <pacman.conf> -M <makepkg.conf> <chrootdir>/root base base-devel sudo For starters, I have to trust mkarchroot pretty well not to have any bugs, because I'm sudoing it. Then I need to find the pacman.conf and makepkg.conf. Then there's the chrootdir/root--- now do I create that first or not? How much of it? Who's going to own the stuff in there? What stuff will it put in there? Where will it keep the differencing data for the unionfs partition (so I can be sure to have enough disk around)? Then I should decide-- if I have i686 and x86_64 boxes, is it better to do all my builds on 1 machine, or on separate boxes? Can I build i686 on x86_64? Can I build x86_64 on i686? If I'm going to set all this up, I'm probably going to set it up on several machines-- how can I do that easily? I'm not saying people can't/won't/shouldn't do this. I'm saying it's not a particularly kept-simple part of a distro that prides itself on keeping things simple. People will do it if they trust it and it's simple and elegant, just like the rest of Arch. - P
2009/12/6, Paul Mattal <paul@mattal.com>:
I'm building on a x86_64 machine and I've got chroots for all the repos and for all the architectures. -- Arch Linux Developer http://www.archlinux.org http://www.archlinux.it
Paul Mattal wrote:
I'll never understand why people are more scared about their root filesystem being damaged when you can easily recover that using pacman. All our db-scripts could destroy your /home/<user> data which is much more difficult to recover from backups but I bet you never second guess them.
Then I need to find the pacman.conf and makepkg.conf.
That is optional. But most people want to keep their makepkg.conf for the PACKAGER variable and MAKEFLAGS.
Then there's the chrootdir/root--- now do I create that first or not? How much of it?
No idea... my guess is that "mkdir -p" is used, but I have always created <chrootdir>/
Who's going to own the stuff in there?
It is a system chroot, so files are owned by whoever owns the corresponding files in the packages your are installing.
Everything is in <chrootdir>. <chrootdir>/rw for the difference data and the union is created in <chrootdir>/union
Up to you... I build for x86_64 and i686 on my i686 comp (running x86_64 kernel). I know many people build for i686 on x86_64. I used to have a separate build machine for x86_64. All combinations can work.
In all honesty, I rarely ever create a new chroot but that is definitely the most difficult part of the lot. You should only need to create a chroot once. My workflow goes: update chroot: sudo mkarchroot -u /home/stable-i686/root/ build package: sudo makechrootpkg -c -r /home/stable-i686/ Done. Alias them to "mar" and "mcp" and possibly add them to the NOPASSWD section of your sudoers file and the process will become even simpler... For soname bumps it is slightly more complicated. build package with library bump and install in rw layer: sudo makechrootpkg -c -r /home/stable-i686/ -- -i then, build next package against new library from installed package: sudo makechrootpkg -r /home/stable-i686/ The creation of the chroot could be simpler, but I'm not sure we can make the building steps much simpler without removing the flexibility of the tools. Allan
On Sun, Dec 6, 2009 at 8:58 PM, Allan McRae <allan@archlinux.org> wrote:
I have an helper script to manage my many chroots (testing, non-testing, i686, x86_64). It's somewhat trivial but I could post it if someone's interested.
2009/12/6, Eric Bélanger <snowmaniscool@gmail.com>:
I'm interested, please post it! -- Arch Linux Developer http://www.archlinux.org http://www.archlinux.it
On Sun, Dec 6, 2009 at 9:36 PM, Giovanni Scafora <giovanni@archlinux.org> wrote:
Script is here: http://dev.archlinux.org/~eric/buildroot It assumes that your chroots are in /mnt and follow this naming scheme: for non-testing chroot: /mnt/arch$ARCH_$NUM for testing chroot: /mnt/arch$ARCH-testing_$NUM $ARCH is either 32 or 64 and $NUM is the number of the chroot. If your chroot don't follow that scheme, add symlinks or modify the script accordingly. Usage: there are 3 switches that basically implements what Allan's posted before. - to update a specific chroot or group of chroots : # buildroot -u current64_1 # buildroot -u testing64_1 - to build package in a specific chroot: # buildroot -b current64_1 # buildroot -b testing64_1 - to build package in a specific chroot and use custom repo in chroot (for rebuild): # buildroot -i current64_1 # buildroot -i testing64_1 You'll need to add: [repo] Server = file:///repo in your chroot's pacman.conf to be able to use the chroot repo. Notes: - the -i switch is relatively new. I only used it when updating gimp and gimp-devel. It seem to work fine but it hadn't seen much use/testing yet. - the chroot32 reference is a i686 chroot that I use to install and test i686 packages on my x86_64 system. I use Thomas' capchroot for that. - BTW, the script currently copy the built package in your cache. This is a workaround for a makepkg behavior that will be improved next time it'll be updated. Feel free to remove the cp lines. Hopefully that explains everything. Questions/comments welcome. If there's enough interest by many people, I could make the script more general and less hackish.
Allan McRae wrote:
Hmm. It seems my trouble starts in that I'm using a linode as my x86_64 box, and the linode kernel doesn't have aufs2 in it. I typically use that for all my x86_64 builds. I should have another fast x86_64 build box at my disposal soon. My laptop is still running i686, which I sometimes still need for some things. Can I switch *just* to the x86_64 kernel but leave the rest i686? Then I could use it to build both. This somehow seems like a bad idea, but maybe that's just because I'm too tired at the moment to think through all the possible bad things that could happen. - P
Paul Mattal wrote:
I run the x86_64 kernel on i686: http://allanmcrae.com/2009/06/using-an-x86_64-kernel-on-an-i686-userland/ You will also need (at least) aufs2 packages from x86_64:
It does have its issues... Allan
OK so I'm done with school and now catching up on dev stuff. Want to update some packages, but I don't want to be beaten for my good deeds =) After reading this thread I still have questions...and I'll preface this by saying I have no previous experience with chroot environments (maybe I'm the only one?): On Dec 06, 2009 at 05:58 PM, Allan McRae <allan@archlinux.org> wrote:
I assumed we are suppose to build our packages in the svn-packages directory which resides outside the chroot (as specified in the 'HOWTO Be A Packager' article). This was a bit ambiguous in the chroot article and should probably be clarified.
What happens if we don't provide the -C and -M flags when building the chroot? Are default configs provided? The wiki seems to suggest that by not providing these flags mkarchroot will instead use 'dirty' configs. The first time through the steps I grabbed the vanilla makepkg.conf and pacman.conf from the pacman source, thinking that's what the wiki meant. Of course that didn't work because makepkg.conf contains @ARCH@ instead of the actual architecture. I'm probably the only idiot who's done this, but ambiguity remains for those with no experience in this stuff.
Do we have a x86_64 build box at our dispoosal yet? If not, is anyone voluntarily building 64-bit packages? If not, what are we to do if all we have is i686? I read Allan's blog post about userland x86_64 kernels, but that doesn't sound like a solution I want to pursue.
So, if I'm understanding the advice from others, we should technically have at least two chrootdirs...one for [testing] and one for everything else. Is that right? Seeing how this is all mandatory now, we should really add another step to the HOWTO Be A Packager article, along with a link to the chroot article. I can take care of that (and other minor edits) once I have a solid understanding of the overall process.
On Dec 23, 2009 at 10:48 AM, Thayer Williams <thayerw@gmail.com> wrote:
Ignore this question, I just saw the subsequent post from Paul. I think the whole -C -M bit should be removed from the article, or at least listed as a separate (alternative) command. I'll update the wiki at some point today.
On Wed, Dec 23, 2009 at 12:48 PM, Thayer Williams <thayerw@gmail.com> wrote:
makechrootpkg is smart. You can replace "makepkg" with "makechrootpkg" wherever you may be and it will use your chroot. That's what it was made for - it handles copying files back and forth, entering the chroot, and all that other stuff. You just need a chroot for it to work.
Nope, had one for a while. I ended up donating that hardware as it was half functional and collecting dust.
Ask a kind dev to help you out? I guess it'd depend on the volume.
Yes. I keep them in /var as /var/archroot and /var/archroot-testing
Tis a wiki, good sir. It can always be edited and redited
On Dec 23, 2009 at 12:45 PM, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
Yeah no worries, I figured that by trial and error. I was just pointing out it was a bit ambiguous on the wiki and thus possibly why the chroot build method wasn't being followed sooner.
Aye alright, I just feel like a mooch (heel, slouch?) for asking for help every time I want/need to update a package.
Gotcha.
Aye, I've already updated a bit today and will continue to do so over the next week.
participants (8)
-
Aaron Griffin
-
Allan McRae
-
Andrea Scarpino
-
Daenyth Blank
-
Eric Bélanger
-
Giovanni Scafora
-
Paul Mattal
-
Thayer Williams