2009/12/6, Allan McRae :

So, we need a creative punishment for those that causes bugs by not building in a clean chroot. It is too early in the morning for me to be creative so I am struggling to come up with ideas besides beatings and removal of commit privileges. Any better ideas?

I noticed that mpd packages have been built by Ionut (for x86_64) and Andrea (for i686) I know, and I'm sure, that Ionut always use a clean chroot (I always use a clean chroot too). Andrea, do you use a clean chroot for building your i686 packages? -- Arch Linux Developer http://www.archlinux.org http://www.archlinux.it

2009/12/6, Allan McRae :

...

So, we need a creative punishment for those that causes bugs by not building in a clean chroot. It is too early in the morning for me to be creative so I am struggling to come up with ideas besides beatings and removal of commit privileges. Any better ideas?

I noticed that mpd packages have been built by Ionut (for x86_64) and Andrea (for i686) I know, and I'm sure, that Ionut always use a clean chroot (I always use a clean chroot too). Andrea, do you use a clean chroot for building your i686 packages? I ever use a clean chroot to build packages, you can trust or untrust me for this. You can see how much bugs like this was assigned to me in the last months. I think none. I updated to kernel 2.6.32 manually and without aufs2, so I built mpd in my system, but I am sure I used namcap as ever I do. I don't understand how this happened, but the only explanation is that I was distracted and I am sorry for this. Don't start a flame against me cause I think I am one of the few

On 07/12/2009, Giovanni Scafora wrote: people which ever use ours development tools. Regards -- Andrea `bash` Scarpino Arch Linux Developer

2009/12/7 Giovanni Scafora :

I agree with Allan, all of us must always use ours development tools and we must building in a clean chroot. I also am agree and I am peeved as you when I read avoidable (or stupid) bug reports.

-- Andrea `bash` Scarpino Arch Linux Developer

Allan McRae wrote:

The tools are very simple to use and are described in the wiki (http://wiki.archlinux.org/index.php/DeveloperWiki:Building_in_a_Clean_Chroot). There is _no_ excuse not to use them. The are minor changes needed for doing i686 builds on x86_64 and vise versa, but there are plenty of us doing that so help is available. Ask for help if needed.

Is there some obstacle to integrating it further with makepkg? I think everyone will use it when it's as easy to use as makepkg. It doesn't seem like this should be unattainable. It would be great to be able to do a make_arch_chroots and have it make chroots for both architectures and then do a makepkg -c or something, to tell it to make in the chroot. - P

Daenyth Blank wrote:

On Sun, Dec 6, 2009 at 17:53, Allan McRae wrote:

...

There is _no_ excuse not to use them.

I don't have enough hard drive space to create a build chroot. That being said, I try to use namcap and also try to resolve any bug reports within the same day it's assigned. Most of the time I don't get any issues.

Seriously? A chroot takes ~600MB. At build time, even for the most dep heavy application, I doubt you will go much beyond 1GB plus space for the build files. Say 1.5GB for most packages. How do you build packages against [(community-)testing] or not [testing] as required? Or do you not help in the rebuilds for library soname bumps? I am very much of the opinion that if you can not build a package in a clean chroot, then you should not build the package at all. Fixing a bug quickly is no excuse for making the bug in the first place. Allan

Seriously? A chroot takes ~600MB.  At build time, even for the most dep heavy application, I doubt you will go much beyond 1GB plus space for the build files.  Say 1.5GB for most packages. That's true, but I frequently don't have that much space. I'll see if I can play with my partition scheme and take some space from another

On Sun, Dec 6, 2009 at 19:18, Allan McRae wrote: though.

How do you build packages against [(community-)testing] or not [testing] as required?  Or do you not help in the rebuilds for library soname bumps? I don't believe any of my packages have needed any rebuilds needed since community-testing was enabled. I'll probably use the arch-games machine to be honest if I need to in the future.

I am very much of the opinion that if you can not build a package in a clean chroot, then you should not build the package at all.  Fixing a bug quickly is no excuse for making the bug in the first place. To my memory, I haven't had any bugs related to extra deps being pulled in because of not building in a chroot. I was talking about the bugs in a more general sense.

2009/12/6, Paul Mattal :

Then I should decide-- if I have i686 and x86_64 boxes, is it better to do all my builds on 1 machine, or on separate boxes? Can I build i686 on x86_64? Can I build x86_64 on i686? If I'm going to set all this up, I'm probably going to set it up on several machines-- how can I do that easily?

I'm building on a x86_64 machine and I've got chroots for all the repos and for all the architectures. -- Arch Linux Developer http://www.archlinux.org http://www.archlinux.it

On Sun, Dec 6, 2009 at 8:58 PM, Allan McRae wrote:

Paul Mattal wrote:

...

Giovanni Scafora wrote:

...

2009/12/6, Allan McRae :

...

 It is as simple as mkarchroot to make the chroot and makechrootpkg to build the package (providing the path to the chroot as an arguement).

 Making a chroot for the opposite architecture is slightly more difficult, but I can provide patches if needed.

mkarchroot to make the chroot and makechrootpkg to build is very sample. I think that patches are not necessary...

I agree that it's pretty simple, but not simple enough to have removed the barrier to everyone doing it. So why not make it easier and get rid of more of the mystery?

Let's start at the beginning, with the recommended command for creating chroots:

sudo mkarchroot -C -M <chrootdir>/root base base-devel sudo

For starters, I have to trust mkarchroot pretty well not to have any bugs, because I'm sudoing it.

I'll never understand why people are more scared about their root filesystem being damaged when you can easily recover that using pacman.  All our db-scripts could destroy your /home/<user> data which is much more difficult to recover from backups but I bet you never second guess them.

...

Then I need to find the pacman.conf and  makepkg.conf.

That is optional.  But most people want to keep their makepkg.conf for the PACKAGER variable and MAKEFLAGS.

...

Then there's the chrootdir/root--- now do I create that first or not? How much of it?

No idea...  my guess is that "mkdir -p" is used, but I have always created <chrootdir>/

...

Who's going to own the stuff in there?

It is a system chroot, so files are owned by whoever owns the corresponding files in the packages your are installing.

...

What stuff will it put in there? Where will it keep the differencing data for the unionfs partition (so I can be sure to have enough disk around)?

Everything is in <chrootdir>.  <chrootdir>/rw for the difference data and the union is created in <chrootdir>/union

...

Then I should decide-- if I have i686 and x86_64 boxes, is it better to do all my builds on 1 machine, or on separate boxes? Can I build i686 on x86_64? Can I build x86_64 on i686? If I'm going to set all this up, I'm probably going to set it up on several machines-- how can I do that easily?

Up to you...   I build for x86_64 and i686 on my i686 comp (running x86_64 kernel).  I know many people build for i686 on x86_64.  I used to have a separate build machine for x86_64.  All combinations can work.

...

I'm not saying people can't/won't/shouldn't do this. I'm saying it's not a particularly kept-simple part of a distro that prides itself on keeping things simple. People will do it if they trust it and it's simple and elegant, just like the rest of Arch.

In all honesty, I rarely ever create a new chroot but that is definitely the most difficult part of the lot. You should only need to create a chroot once.  My workflow goes:

update chroot: sudo mkarchroot -u /home/stable-i686/root/

build package: sudo makechrootpkg -c -r /home/stable-i686/

Done.  Alias them to "mar" and "mcp" and possibly add them to the NOPASSWD section of your sudoers file and the process will become even simpler...

For soname bumps it is slightly more complicated. build package with library bump and install in rw layer: sudo makechrootpkg -c -r /home/stable-i686/ -- -i

then, build next package against new library from installed package: sudo makechrootpkg -r /home/stable-i686/

The creation of the chroot could be simpler, but I'm not sure we can make the building steps much simpler without removing the flexibility of the tools.

Allan

I have an helper script to manage my many chroots (testing, non-testing, i686, x86_64). It's somewhat trivial but I could post it if someone's interested.

Allan McRae wrote:

Paul Mattal wrote:

...

Giovanni Scafora wrote:

...

2009/12/6, Allan McRae :

...

It is as simple as mkarchroot to make the chroot and makechrootpkg to build the package (providing the path to the chroot as an arguement).

Making a chroot for the opposite architecture is slightly more difficult, but I can provide patches if needed.

mkarchroot to make the chroot and makechrootpkg to build is very sample. I think that patches are not necessary...

I agree that it's pretty simple, but not simple enough to have removed the barrier to everyone doing it. So why not make it easier and get rid of more of the mystery?

Let's start at the beginning, with the recommended command for creating chroots:

sudo mkarchroot -C -M <chrootdir>/root base base-devel sudo

For starters, I have to trust mkarchroot pretty well not to have any bugs, because I'm sudoing it.

I'll never understand why people are more scared about their root filesystem being damaged when you can easily recover that using pacman. All our db-scripts could destroy your /home/<user> data which is much more difficult to recover from backups but I bet you never second guess them.

...

Then I need to find the pacman.conf and makepkg.conf.

That is optional. But most people want to keep their makepkg.conf for the PACKAGER variable and MAKEFLAGS.

...

Then there's the chrootdir/root--- now do I create that first or not? How much of it?

No idea... my guess is that "mkdir -p" is used, but I have always created <chrootdir>/

...

Who's going to own the stuff in there?

It is a system chroot, so files are owned by whoever owns the corresponding files in the packages your are installing.

...

What stuff will it put in there? Where will it keep the differencing data for the unionfs partition (so I can be sure to have enough disk around)?

Everything is in <chrootdir>. <chrootdir>/rw for the difference data and the union is created in <chrootdir>/union

...

Then I should decide-- if I have i686 and x86_64 boxes, is it better to do all my builds on 1 machine, or on separate boxes? Can I build i686 on x86_64? Can I build x86_64 on i686? If I'm going to set all this up, I'm probably going to set it up on several machines-- how can I do that easily?

Up to you... I build for x86_64 and i686 on my i686 comp (running x86_64 kernel). I know many people build for i686 on x86_64. I used to have a separate build machine for x86_64. All combinations can work.

...

I'm not saying people can't/won't/shouldn't do this. I'm saying it's not a particularly kept-simple part of a distro that prides itself on keeping things simple. People will do it if they trust it and it's simple and elegant, just like the rest of Arch.

In all honesty, I rarely ever create a new chroot but that is definitely the most difficult part of the lot. You should only need to create a chroot once. My workflow goes:

update chroot: sudo mkarchroot -u /home/stable-i686/root/

build package: sudo makechrootpkg -c -r /home/stable-i686/

Done. Alias them to "mar" and "mcp" and possibly add them to the NOPASSWD section of your sudoers file and the process will become even simpler...

For soname bumps it is slightly more complicated. build package with library bump and install in rw layer: sudo makechrootpkg -c -r /home/stable-i686/ -- -i

then, build next package against new library from installed package: sudo makechrootpkg -r /home/stable-i686/

The creation of the chroot could be simpler, but I'm not sure we can make the building steps much simpler without removing the flexibility of the tools.

Hmm. It seems my trouble starts in that I'm using a linode as my x86_64 box, and the linode kernel doesn't have aufs2 in it. I typically use that for all my x86_64 builds. I should have another fast x86_64 build box at my disposal soon. My laptop is still running i686, which I sometimes still need for some things. Can I switch *just* to the x86_64 kernel but leave the rest i686? Then I could use it to build both. This somehow seems like a bad idea, but maybe that's just because I'm too tired at the moment to think through all the possible bad things that could happen. - P

On Sun, Dec 6, 2009 at 9:36 PM, Giovanni Scafora wrote:

2009/12/6, Eric Bélanger :

...

I have an helper script to manage my many chroots (testing,  non-testing, i686, x86_64). It's somewhat trivial but I could post it  if someone's interested.

I'm interested, please post it!

Script is here: http://dev.archlinux.org/~eric/buildroot It assumes that your chroots are in /mnt and follow this naming scheme: for non-testing chroot: /mnt/arch$ARCH_$NUM for testing chroot: /mnt/arch$ARCH-testing_$NUM $ARCH is either 32 or 64 and $NUM is the number of the chroot. If your chroot don't follow that scheme, add symlinks or modify the script accordingly. Usage: there are 3 switches that basically implements what Allan's posted before. - to update a specific chroot or group of chroots : # buildroot -u current64_1 # buildroot -u testing64_1 - to build package in a specific chroot: # buildroot -b current64_1 # buildroot -b testing64_1 - to build package in a specific chroot and use custom repo in chroot (for rebuild): # buildroot -i current64_1 # buildroot -i testing64_1 You'll need to add: [repo] Server = file:///repo in your chroot's pacman.conf to be able to use the chroot repo. Notes: - the -i switch is relatively new. I only used it when updating gimp and gimp-devel. It seem to work fine but it hadn't seen much use/testing yet. - the chroot32 reference is a i686 chroot that I use to install and test i686 packages on my x86_64 system. I use Thomas' capchroot for that. - BTW, the script currently copy the built package in your cache. This is a workaround for a makepkg behavior that will be improved next time it'll be updated. Feel free to remove the cp lines. Hopefully that explains everything. Questions/comments welcome. If there's enough interest by many people, I could make the script more general and less hackish.

On Dec 23, 2009 at 10:48 AM, Thayer Williams wrote:

On Dec 06, 2009 at 05:58 PM, Allan McRae wrote:

...

Paul Mattal wrote:

...

Then I need to find the pacman.conf and makepkg.conf.

That is optional. But most people want to keep their makepkg.conf for the PACKAGER variable and MAKEFLAGS.

What happens if we don't provide the -C and -M flags when building the chroot? Are default configs provided? The wiki seems to suggest that by not providing these flags mkarchroot will instead use 'dirty' configs.

The first time through the steps I grabbed the vanilla makepkg.conf and pacman.conf from the pacman source, thinking that's what the wiki meant. Of course that didn't work because makepkg.conf contains @ARCH@ instead of the actual architecture.

I'm probably the only idiot who's done this, but ambiguity remains for those with no experience in this stuff.

Ignore this question, I just saw the subsequent post from Paul. I think the whole -C -M bit should be removed from the article, or at least listed as a separate (alternative) command. I'll update the wiki at some point today.

On Dec 23, 2009 at 12:45 PM, Aaron Griffin wrote:

On Wed, Dec 23, 2009 at 12:48 PM, Thayer Williams wrote:

...

OK so I'm done with school and now catching up on dev stuff.  Want to update some packages, but I don't want to be beaten for my good deeds =)

After reading this thread I still have questions...and I'll preface this by saying I have no previous experience with chroot environments (maybe I'm the only one?):

On Dec 06, 2009 at 05:58 PM, Allan McRae wrote:

...

Paul Mattal wrote:

...

I agree that it's pretty simple, but not simple enough to have removed the barrier to everyone doing it. So why not make it easier and get rid of more of the mystery?

I assumed we are suppose to build our packages in the svn-packages directory which resides outside the chroot (as specified in the 'HOWTO Be A Packager' article).  This was a bit ambiguous in the chroot article and should probably be clarified.

makechrootpkg is smart. You can replace "makepkg" with "makechrootpkg" wherever you may be and it will use your chroot. That's what it was made for - it handles copying files back and forth, entering the chroot, and all that other stuff. You just need a chroot for it to work.

Yeah no worries, I figured that by trial and error. I was just pointing out it was a bit ambiguous on the wiki and thus possibly why the chroot build method wasn't being followed sooner.

...

...

...

Then I should decide-- if I have i686 and x86_64 boxes, is it better to do all my builds on 1 machine, or on separate boxes? Can I build i686 on x86_64? Can I build x86_64 on i686? If I'm going to set all this up, I'm probably going to set it up on several machines-- how can I do that easily?

Up to you...   I build for x86_64 and i686 on my i686 comp (running x86_64 kernel).  I know many people build for i686 on x86_64.  I used to have a separate build machine for x86_64.  All combinations can work.

Do we have a x86_64 build box at our dispoosal yet?

Nope, had one for a while. I ended up donating that hardware as it was half functional and collecting dust.

...

If not, is anyone voluntarily building 64-bit packages?

If not, what are we to do if all we have is i686?  I read Allan's blog post about userland x86_64 kernels, but that doesn't sound like a solution I want to pursue.

Ask a kind dev to help you out? I guess it'd depend on the volume.

Aye alright, I just feel like a mooch (heel, slouch?) for asking for help every time I want/need to update a package.

...

...

...

I'm not saying people can't/won't/shouldn't do this. I'm saying it's not a particularly kept-simple part of a distro that prides itself on keeping things simple. People will do it if they trust it and it's simple and elegant, just like the rest of Arch.

In all honesty, I rarely ever create a new chroot but that is definitely the most difficult part of the lot. You should only need to create a chroot once...

So, if I'm understanding the advice from others, we should technically have at least two chrootdirs...one for [testing] and one for everything else.  Is that right?

Yes. I keep them in /var as /var/archroot and /var/archroot-testing

Gotcha.

...

Seeing how this is all mandatory now, we should really add another step to the HOWTO Be A Packager article, along with a link to the chroot article.  I can take care of that (and other minor edits) once I have a solid understanding of the overall process.

Tis a wiki, good sir. It can always be edited and redited

Aye, I've already updated a bit today and will continue to do so over the next week.