[arch-dev-public] Arch Linux Docker / Vagrant: Current situation

newer
[arch-dev-public] LLVM 6.0 rebuild

Christian Rebischke

20 Jan 2018 20 Jan '18

7:19 p.m.

Hello Everybody, It's now over a half year ago that I've started working together with sangy and pierre on our vagrant and docker images. I would like to give you a short update on this topic. The Arch Linux Vagrant images are currently be build for libvirt and virtualbox. We have over 3800 downloads at the moment and slowly catching up to the community based arch linux vagrant images.[1] My first goal has been to add some hypervisors, but due to the fact that we have only libvirt and virtualbox in our repositories I have dismissed this plan. The automated build process works fine so far (except some issues with qemu[2] and the dependency on punctual iso releases on soyuz. The latter should we definitly fix. Currently the iso images are build manually. Can we automate this somehow? I really rely on punctual releases otherwise the automated build will fail and somebody needs to trigger the build again manually. Happened about 1-2 times..) The other topic is the docker image. Has anybody of you contact to pierre? He doesn't answer my mails and he is the only one with access to the repository on github. Sangy/Santiago[3] was so nice to speak with the docker guys. They said they would approve our docker image and we could move it to the other official images[4]. But for this we need to do some changes on our docker repository on github. (As long I understood sangy correct it would be just some new branches). best regards, chris / shibumi [1] https://app.vagrantup.com/archlinux/boxes/archlinux [2] https://github.com/hashicorp/packer/issues/5769 [3] https://www.archlinux.org/people/support-staff/#sangy [4] https://hub.docker.com/explore/

Attachments:

signature.asc (application/pgp-signature — 833 bytes)

Show replies by date

Jelle van der Waa

21 Jan 21 Jan

2:23 p.m.

On 01/20/18 at 08:19pm, Christian Rebischke via arch-dev-public wrote:

...

Hello Everybody, It's now over a half year ago that I've started working together with sangy and pierre on our vagrant and docker images. I would like to give you a short update on this topic.

The Arch Linux Vagrant images are currently be build for libvirt and virtualbox. We have over 3800 downloads at the moment and slowly catching up to the community based arch linux vagrant images.[1]

My first goal has been to add some hypervisors, but due to the fact that we have only libvirt and virtualbox in our repositories I have dismissed this plan.

Sounds good enough, this is also easier to manage.

...

The automated build process works fine so far (except some issues with qemu[2] and the dependency on punctual iso releases on soyuz. The latter should we definitly fix. Currently the iso images are build manually. Can we automate this somehow? I really rely on punctual releases otherwise the automated build will fail and somebody needs to trigger the build again manually. Happened about 1-2 times..)

I'm all for automatic builds, and the improvement where multiple people know how the ISO's are build & released. How would we however sign these builds? And is the bootstrap image also generated in the same manner?

...

The other topic is the docker image. Has anybody of you contact to pierre? He doesn't answer my mails and he is the only one with access to the repository on github.

I or others can grant write access to the repository, see the owners on the Github organization. I would however first want to ask pierres about it since he irregurarly is on IRC. Are these docker builds going to be automated as well btw? Oh and testing images :-)

...

Sangy/Santiago[3] was so nice to speak with the docker guys. They said they would approve our docker image and we could move it to the other official images[4]. But for this we need to do some changes on our docker repository on github. (As long I understood sangy correct it would be just some new branches).

This would be nice, how do we share the credentials to the docker login though? Or can we make multiple people owners? [1] https://github.com/orgs/archlinux/people -- Jelle van der Waa

Christian Rebischke

7:39 p.m.

On Sun, Jan 21, 2018 at 03:23:44PM +0100, Jelle van der Waa wrote:

...

I'm all for automatic builds, and the improvement where multiple people know how the ISO's are build & released. How would we however sign these builds? And is the bootstrap image also generated in the same manner? Well, we have two options:

Option 1: More people can build ISOs and more people can sign the ISO. Option 2: We automate the process and sign the ISO on our server with a new public key. Thats how nixOS or voidlinux do this. No idea about the bootstrap image. Is there a big difference between the bootstrap image and `pacstrap` in some random directory?

...

I or others can grant write access to the repository, see the owners on the Github organization. I would however first want to ask pierres about it since he irregurarly is on IRC. Are these docker builds going to be automated as well btw? Oh and testing images :-)

I would love to automate that as well, but I don't know a good process for it yet. My first thought was that we automate docker via packer as well, but unfortunately packer doesn't support docker builds. It shouldn't be that difficult to automate it with some bash magic..

...

This would be nice, how do we share the credentials to the docker login though? Or can we make multiple people owners?

Just register at dockerhub and I can add you to the Arch Linux Group. (this applies to every other developer as well. If you want access just register an account and message me)

Eli Schwartz

8:56 p.m.

On 01/21/2018 02:39 PM, Christian Rebischke via arch-dev-public wrote:

...

No idea about the bootstrap image. Is there a big difference between the bootstrap image and `pacstrap` in some random directory?

There is no difference, the bootstrap image is what you use to pacstrap. The issue is merely that getting a pacman binary compiled for some other linux distro (in case you wish to install Arch from within your currently existing Linux distribution that isn't Arch) is not necessarily easy, and in the event that it isn't, the bootstrap image can be used instead as the bare minimum environment to enter a chroot and run the Arch pacman package. Well, and it is easy enough to pacstrap from, say, Gentoo, which I know for a fact has a pacman ebuild for managing Arch installations from Gentoo: https://packages.gentoo.org/packages/sys-apps/pacman -- Eli Schwartz Bug Wrangler and Trusted User

Bartłomiej Piotrowski

6:41 p.m.

On 2018-01-20 20:19, Christian Rebischke via arch-dev-public wrote:

...

The Arch Linux Vagrant images are currently be build for libvirt and virtualbox. We have over 3800 downloads at the moment and slowly catching up to the community based arch linux vagrant images.[1]

It would be probably seen as more "official" if it was mentioned on our website.

...

My first goal has been to add some hypervisors, but due to the fact that we have only libvirt and virtualbox in our repositories I have dismissed this plan. The automated build process works fine so far (except some issues with qemu[2] and the dependency on punctual iso releases on soyuz. The latter should we definitly fix. Currently the iso images are build manually. Can we automate this somehow? I really rely on punctual releases otherwise the automated build will fail and somebody needs to trigger the build again manually. Happened about 1-2 times..)

This is something that would require a virtually offline box to be fully automated. For the time being, it sounds better to upgrade your vagrant image after first week of the month. Also, any plans to expand the scope to support OpenStack, AWS and GCE? At the moment the scope sounds rather limited.

...

Sangy/Santiago[3] was so nice to speak with the docker guys. They said they would approve our docker image and we could move it to the other official images[4]. But for this we need to do some changes on our docker repository on github. (As long I understood sangy correct it would be just some new branches).

Can you actually give more details how it's going to look like? Bartłomiej

Christian Rebischke

7:50 p.m.

On Sun, Jan 21, 2018 at 07:41:53PM +0100, Public mailing list for Arch Linux development wrote:

...

It would be probably seen as more "official" if it was mentioned on our website.

Absolutly! I will see what I can do to bring that on our website.

...

...
My first goal has been to add some hypervisors, but due to the fact that we have only libvirt and virtualbox in our repositories I have dismissed this plan. The automated build process works fine so far (except some issues with qemu[2] and the dependency on punctual iso releases on soyuz. The latter should we definitly fix. Currently the iso images are build manually. Can we automate this somehow? I really rely on punctual releases otherwise the automated build will fail and somebody needs to trigger the build again manually. Happened about 1-2 times..)

This is something that would require a virtually offline box to be fully automated. For the time being, it sounds better to upgrade your vagrant image after first week of the month.

Actually I do this already. The systemd timer for the automated builds is set on the 5th of each month (round about a week). Normally the ISO was there on 1st or 2nd each month, but in the last months its more like 6th or 9th. I could move that timer on 14th, but I would definitly prefer to have the vagrant boxes around the same date as we release our ISO images.

...

Also, any plans to expand the scope to support OpenStack, AWS and GCE? At the moment the scope sounds rather limited.

It's possible to build images for OpenStack, AWS and GCE, but as far as I understand the docs for it we would need an OpenStack cloud for OpenStack images, an AWS account for generating Amazon EC2 Images and a Google Account for generating GCE Images[1], because the packer connects to these services to generate the image.

...

Can you actually give more details how it's going to look like? Sangy worked on this. He explained it one day, but It's already some months ago. I hope he will attend our discussion here and give us a short summary about that process.

Florian Pritz

23 Jan 23 Jan

9:58 a.m.

On 21.01.2018 20:50, Christian Rebischke via arch-dev-public wrote:

...

I would definitly prefer to have the vagrant boxes around the same date as we release our ISO images.

How about letting it run every day and adding a quick check to the script that only starts the build when the iso is available and no image has been built yet? Florian

Christian Rebischke

25 Jan 25 Jan

1:02 a.m.

On Tue, Jan 23, 2018 at 10:58:48AM +0100, Public mailing list for Arch Linux development wrote:

...

On 21.01.2018 20:50, Christian Rebischke via arch-dev-public wrote:

...
I would definitly prefer to have the vagrant boxes around the same date as we release our ISO images.

How about letting it run every day and adding a quick check to the script that only starts the build when the iso is available and no image has been built yet?

Florian

Isn't this more like a workaround? I would really appreciate automated ISO image builds or at least the option that more than one person is able to generate these images. Chris

Florian Pritz

26 Jan 26 Jan

4:57 p.m.

On 25.01.2018 02:02, Christian Rebischke wrote:

...

Isn't this more like a workaround? I would really appreciate automated ISO image builds or at least the option that more than one person is able to generate these images.

Reducing the bus factor should really be a goal for us, so yes, it's a workaround. Florian

Santiago Torres-Arias

29 Jan 29 Jan

5:31 p.m.

Hi, Sorry I've been quite sick (to the point of barely having energy to look at the computer). I'm back on my feet now though :)

...

...
Sangy/Santiago[3] was so nice to speak with the docker guys. They said they would approve our docker image and we could move it to the other official images[4]. But for this we need to do some changes on our docker repository on github. (As long I understood sangy correct it would be just some new branches).

Can you actually give more details how it's going to look like?

The official images projects info is on [1] and [2] if you want to read more in-depth/updated information. I'll summarize here though: 1) A TU/Arch Linux "affiliate" submits a PR to the official images repository, which basically contains the following: 1. A tag name/image name 2. A sha256/ref of a commit/tag containig the image's information on *another* repository (in this case, our official dockerr image repo) 3. Image building instructions. 2) In parallel, we put this information on our repository. At least, a rootfs and a Dockerfile (as otherdistros do). 3) once the PR is updated, it will fetch our rootfs and Dockerfile (and other relevant info), build the docker image, and perform some quality checks on it. 4) The image is published as an "official image" on the dockerhub. The benefits from this is that industry/paranoid users often don't trust non-official images to build upon. Also, if I recall correctly, official images are periodically scanned for vulnerabilities, and (IIRC) signed with the docker-controlled signing keys, so they can be used with docker content trust[3]. I think it'd be not too difficult to schedule script the rootfs build process in the same way we do with boxes right now, publish these as tags and then update the official dockerfile repositories. Sorry for the delay. Cheers! -Santiago. [1] https://docs.docker.com/docker-hub/official_repos/ [2] https://github.com/docker-library/official-images/ [3] https://blog.docker.com/2015/08/content-trust-docker-1-8/

Jelle van der Waa

7:09 p.m.

On 01/29/18 at 12:31pm, Santiago Torres-Arias via arch-dev-public wrote:

...

Hi,

Sorry I've been quite sick (to the point of barely having energy to look at the computer). I'm back on my feet now though :)

...
...
Sangy/Santiago[3] was so nice to speak with the docker guys. They said they would approve our docker image and we could move it to the other official images[4]. But for this we need to do some changes on our docker repository on github. (As long I understood sangy correct it would be just some new branches).

Can you actually give more details how it's going to look like?

The official images projects info is on [1] and [2] if you want to read more in-depth/updated information. I'll summarize here though:

1) A TU/Arch Linux "affiliate" submits a PR to the official images repository, which basically contains the following: 1. A tag name/image name 2. A sha256/ref of a commit/tag containig the image's information on *another* repository (in this case, our official dockerr image repo) 3. Image building instructions.

A PR to this repository is also required, not sure if you mentioned it :) [1] -- Jelle van der Waa [1] https://github.com/docker-library/docs

Santiago Torres-Arias

7:20 p.m.

...

...
The official images projects info is on [1] and [2] if you want to read more in-depth/updated information. I'll summarize here though:

1) A TU/Arch Linux "affiliate" submits a PR to the official images repository, which basically contains the following: 1. A tag name/image name 2. A sha256/ref of a commit/tag containig the image's information on *another* repository (in this case, our official dockerr image repo) 3. Image building instructions.

A PR to this repository is also required, not sure if you mentioned it :) [1]

Right, I omitted that one for the sake of brevity. Thanks! -Santiago.

Pierre Schmitz

7:29 p.m.

About the ISO bus factor: * I just recently put the whole process into a simple script to make it easier for anybody else to build ISOs. Unfortunately at least the signing process requires some manual work. See https://github.com/pierres/archiso-manager About the official Docker Image: * The docker image archlinux/base is pretty young and I did some significant changes to its content lately. I am pretty happy with the result right now but some more field testing shouldn't hurt. On my virtual todo list is also figuring out whether we could reuse the created archlinux.tar for other containers as well * I did not look into the details of how we exactly need to proceed with making an "official" image. A few pull requests or some kind of setp-by-step plan (wiki or github) would help. I am glad the interest in Arch within Docker and other containers is increasing. Greetings, Pierre On Mon, Jan 29, 2018 at 8:20 PM, Santiago Torres-Arias via arch-dev-public wrote:

...

...
...
The official images projects info is on [1] and [2] if you want to read more in-depth/updated information. I'll summarize here though:

1) A TU/Arch Linux "affiliate" submits a PR to the official images repository, which basically contains the following: 1. A tag name/image name 2. A sha256/ref of a commit/tag containig the image's information on *another* repository (in this case, our official dockerr image repo) 3. Image building instructions.

A PR to this repository is also required, not sure if you mentioned it :) [1]

Right, I omitted that one for the sake of brevity.

Thanks! -Santiago.

Bartłomiej Piotrowski

9 Mar 9 Mar

8:46 p.m.

On 2018-01-29 20:29, Pierre Schmitz wrote:

...

* I did not look into the details of how we exactly need to proceed with making an "official" image. A few pull requests or some kind of setp-by-step plan (wiki or github) would help.

Necrobumping. You actually quoted the message from Santiago that describes the process. I'm going to grant him and Christian write permissions so they can start working on this on separate branch. Bartłomiej

Santiago Torres-Arias

11 Mar 11 Mar

11:01 p.m.

On Fri, Mar 09, 2018 at 09:46:48PM +0100, Bartłomiej Piotrowski via arch-dev-public wrote:

...

On 2018-01-29 20:29, Pierre Schmitz wrote:

...
* I did not look into the details of how we exactly need to proceed with making an "official" image. A few pull requests or some kind of setp-by-step plan (wiki or github) would help.

Necrobumping. You actually quoted the message from Santiago that describes the process. I'm going to grant him and Christian write permissions so they can start working on this on separate branch.

Thanks! I'll dig this up from my pile of things to do now. -Santiago.

Pierre Schmitz

12 Mar 12 Mar

4:33 a.m.

Thanks for digging this up again. You may use the github issue or project system to plan the different steps. Also (and please don't get it the wrong way) let's keep the purpose I intended for our Docker image intact. Greetings, Pierre On Mon, Mar 12, 2018 at 12:01 AM, Santiago Torres-Arias via arch-dev-public wrote:

...

On Fri, Mar 09, 2018 at 09:46:48PM +0100, Bartłomiej Piotrowski via arch-dev-public wrote:

...
On 2018-01-29 20:29, Pierre Schmitz wrote:

...
* I did not look into the details of how we exactly need to proceed with making an "official" image. A few pull requests or some kind of setp-by-step plan (wiki or github) would help.

Necrobumping. You actually quoted the message from Santiago that describes the process. I'm going to grant him and Christian write permissions so they can start working on this on separate branch.

Thanks!

I'll dig this up from my pile of things to do now.

-Santiago.

Bartłomiej Piotrowski

7:19 a.m.

On 2018-03-12 05:33, Pierre Schmitz wrote:

...

Thanks for digging this up again. You may use the github issue or project system to plan the different steps. Also (and please don't get it the wrong way) let's keep the purpose I intended for our Docker image intact.

No one has suggested changing "the purpose". I'm not sure what is unclear in Santiago's mail. Bartłomiej

Pierre Schmitz

2:41 p.m.

Yeah, that was the "and please don't get it the wrong way" part which obviously did not work. I thought I just put this out there as I already got PRs and mails from different people who wanted to make the image more minimal by removing files from packages or provide different images for all kind of uses cases. I just wanted to avoid frustration beforehand but not discourage your work. I probably failed on this one :-) Greetings, Pierre On Mon, Mar 12, 2018 at 8:19 AM, Bartłomiej Piotrowski via arch-dev-public wrote:

...

On 2018-03-12 05:33, Pierre Schmitz wrote:

...
Thanks for digging this up again. You may use the github issue or project system to plan the different steps. Also (and please don't get it the wrong way) let's keep the purpose I intended for our Docker image intact.

No one has suggested changing "the purpose". I'm not sure what is unclear in Santiago's mail.

Bartłomiej

Santiago Torres-Arias

6:06 p.m.

Understood. Although I don't exactly know what's the "original purpose" I'll try to make sure no big radical changes are made without consensus from the community :) Thanks both of you! -Santiago. On Mon, Mar 12, 2018 at 03:41:56PM +0100, Pierre Schmitz wrote:

...

Yeah, that was the "and please don't get it the wrong way" part which obviously did not work. I thought I just put this out there as I already got PRs and mails from different people who wanted to make the image more minimal by removing files from packages or provide different images for all kind of uses cases.

I just wanted to avoid frustration beforehand but not discourage your work. I probably failed on this one :-)

Greetings,

Pierre

On Mon, Mar 12, 2018 at 8:19 AM, Bartłomiej Piotrowski via arch-dev-public wrote:

...
On 2018-03-12 05:33, Pierre Schmitz wrote:

...
Thanks for digging this up again. You may use the github issue or project system to plan the different steps. Also (and please don't get it the wrong way) let's keep the purpose I intended for our Docker image intact.

No one has suggested changing "the purpose". I'm not sure what is unclear in Santiago's mail.

Bartłomiej

Pierre Schmitz

6:26 p.m.

Great. I tried to define it briefly in the README: https://github.com/archlinux/archlinux-docker#purpose I am open to any suggestions though. side note: The "release process" is described within a Makefile at https://github.com/pierres/archiso-manager On Mon, Mar 12, 2018 at 7:06 PM, Santiago Torres-Arias via arch-dev-public wrote:

...

Understood.

Although I don't exactly know what's the "original purpose" I'll try to make sure no big radical changes are made without consensus from the community :)

Thanks both of you! -Santiago.

On Mon, Mar 12, 2018 at 03:41:56PM +0100, Pierre Schmitz wrote:

...
Yeah, that was the "and please don't get it the wrong way" part which obviously did not work. I thought I just put this out there as I already got PRs and mails from different people who wanted to make the image more minimal by removing files from packages or provide different images for all kind of uses cases.

I just wanted to avoid frustration beforehand but not discourage your work. I probably failed on this one :-)

Greetings,

Pierre

On Mon, Mar 12, 2018 at 8:19 AM, Bartłomiej Piotrowski via arch-dev-public wrote:

...
On 2018-03-12 05:33, Pierre Schmitz wrote:

...
Thanks for digging this up again. You may use the github issue or project system to plan the different steps. Also (and please don't get it the wrong way) let's keep the purpose I intended for our Docker image intact.

No one has suggested changing "the purpose". I'm not sure what is unclear in Santiago's mail.

Bartłomiej

2236

Age (days ago)

2287

Last active (days ago)

List overview

Download

19 comments

7 participants

participants (7)

Bartłomiej Piotrowski
Christian Rebischke
Eli Schwartz
Florian Pritz
Jelle van der Waa
Pierre Schmitz
Santiago Torres-Arias