[arch-releng] [RFCv2] Add PXE HTTP/FTP support with curl

Gerardo Exequiel Pozzi

23 Nov 2011 23 Nov '11

midnight

In this second version also allow to download only needed *.sfs files from unpacked ISO image, instead of the whole ISO. For example running an small http daemon (darkhttp) inside live enviroment: (plus tftpd and dhcpd) [root@archiso ~]# darkhttpd /run/archiso/bootmnt/ darkhttpd/1.8, copyright (c) 2003-2011 Emil Mikulic. listening on: http://0.0.0.0:80/ Then booting from PXE in another machine with boot parameter archiso_curl_url=http://${pxeserver}/ results in: :: Running Hook [archiso_pxe_common] IP-Config: eth0 hardware address 52:54:00:12:34:57 mtu 1500 IP-Config: eth0 guessed broadcast address 192.168.0.255 IP-Config: eth0 guessed nameserver address 192.168.0.218 IP-Config: eth0 complete (from 192.168.0.218): address: 192.168.0.90 broadcast: 192.168.0.255 netmask: 255.255.255.0 gateway: 192.168.0.218 dns0 : 192.168.0.218 dns1 : 0.0.0.0 rootserver: 192.168.0.218 rootpath: filename : :: Running Hook [archiso_pxe_nbd] :: Running Hook [archiso_pxe_curl] :: Running Hook [archiso_pxe_nfs] :: Mounting /run/archiso/curlspace (tmpfs) filesystem, size='75%' :: Downloading 'http://192.168.0.218/arch/aitab' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 298 100 298 0 0 15051 0 --:--:-- --:--:-- --:--:-- 17529 :: Downloading 'http://192.168.0.218/arch/i686/root-image.fs.sfs' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 84.5M 100 84.5M 0 0 106M 0 --:--:-- --:--:-- --:--:-- 107M :: Downloading 'http://192.168.0.218/arch/i686/lib-modules.fs.sfs' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 33.7M 100 33.7M 0 0 97.9M 0 --:--:-- --:--:-- --:--:-- 98.3M :: Downloading 'http://192.168.0.218/arch/any/usr-share.fs.sfs' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 52.6M 100 52.6M 0 0 100M 0 --:--:-- --:--:-- --:--:-- 100M :: Mounting /run/archiso/cowspace (tmpfs) filesystem, size=75%...

Show replies by date

Gerardo Exequiel Pozzi

23 Nov 23 Nov

midnight

New subject: [arch-releng] [PATCH 2/3] [archiso] Add archiso_pxe_curl hook

Its add support for downloading the ISO image via HTTP/FTP with curl or download *.sfs files. The files are downloaded on /run/archiso/curlspace (tmpfs) and setup a loop dev from it (like in archiso_loop_mnt) in the ISO case, or just do a bind mount to /run/archiso/bootmnt in the other case. This hook must be located after archiso_pxe_nbd in mkinitcpio.conf, in that way NBD is the default. New bootparm: * archiso_curl_url=(first form) archiso_curl_url=http://192.168.0.7/archlinux/iso/archlinux.iso (absolute form) Use an absolute URL. Fetch the entire ISO archiso_curl_url=ftp://${pxeserver}/archlinux/iso/archlinux.iso (relative form) Use the as IP the same server where PXE reside. Fetch the entire ISO * archiso_curl_url=(second form) archiso_curl_url=http://192.168.0.7/archlinux/iso/unpacked (absolute form) Use an absolute URL. Where unpacked (an example name) is a directory where ${archisobasedir} ("arch" by default) is located with all *.sfs. This fetch only needed files. archiso_curl_url=ftp://${pxeserver}/archlinux/iso/unpacked (relative form) Use the as IP the same server where PXE reside. Where unpacked (an example name) is a directory where ${archisobasedir} ("arch" by default) is located with all *.sfs. This fetch only needed files. * curlspace_size= Set a size for a tmpfs filesystem where files are downloaded. Signed-off-by: Gerardo Exequiel Pozzi --- README | 15 ++++++++ archiso/Makefile | 4 ++ archiso/hooks/archiso_pxe_curl | 75 ++++++++++++++++++++++++++++++++++++++ archiso/install/archiso_pxe_curl | 18 +++++++++ configs/releng/build.sh | 2 +- configs/releng/mkinitcpio.conf | 2 +- 6 files changed, 114 insertions(+), 2 deletions(-) create mode 100644 archiso/hooks/archiso_pxe_curl create mode 100644 archiso/install/archiso_pxe_curl diff --git a/README b/README index 0dd2179..dede082 100644 --- a/README +++ b/README @@ -74,6 +74,7 @@ INDEX BOOTIF=<hardware-address-of-boot-interface> Default: (set via PXELINUX) + ** hooks/archiso_pxe_nbd * archiso_nbd_name= Set NBD export name used by the server. @@ -82,6 +83,18 @@ INDEX Default: "${pxeserver}" (The

Thomas Bächler

8:02 p.m.

New subject: [arch-releng] [PATCH 2/3] [archiso] Add archiso_pxe_curl hook

Am 23.11.2011 01:00, schrieb Gerardo Exequiel Pozzi:

...

Its add support for downloading the ISO image via HTTP/FTP with curl or download *.sfs files. The files are downloaded on /run/archiso/curlspace (tmpfs) and setup a loop dev from it (like in archiso_loop_mnt) in the ISO case, or just do a bind mount to /run/archiso/bootmnt in the other case.

This hook must be located after archiso_pxe_nbd in mkinitcpio.conf, in that way NBD is the default.

This is extremely useful. Can we get a resolver into initramfs, so we can use URLs instead of IPs? Can we also have an extracted version of the netinstall-dual image on releng, instead of only the ISO? We can use this in the following way: Either load gpxelinux via network boot, or boot gpxe from floppy/USB/CD/hard drive/a "kernel" image. Then, fetch syslinux configuration file and .c32 files via http. The configuration file can contain a menu that allows to choose an architecture and a mirror, allowing to boot the kernel and initrd via http or ftp, then using the archiso_pxe_curl hook to download and mount the .sfs files. Also: when can we get an image on releng that contains this hook?

Thomas Bächler

8:45 p.m.

New subject: [arch-releng] [PATCH 2/3] [archiso] Add archiso_pxe_curl hook

Am 23.11.2011 21:02, schrieb Thomas Bächler:

...

Am 23.11.2011 01:00, schrieb Gerardo Exequiel Pozzi:

...
Its add support for downloading the ISO image via HTTP/FTP with curl or download *.sfs files. The files are downloaded on /run/archiso/curlspace (tmpfs) and setup a loop dev from it (like in archiso_loop_mnt) in the ISO case, or just do a bind mount to /run/archiso/bootmnt in the other case.

This hook must be located after archiso_pxe_nbd in mkinitcpio.conf, in that way NBD is the default.

This is extremely useful. Can we get a resolver into initramfs, so we can use URLs instead of IPs? Can we also have an extracted version of the netinstall-dual image on releng, instead of only the ISO?

We can use this in the following way: Either load gpxelinux via network boot, or boot gpxe from floppy/USB/CD/hard drive/a "kernel" image. Then, fetch syslinux configuration file and .c32 files via http. The configuration file can contain a menu that allows to choose an architecture and a mirror, allowing to boot the kernel and initrd via http or ftp, then using the archiso_pxe_curl hook to download and mount the .sfs files.

Also: when can we get an image on releng that contains this hook?

This is what I have in mind (test it, it's fun). https://releng.archlinux.org/pxeboot/ https://releng.archlinux.org/pxeboot/README As I said, all we need to finalize this is 1) the extracted ISO image (aitab, .sfs) on releng (or, for a release, on all mirrors), 2) preferably a resolver in the initramfs.

Gerardo Exequiel Pozzi

24 Nov 24 Nov

3:49 a.m.

New subject: [arch-releng] [PATCH 2/3] [archiso] Add archiso_pxe_curl hook

On 11/23/2011 05:45 PM, Thomas Bächler wrote:

...

Am 23.11.2011 21:02, schrieb Thomas Bächler:

...
...
Its add support for downloading the ISO image via HTTP/FTP with curl or download *.sfs files. The files are downloaded on /run/archiso/curlspace (tmpfs) and setup a loop dev from it (like in archiso_loop_mnt) in the ISO case, or just do a bind mount to /run/archiso/bootmnt in the other case.

This hook must be located after archiso_pxe_nbd in mkinitcpio.conf, in that way NBD is the default. This is extremely useful. Can we get a resolver into initramfs, so we can use URLs instead of IPs? Can we also have an extracted version of

Am 23.11.2011 01:00, schrieb Gerardo Exequiel Pozzi: the netinstall-dual image on releng, instead of only the ISO?

We can use this in the following way: Either load gpxelinux via network boot, or boot gpxe from floppy/USB/CD/hard drive/a "kernel" image. Then, fetch syslinux configuration file and .c32 files via http. The configuration file can contain a menu that allows to choose an architecture and a mirror, allowing to boot the kernel and initrd via http or ftp, then using the archiso_pxe_curl hook to download and mount the .sfs files.

Also: when can we get an image on releng that contains this hook? This is what I have in mind (test it, it's fun).

https://releng.archlinux.org/pxeboot/ https://releng.archlinux.org/pxeboot/README I read the README, I guess you want to say HTTP server instead of TFTP server ;)

...

As I said, all we need to finalize this is 1) the extracted ISO image (aitab, .sfs) on releng (or, for a release, on all mirrors), 2) preferably a resolver in the initramfs.

-- Gerardo Exequiel Pozzi \cos^2\alpha + \sin^2\alpha = 1

Gerardo Exequiel Pozzi

4:31 a.m.

New subject: [arch-releng] [PATCH 2/3] [archiso] Add archiso_pxe_curl hook

On 11/24/2011 12:49 AM, Gerardo Exequiel Pozzi wrote:

...

On 11/23/2011 05:45 PM, Thomas Bächler wrote:

...
Am 23.11.2011 21:02, schrieb Thomas Bächler:

...
...
Its add support for downloading the ISO image via HTTP/FTP with curl or download *.sfs files. The files are downloaded on /run/archiso/curlspace (tmpfs) and setup a loop dev from it (like in archiso_loop_mnt) in the ISO case, or just do a bind mount to /run/archiso/bootmnt in the other case.

This hook must be located after archiso_pxe_nbd in mkinitcpio.conf, in that way NBD is the default. This is extremely useful. Can we get a resolver into initramfs, so we can use URLs instead of IPs? Can we also have an extracted version of

Am 23.11.2011 01:00, schrieb Gerardo Exequiel Pozzi: the netinstall-dual image on releng, instead of only the ISO?

We can use this in the following way: Either load gpxelinux via network boot, or boot gpxe from floppy/USB/CD/hard drive/a "kernel" image. Then, fetch syslinux configuration file and .c32 files via http. The configuration file can contain a menu that allows to choose an architecture and a mirror, allowing to boot the kernel and initrd via http or ftp, then using the archiso_pxe_curl hook to download and mount the .sfs files.

Also: when can we get an image on releng that contains this hook? This is what I have in mind (test it, it's fun).

https://releng.archlinux.org/pxeboot/ https://releng.archlinux.org/pxeboot/README I read the README, I guess you want to say HTTP server instead of TFTP server ;)

Ignore me. I thinking in a system with gpxe enabled :P

...

...
As I said, all we need to finalize this is 1) the extracted ISO image (aitab, .sfs) on releng (or, for a release, on all mirrors), 2) preferably a resolver in the initramfs.

-- Gerardo Exequiel Pozzi \cos^2\alpha + \sin^2\alpha = 1

Dieter Plaetinck

23 Nov 23 Nov

9:01 p.m.

New subject: [arch-releng] [PATCH 2/3] [archiso] Add archiso_pxe_curl hook

On Wed, 23 Nov 2011 21:02:55 +0100 Thomas Bächler wrote:

...

Can we also have an extracted version of the netinstall-dual image on releng, instead of only the ISO?

Sure, if archiso delivers an "extracted variant", it's fairly trivial for me to track them like I track iso's right now, as long as the naming is somewhat similar

...

Also: when can we get an image on releng that contains this hook?

The releng env is currently set to build every 4 days. And it should auto-update archiso from git IIRC Dieter

Thomas Bächler

9:54 p.m.

New subject: [arch-releng] [PATCH 2/3] [archiso] Add archiso_pxe_curl hook

Am 23.11.2011 22:01, schrieb Dieter Plaetinck:

...

On Wed, 23 Nov 2011 21:02:55 +0100 Thomas Bächler wrote:

...
Can we also have an extracted version of the netinstall-dual image on releng, instead of only the ISO?

Sure, if archiso delivers an "extracted variant", it's fairly trivial for me to track them like I track iso's right now, as long as the naming is somewhat similar

...
Also: when can we get an image on releng that contains this hook?

The releng env is currently set to build every 4 days. And it should auto-update archiso from git IIRC

Gerardo didn't push these patches to archiso.git yet. That's why I asked.

Gerardo Exequiel Pozzi

10:36 p.m.

New subject: [arch-releng] [PATCH 2/3] [archiso] Add archiso_pxe_curl hook

On 11/23/2011 06:54 PM, Thomas Bächler wrote:

...

Am 23.11.2011 22:01, schrieb Dieter Plaetinck:

...
On Wed, 23 Nov 2011 21:02:55 +0100 Thomas Bächler wrote:

...
Can we also have an extracted version of the netinstall-dual image on releng, instead of only the ISO? Sure, if archiso delivers an "extracted variant", it's fairly trivial for me to track them like I track iso's right now, as long as the naming is somewhat similar

...
Also: when can we get an image on releng that contains this hook? The releng env is currently set to build every 4 days. And it should auto-update archiso from git IIRC Gerardo didn't push these patches to archiso.git yet. That's why I asked.

These patches are waiting for next mkinitcpio + pending RFC patch to fix BASEDIR (Or I can just say "good-bye" to -b BASEDIR and use chroot mkinitcpio") Also I want to change a bit the directory structure of the ISO to be more pxelinux friendly. This is a change that I want to do long long time ago, but breaks "the nice scheme" (syslinux only files in one place, kernel/initramfs in other...). Basically now we have this inside install_dir (arch) ($archisobasedir) ├── aitab ├── any │ └── *.sfs ├── boot │ ├── i686 │ │ ├── archiso.img │ │ └── vmlinuz │ ├── memtest │ ├── memtest.COPYING │ └── syslinux │ ├── hdt │ │ ├── modalias.gz │ │ └── pciids.gz │ ├── memdisk │ ├── splash.png │ ├── syslinux.cfg │ ├── *.0 │ ├── *.com │ └── *.c32 ├── checksum.md5 └── i686 └── *.sfs And make it PXE friendly is: ├── aitab ├── any │ └── *.sfs ├── syslinux │ ├── arch │ │ └── boot │ │ ├── i686 │ │ │ ├── archiso.img │ │ │ └── vmlinuz │ │ ├── memtest │ │ └── memtest.COPYING │ ├── hdt │ │ ├── modalias.gz │ │ └── pciids.gz │ ├── pxelinux.cfg │ │ └── default │ ├── memdisk │ ├── splash.png │ ├── syslinux.cfg │ ├── *.0 │ ├── *.com │ └── *.c32 ├── checksum.md5 └── i686 └── *.sfs bah, is still nice... is there any objections? We can start a TFTP/FTP/HTTP/NFS that points directly to /run/archiso/bootmnt from the live-enviroment. Or just extract/mount the ISO in a running system ;) -- Gerardo Exequiel Pozzi \cos^2\alpha + \sin^2\alpha = 1

Thomas Bächler

10:52 p.m.

New subject: [arch-releng] [PATCH 2/3] [archiso] Add archiso_pxe_curl hook

Am 23.11.2011 23:36, schrieb Gerardo Exequiel Pozzi:

...

...
...
The releng env is currently set to build every 4 days. And it should auto-update archiso from git IIRC Gerardo didn't push these patches to archiso.git yet. That's why I asked.

These patches are waiting for next mkinitcpio + pending RFC patch to fix BASEDIR (Or I can just say "good-bye" to -b BASEDIR and use chroot mkinitcpio")

We could install mkinitcpio-git in the releng environments. Dieter?

...

Also I want to change a bit the directory structure of the ISO to be more pxelinux friendly. This is a change that I want to do long long time ago, but breaks "the nice scheme" (syslinux only files in one place, kernel/initramfs in other...).

[...]

bah, is still nice...

is there any objections?

I like the old scheme better. I like how the bootloader files are separate from the linux files.

...

We can start a TFTP/FTP/HTTP/NFS that points directly to /run/archiso/bootmnt from the live-enviroment. Or just extract/mount the ISO in a running system ;)

Why can't we do that with the current scheme?

Gerardo Exequiel Pozzi

24 Nov 24 Nov

12:30 a.m.

New subject: [arch-releng] [PATCH 2/3] [archiso] Add archiso_pxe_curl hook

On 11/23/2011 07:52 PM, Thomas Bächler wrote:

...

Am 23.11.2011 23:36, schrieb Gerardo Exequiel Pozzi:

...
...
...
The releng env is currently set to build every 4 days. And it should auto-update archiso from git IIRC Gerardo didn't push these patches to archiso.git yet. That's why I asked.

These patches are waiting for next mkinitcpio + pending RFC patch to fix BASEDIR (Or I can just say "good-bye" to -b BASEDIR and use chroot mkinitcpio") We could install mkinitcpio-git in the releng environments. Dieter?

...
Also I want to change a bit the directory structure of the ISO to be more pxelinux friendly. This is a change that I want to do long long time ago, but breaks "the nice scheme" (syslinux only files in one place, kernel/initramfs in other...).

[...]

bah, is still nice...

is there any objections? I like the old scheme better. I like how the bootloader files are separate from the linux files. Yes current scheme is nice.

...
We can start a TFTP/FTP/HTTP/NFS that points directly to /run/archiso/bootmnt from the live-enviroment. Or just extract/mount the ISO in a running system ;) Why can't we do that with the current scheme?

Because pxelinux works different compared to other bootloaders (in terms of filesystem). While syslinux/isolinux/extlinux you can see all files from the filesystem where is installed. For example extlinux installed on /boot/syslinux/ld-linux.sys from the bootloader you can access to /anyotherdir. In pxelinux things works differently, You have it installed on /tftpboot/syslinux/pxelinux.0 and you can not access to files outside "syslinux/" when you reference a file as "/pepito.txt" you are trying to access to /tftpboot/syslinux/pepito.txt (even if tftpd root is set to /tftpboot/). Yes sure still can be referenced as tftp:// (or http://) I tried with change the tftp-prefix with dhcp-option 210 (pxelinux.pathprefix) "dnsmasq --dhcp-option-force=210,/" then having this scheme (by default will take syslinux/ as prefix, the directory where pxelinux.0 is found) /tftpboot/arch/boot/i686/vmlinuz /tftpboot/syslinux/with-all-files-here /tftpboot/pxelinux.cfg/default And default file with this content (the same method used in ISO under /isolinux/isolinux.cfg) --- DEFAULT loadconfig LABEL loadconfig CONFIG /syslinux/syslinux.cfg APPEND /syslinux/ --- All things related to syslinux loads OK (c32, splash.png, etc...) but when you try to access to /arch/boot/i686/vmlinuz is not found boot: cat /arch/pepe.txt cat.c32: /arch/pepe.txt: file not found boot: cat tftp://192.168.0.77/arch/pepe.txt I Am /tftpboot/arch/pepe.txt boot: cat pepe.txt I Am /tftpboot/syslinux/pepe.txt boot: cat /pepe.txt I Am /tftpboot/syslinux/pepe.txt boot: ------------- Another alternative directory scheme is /arch/boot/{i686,x86_64}/{vmlinuz,archiso.img,...} /arch/boot/syslinux/*.{cfg,c32,...} /arch/boot/pxelinux.cfg/default <- this is a copy of syslinux.cfg (not a "loadconfig" hack) So in the case of dual-iso, we need *.cfg files here. /arch/boot/{g,p}xelinux.0 -- Gerardo Exequiel Pozzi \cos^2\alpha + \sin^2\alpha = 1

Gerardo Exequiel Pozzi

7:26 a.m.

New subject: [arch-releng] [PATCH 2/3] [archiso] Add archiso_pxe_curl hook

On 11/23/2011 09:30 PM, Gerardo Exequiel Pozzi wrote:

...

On 11/23/2011 07:52 PM, Thomas Bächler wrote:

...
Am 23.11.2011 23:36, schrieb Gerardo Exequiel Pozzi:

...
...
...
The releng env is currently set to build every 4 days. And it should auto-update archiso from git IIRC Gerardo didn't push these patches to archiso.git yet. That's why I asked.

These patches are waiting for next mkinitcpio + pending RFC patch to fix BASEDIR (Or I can just say "good-bye" to -b BASEDIR and use chroot mkinitcpio") We could install mkinitcpio-git in the releng environments. Dieter?

...
Also I want to change a bit the directory structure of the ISO to be more pxelinux friendly. This is a change that I want to do long long time ago, but breaks "the nice scheme" (syslinux only files in one place, kernel/initramfs in other...).

[...]

bah, is still nice...

is there any objections? I like the old scheme better. I like how the bootloader files are separate from the linux files. Yes current scheme is nice.

...
We can start a TFTP/FTP/HTTP/NFS that points directly to /run/archiso/bootmnt from the live-enviroment. Or just extract/mount the ISO in a running system ;) Why can't we do that with the current scheme?

Because pxelinux works different compared to other bootloaders (in terms of filesystem). While syslinux/isolinux/extlinux you can see all files from the filesystem where is installed. For example extlinux installed on /boot/syslinux/ld-linux.sys from the bootloader you can access to /anyotherdir. In pxelinux things works differently, You have it installed on /tftpboot/syslinux/pxelinux.0 and you can not access to files outside "syslinux/" when you reference a file as "/pepito.txt" you are trying to access to /tftpboot/syslinux/pepito.txt (even if tftpd root is set to /tftpboot/). Yes sure still can be referenced as tftp:// (or http://)

I tried with change the tftp-prefix with dhcp-option 210 (pxelinux.pathprefix) "dnsmasq --dhcp-option-force=210,/" then having this scheme (by default will take syslinux/ as prefix, the directory where pxelinux.0 is found)

/tftpboot/arch/boot/i686/vmlinuz /tftpboot/syslinux/with-all-files-here /tftpboot/pxelinux.cfg/default

And default file with this content (the same method used in ISO under /isolinux/isolinux.cfg)

--- DEFAULT loadconfig

LABEL loadconfig CONFIG /syslinux/syslinux.cfg APPEND /syslinux/ ---

All things related to syslinux loads OK (c32, splash.png, etc...) but when you try to access to /arch/boot/i686/vmlinuz is not found

boot: cat /arch/pepe.txt cat.c32: /arch/pepe.txt: file not found boot: cat tftp://192.168.0.77/arch/pepe.txt I Am /tftpboot/arch/pepe.txt boot: cat pepe.txt I Am /tftpboot/syslinux/pepe.txt boot: cat /pepe.txt I Am /tftpboot/syslinux/pepe.txt boot:

-------------

Another alternative directory scheme is

/arch/boot/{i686,x86_64}/{vmlinuz,archiso.img,...} /arch/boot/syslinux/*.{cfg,c32,...} /arch/boot/pxelinux.cfg/default <- this is a copy of syslinux.cfg (not a "loadconfig" hack) So in the case of dual-iso, we need *.cfg files here. /arch/boot/{g,p}xelinux.0

What do you think about keep scheme like now but... * Set the dhcp to send that pxeconfig is syslinux/syslinux.cfg (same path for all *.cfg) * Convert all relative path in syslinux.cfg to absolute paths * Enable IPAPPEND even for non-pxelinux (you only see a non-harm warning mesage if you look quickly at screen (I can not see now, maybe newer syslinux versions...)). In this way avoid two same files that differs on in this directive. (anyway we can have a nice menu for differents boot options) In that way we have unified the same directory scheme / config files for all syslinux bootloaders. # mount -o ro /tmp/out/archlinux-2011.11.24-netinstall-i686.iso /mnt/iso # dnsmasq --port=0 --enable-tftp --tftp-root=/mnt/iso --dhcp-boot=/arch/boot/syslinux/gpxelinux.0,192.168.0.77 --dhcp-range=192.168.0.2,192.168.0.254,8640 --dhcp-option-force=209,/arch/boot/syslinux/syslinux.cfg --dhcp-option-force=210,http://192.168.0.77 or using an embbeded gpxe script ------ #!gpxe dhcp net0 set 209:string /arch/boot/syslinux/syslinux.cfg set 210:string http://192.168.0.77 chain ${210:string}/arch/boot/syslinux/gpxelinux.0 boot ------- Then adding archiso_curl_url=http://192.168.0.77/ # darkhttpd /mnt/iso darkhttpd/1.8, copyright (c) 2003-2011 Emil Mikulic. listening on: http://0.0.0.0:80/ 1322117543 192.168.0.219 "GET /arch/boot/syslinux/gpxelinux.0" 200 89814 "" "gPXE/1.0.1+" 1322117545 192.168.0.219 "GET /arch/boot/syslinux/syslinux.cfg" 200 2112 "" "gPXE/1.0.0" 1322117545 192.168.0.219 "GET /arch/boot/syslinux/vesamenu.c32" 200 155894 "" "gPXE/1.0.0" 1322117545 192.168.0.219 "GET /arch/boot/syslinux/syslinux.cfg" 200 2112 "" "gPXE/1.0.0" 1322117546 192.168.0.219 "GET /arch/boot/syslinux/splash.png" 200 45614 "" "gPXE/1.0.0" 1322117580 192.168.0.219 "GET /arch/boot/i686/vmlinuz" 200 2947463 "" "gPXE/1.0.0" 1322117583 192.168.0.219 "GET /arch/boot/i686/archiso.img" 200 16698712 "" "gPXE/1.0.0" 1322117586 192.168.0.219 "GET /arch/aitab" 200 668 "" "curl/7.23.1 (i686-pc-linux-gnu) libcurl/7.23.1 OpenSSL/1.0.0e zlib/1.2.5 libssh2/1.3.0" 1322117586 192.168.0.219 "GET /arch/i686/root-image.fs.sfs" 200 88629485 "" "curl/7.23.1 (i686-pc-linux-gnu) libcurl/7.23.1 OpenSSL/1.0.0e zlib/1.2.5 libssh2/1.3.0" 1322117587 192.168.0.219 "GET /arch/i686/lib-modules.fs.sfs" 200 35356909 "" "curl/7.23.1 (i686-pc-linux-gnu) libcurl/7.23.1 OpenSSL/1.0.0e zlib/1.2.5 libssh2/1.3.0" 1322117587 192.168.0.219 "GET /arch/any/usr-share.fs.sfs" 200 55234797 "" "curl/7.23.1 (i686-pc-linux-gnu) libcurl/7.23.1 OpenSSL/1.0.0e zlib/1.2.5 libssh2/1.3.0" I need to sleep, see later! -- Gerardo Exequiel Pozzi \cos^2\alpha + \sin^2\alpha = 1

Thomas Bächler

9:38 a.m.

New subject: [arch-releng] [PATCH 2/3] [archiso] Add archiso_pxe_curl hook

Am 24.11.2011 08:26, schrieb Gerardo Exequiel Pozzi:

...

On 11/23/2011 09:30 PM, Gerardo Exequiel Pozzi wrote:

...
...
Am 23.11.2011 23:36, schrieb Gerardo Exequiel Pozzi: I tried with change the tftp-prefix with dhcp-option 210 (pxelinux.pathprefix) "dnsmasq --dhcp-option-force=210,/" then having

On 11/23/2011 07:52 PM, Thomas Bächler wrote: this scheme (by default will take syslinux/ as prefix, the directory where pxelinux.0 is found)

If you change option 210 to the /arch folder and set the filename to "boot/syslinux/pxelinux.0", everything should work fine. You also need to set option 209 to "boot/syslinux/syslinux.cfg". Then, you should be able to access everything under /arch via paths relative to /arch.

...

...
Another alternative directory scheme is

/arch/boot/{i686,x86_64}/{vmlinuz,archiso.img,...} /arch/boot/syslinux/*.{cfg,c32,...} /arch/boot/pxelinux.cfg/default <- this is a copy of syslinux.cfg (not a "loadconfig" hack) So in the case of dual-iso, we need *.cfg files here. /arch/boot/{g,p}xelinux.0

We can do the same with the old scheme, when we do as I outlined above.

...

What do you think about keep scheme like now but...

* Set the dhcp to send that pxeconfig is syslinux/syslinux.cfg (same path for all *.cfg)

"boot/syslinux/syslinux.cfg" (as outlined above)

...

* Convert all relative path in syslinux.cfg to absolute paths

This will give problems with isolinux and syslinux, because / is the root of the file system, not /arch. I'm a bit confused. Paths relative to /arch would work, except when you put it on USB and put ldlinux.sys into /arch/boot/syslinux, all paths will be considered relative to that, not to /arch. This is indeed a problem. It would be better if all paths were interpreted relative to the config file, rather than the PXE prefix / current directory.

...

* Enable IPAPPEND even for non-pxelinux (you only see a non-harm warning mesage if you look quickly at screen (I can not see now, maybe newer syslinux versions...)). In this way avoid two same files that differs on in this directive. (anyway we can have a nice menu for differents boot options)

Well, okay.

...

In that way we have unified the same directory scheme / config files for all syslinux bootloaders.

# mount -o ro /tmp/out/archlinux-2011.11.24-netinstall-i686.iso /mnt/iso

# dnsmasq --port=0 --enable-tftp --tftp-root=/mnt/iso --dhcp-boot=/arch/boot/syslinux/gpxelinux.0,192.168.0.77 --dhcp-range=192.168.0.2,192.168.0.254,8640 --dhcp-option-force=209,/arch/boot/syslinux/syslinux.cfg --dhcp-option-force=210,http://192.168.0.77

I don't like that: I want the possibility to copy /arch from any ISO to an arbitrary path on a USB drive, install syslinux, and make it boot. If you have absolute paths, this will force people to use the /arch folder everywhere. That's why absolute paths are evil.

Dieter Plaetinck

9:24 a.m.

New subject: [arch-releng] using mkinitcpio-git in releng env

On Wed, 23 Nov 2011 23:52:53 +0100 Thomas Bächler wrote:

...

We could install mkinitcpio-git in the releng environments. Dieter?

This is technically very possible, of course. OTOH that seems messy: the goal of the releng environment is to build isos based on stable/known packages. testing isos which can turn into stable isos after testing. (stable isos definitely need to be built from packages) archiso is the exception because it's only used for iso building anyway. and sometimes i enable testing packages and then disable it again, to test specific packages. but git packages are a step further (too far imho) I would like to keep a clean env to build "testing->stable" images, is using testing packages not enough? or can't you locally build isos with mkinitcpio from git? if "no and no", i would rather setup a 2nd env rather then "unstabilizing" the current one. Dieter

Thomas Bächler

9:41 a.m.

New subject: [arch-releng] using mkinitcpio-git in releng env

Am 24.11.2011 10:24, schrieb Dieter Plaetinck:

...

On Wed, 23 Nov 2011 23:52:53 +0100 Thomas Bächler wrote:

...
We could install mkinitcpio-git in the releng environments. Dieter?

This is technically very possible, of course. OTOH that seems messy: the goal of the releng environment is to build isos based on stable/known packages. testing isos which can turn into stable isos after testing. (stable isos definitely need to be built from packages) archiso is the exception because it's only used for iso building anyway. and sometimes i enable testing packages and then disable it again, to test specific packages. but git packages are a step further (too far imho)

I would like to keep a clean env to build "testing->stable" images, is using testing packages not enough? or can't you locally build isos with mkinitcpio from git? if "no and no", i would rather setup a 2nd env rather then "unstabilizing" the current one.

The packages that are installed on the ISO are standard core packages regardless. This is only about the mkinitcpio version used to build the image. If we want to test new features now, but mkinitcpio releases take too long, this is the only way. This is not about destabilizing, this is about fixing bugs that prevent us from moving the archiso development ahead. The 'mkinitcpio' package is only used for building archiso's initramfs, so there is no way to do harm, other than having a broken test ISO.

Dieter Plaetinck

10:45 a.m.

New subject: [arch-releng] using mkinitcpio-git in releng env

On Thu, 24 Nov 2011 10:41:59 +0100 Thomas Bächler wrote:

...

Am 24.11.2011 10:24, schrieb Dieter Plaetinck:

...
On Wed, 23 Nov 2011 23:52:53 +0100 Thomas Bächler wrote:

...
We could install mkinitcpio-git in the releng environments. Dieter?

This is technically very possible, of course. OTOH that seems messy: the goal of the releng environment is to build isos based on stable/known packages. testing isos which can turn into stable isos after testing. (stable isos definitely need to be built from packages) archiso is the exception because it's only used for iso building anyway. and sometimes i enable testing packages and then disable it again, to test specific packages. but git packages are a step further (too far imho)

I would like to keep a clean env to build "testing->stable" images, is using testing packages not enough? or can't you locally build isos with mkinitcpio from git? if "no and no", i would rather setup a 2nd env rather then "unstabilizing" the current one.

The packages that are installed on the ISO are standard core packages regardless. This is only about the mkinitcpio version used to build the image. If we want to test new features now, but mkinitcpio releases take too long, this is the only way. This is not about destabilizing, this is about fixing bugs that prevent us from moving the archiso development ahead.

The 'mkinitcpio' package is only used for building archiso's initramfs, so there is no way to do harm, other than having a broken test ISO.

mkinitcpio is also used to build users' initramfs and users' isos. You may say "there is no way to do harm, other than having a broken test ISO.", but I find that hard to believe, IMHO this would open the gate for subtle bugs in our isos which may not be known because we used a newer version then what's in the testing repository, for example. Having an opening for sneaky bugs is not good for isos that get promoted to official media, unless by the time we promote isos, the changes that were in git ended up in stable mkinitcpio packages, or if we can switch back and forth between mkinitcpio-git and the packaged one. personally, i think the cleanest solution is just to have 2 environments. one with only stable packages (and archiso-git), and another one with archiso-git, mkinitcpio-git and whatever else from git. Dieter

Thomas Bächler

12:06 p.m.

New subject: [arch-releng] using mkinitcpio-git in releng env

Am 24.11.2011 11:45, schrieb Dieter Plaetinck:

...

mkinitcpio is also used to build users' initramfs and users' isos.

No. Right now, the mkinitcpio version in the releng environment is only used for building the initramfs used in the ISO. The version installed into the ISO environment itself can be a different one.

Gerardo Exequiel Pozzi

12:52 p.m.

New subject: [arch-releng] using mkinitcpio-git in releng env

On 11/24/2011 09:06 AM, Thomas Bächler wrote:

...

Am 24.11.2011 11:45, schrieb Dieter Plaetinck:

...
mkinitcpio is also used to build users' initramfs and users' isos. No. Right now, the mkinitcpio version in the releng environment is only used for building the initramfs used in the ISO. The version installed into the ISO environment itself can be a different one.

mkinitcpio must match in build env and in "root-image". But now, I changed this because mkinitcpio -b is broken, so only needs mkinitcpio to be installed on "root-image". (see recent RFC patch) -- Gerardo Exequiel Pozzi \cos^2\alpha + \sin^2\alpha = 1

Dieter Plaetinck

2:54 p.m.

New subject: [arch-releng] using mkinitcpio-git in releng env

On Thu, 24 Nov 2011 09:52:03 -0300 Gerardo Exequiel Pozzi wrote:

...

On 11/24/2011 09:06 AM, Thomas Bächler wrote:

...
Am 24.11.2011 11:45, schrieb Dieter Plaetinck:

...
mkinitcpio is also used to build users' initramfs and users' isos. No. Right now, the mkinitcpio version in the releng environment is only used for building the initramfs used in the ISO. The version installed into the ISO environment itself can be a different one.

mkinitcpio must match in build env and in "root-image".

But now, I changed this because mkinitcpio -b is broken, so only needs mkinitcpio to be installed on "root-image". (see recent RFC patch)

Okay, thanks. Btw Thomas, all I'm saying is I think it makes more sense to have either: A) a "stable" build env (like we have now) + building very early prototypes (i.e. using very bleeding edge mkinitcpio stuff) on your own pc B) a "stable" build env + a new build env using mkinitcpio from git. You don't like those solutions? If not, why? Dieter

Gerardo Exequiel Pozzi

23 Nov 23 Nov

10:34 p.m.

New subject: [arch-releng] [PATCH 2/3] [archiso] Add archiso_pxe_curl hook

On 11/23/2011 05:02 PM, Thomas Bächler wrote:
> Am 23.11.2011 01:00, schrieb Gerardo Exequiel Pozzi:
>> Its add support for downloading the ISO image via HTTP/FTP with curl or download *.sfs files.
>> The files are downloaded on /run/archiso/curlspace (tmpfs) and setup a loop dev from it
>> (like in archiso_loop_mnt) in the ISO case, or just do a bind mount to
>> /run/archiso/bootmnt in the other case.
>>
>> This hook must be located after archiso_pxe_nbd in mkinitcpio.conf, in that way
>> NBD is the default.
> This is extremely useful. Can we get a resolver into initramfs, so we
> can use URLs instead of IPs? Can we also have an extracted version of
> the netinstall-dual image on releng, instead of only the ISO?
Resolver, yes sure I can add libnss_dns + needed .conf.
About releng.archlinux.org, ask to Dieter.
>
> We can use this in the following way: Either load gpxelinux via network
> boot, or boot gpxe from floppy/USB/CD/hard drive/a "kernel" image. Then,
> fetch syslinux configuration file and .c32 files via http. The
> configuration file can contain a menu that allows to choose an
> architecture and a mirror, allowing to boot the kernel and initrd via
> http or ftp, then using the archiso_pxe_curl hook to download and mount
> the .sfs files.
+ bootrom Flashing :P
>
> Also: when can we get an image on releng that contains this hook?
>


-- 
Gerardo Exequiel Pozzi
\cos^2\alpha + \sin^2\alpha = 1

Thomas Bächler

10:46 p.m.

New subject: [arch-releng] [PATCH 2/3] [archiso] Add archiso_pxe_curl hook

Am 23.11.2011 23:34, schrieb Gerardo Exequiel Pozzi:
> On 11/23/2011 05:02 PM, Thomas Bächler wrote:
>> This is extremely useful. Can we get a resolver into initramfs, so we
>> can use URLs instead of IPs? Can we also have an extracted version of
>> the netinstall-dual image on releng, instead of only the ISO?
> Resolver, yes sure I can add libnss_dns + needed .conf.

Good.

> About releng.archlinux.org, ask to Dieter.

Archiso needs to prepare a folder with the needed files (kernel,
initramfs, aitab, checksum.md5 and *.sfs, I think) in addition to the
.iso files. These files could be added as an uncompressed .tar archive.

>> We can use this in the following way: Either load gpxelinux via network
>> boot, or boot gpxe from floppy/USB/CD/hard drive/a "kernel" image. Then,
>> fetch syslinux configuration file and .c32 files via http. The
>> configuration file can contain a menu that allows to choose an
>> architecture and a mirror, allowing to boot the kernel and initrd via
>> http or ftp, then using the archiso_pxe_curl hook to download and mount
>> the .sfs files.
> + bootrom Flashing :P

A little extreme, but possible.

Gerardo Exequiel Pozzi

11:24 p.m.

New subject: [arch-releng] [PATCH 2/3] [archiso] Add archiso_pxe_curl hook

On 11/23/2011 07:46 PM, Thomas Bächler wrote:
> Am 23.11.2011 23:34, schrieb Gerardo Exequiel Pozzi:
>> On 11/23/2011 05:02 PM, Thomas Bächler wrote:
>>> This is extremely useful. Can we get a resolver into initramfs, so we
>>> can use URLs instead of IPs? Can we also have an extracted version of
>>> the netinstall-dual image on releng, instead of only the ISO?
>> Resolver, yes sure I can add libnss_dns + needed .conf.
> Good.
>
>> About releng.archlinux.org, ask to Dieter.
> Archiso needs to prepare a folder with the needed files (kernel,
> initramfs, aitab, checksum.md5 and *.sfs, I think) in addition to the
> .iso files. These files could be added as an uncompressed .tar archive.
No problem.
>
>>> We can use this in the following way: Either load gpxelinux via network
>>> boot, or boot gpxe from floppy/USB/CD/hard drive/a "kernel" image. Then,
>>> fetch syslinux configuration file and .c32 files via http. The
>>> configuration file can contain a menu that allows to choose an
>>> architecture and a mirror, allowing to boot the kernel and initrd via
>>> http or ftp, then using the archiso_pxe_curl hook to download and mount
>>> the .sfs files.
>> + bootrom Flashing :P
> A little extreme, but possible.
>
I did it some time ago in a old motherboard replacing the native pxe 
with gpxe, it boots from BKO (boot.kernel.org) (Arch Linux is here, not 
available now) and also you have the nice shell of gpxe :P


-- 
Gerardo Exequiel Pozzi
\cos^2\alpha + \sin^2\alpha = 1

Gerardo Exequiel Pozzi

26 Nov 26 Nov

12:41 a.m.

On 11/22/2011 09:00 PM, Gerardo Exequiel Pozzi wrote:

...

In this second version also allow to download only needed *.sfs files from unpacked ISO image, instead of the whole ISO.

For example running an small http daemon (darkhttp) inside live enviroment: (plus tftpd and dhcpd)

[root@archiso ~]# darkhttpd /run/archiso/bootmnt/ darkhttpd/1.8, copyright (c) 2003-2011 Emil Mikulic. listening on: http://0.0.0.0:80/

Then booting from PXE in another machine with boot parameter archiso_curl_url=http://${pxeserver}/ results in:

:: Running Hook [archiso_pxe_common] IP-Config: eth0 hardware address 52:54:00:12:34:57 mtu 1500 IP-Config: eth0 guessed broadcast address 192.168.0.255 IP-Config: eth0 guessed nameserver address 192.168.0.218 IP-Config: eth0 complete (from 192.168.0.218): address: 192.168.0.90 broadcast: 192.168.0.255 netmask: 255.255.255.0 gateway: 192.168.0.218 dns0 : 192.168.0.218 dns1 : 0.0.0.0 rootserver: 192.168.0.218 rootpath: filename : :: Running Hook [archiso_pxe_nbd] :: Running Hook [archiso_pxe_curl] :: Running Hook [archiso_pxe_nfs] :: Mounting /run/archiso/curlspace (tmpfs) filesystem, size='75%' :: Downloading 'http://192.168.0.218/arch/aitab' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 298 100 298 0 0 15051 0 --:--:-- --:--:-- --:--:-- 17529 :: Downloading 'http://192.168.0.218/arch/i686/root-image.fs.sfs' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 84.5M 100 84.5M 0 0 106M 0 --:--:-- --:--:-- --:--:-- 107M :: Downloading 'http://192.168.0.218/arch/i686/lib-modules.fs.sfs' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 33.7M 100 33.7M 0 0 97.9M 0 --:--:-- --:--:-- --:--:-- 98.3M :: Downloading 'http://192.168.0.218/arch/any/usr-share.fs.sfs' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 52.6M 100 52.6M 0 0 100M 0 --:--:-- --:--:-- --:--:-- 100M :: Mounting /run/archiso/cowspace (tmpfs) filesystem, size=75%...

TODO: I need to refactor checksum code a bit to work propertly here for only checksum downloaded *.sfs files. (current checksum is for all files inside $archisobasedir) -- Gerardo Exequiel Pozzi \cos^2\alpha + \sin^2\alpha = 1

4528

Age (days ago)

4531

Last active (days ago)

List overview

Download

22 comments

3 participants

participants (3)

Dieter Plaetinck
Gerardo Exequiel Pozzi
Thomas Bächler