[arch-releng] [RFCv2] Add PXE HTTP/FTP support with curl
In this second version also allow to download only needed *.sfs files from unpacked ISO image, instead of the whole ISO. For example running an small http daemon (darkhttp) inside live enviroment: (plus tftpd and dhcpd) [root@archiso ~]# darkhttpd /run/archiso/bootmnt/ darkhttpd/1.8, copyright (c) 2003-2011 Emil Mikulic. listening on: http://0.0.0.0:80/ Then booting from PXE in another machine with boot parameter archiso_curl_url=http://${pxeserver}/ results in: :: Running Hook [archiso_pxe_common] IP-Config: eth0 hardware address 52:54:00:12:34:57 mtu 1500 IP-Config: eth0 guessed broadcast address 192.168.0.255 IP-Config: eth0 guessed nameserver address 192.168.0.218 IP-Config: eth0 complete (from 192.168.0.218): address: 192.168.0.90 broadcast: 192.168.0.255 netmask: 255.255.255.0 gateway: 192.168.0.218 dns0 : 192.168.0.218 dns1 : 0.0.0.0 rootserver: 192.168.0.218 rootpath: filename : :: Running Hook [archiso_pxe_nbd] :: Running Hook [archiso_pxe_curl] :: Running Hook [archiso_pxe_nfs] :: Mounting /run/archiso/curlspace (tmpfs) filesystem, size='75%' :: Downloading 'http://192.168.0.218/arch/aitab' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 298 100 298 0 0 15051 0 --:--:-- --:--:-- --:--:-- 17529 :: Downloading 'http://192.168.0.218/arch/i686/root-image.fs.sfs' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 84.5M 100 84.5M 0 0 106M 0 --:--:-- --:--:-- --:--:-- 107M :: Downloading 'http://192.168.0.218/arch/i686/lib-modules.fs.sfs' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 33.7M 100 33.7M 0 0 97.9M 0 --:--:-- --:--:-- --:--:-- 98.3M :: Downloading 'http://192.168.0.218/arch/any/usr-share.fs.sfs' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 52.6M 100 52.6M 0 0 100M 0 --:--:-- --:--:-- --:--:-- 100M :: Mounting /run/archiso/cowspace (tmpfs) filesystem, size=75%...
Its add support for downloading the ISO image via HTTP/FTP with curl or download *.sfs files. The files are downloaded on /run/archiso/curlspace (tmpfs) and setup a loop dev from it (like in archiso_loop_mnt) in the ISO case, or just do a bind mount to /run/archiso/bootmnt in the other case. This hook must be located after archiso_pxe_nbd in mkinitcpio.conf, in that way NBD is the default. New bootparm: * archiso_curl_url=(first form) archiso_curl_url=http://192.168.0.7/archlinux/iso/archlinux.iso (absolute form) Use an absolute URL. Fetch the entire ISO archiso_curl_url=ftp://${pxeserver}/archlinux/iso/archlinux.iso (relative form) Use the as IP the same server where PXE reside. Fetch the entire ISO * archiso_curl_url=(second form) archiso_curl_url=http://192.168.0.7/archlinux/iso/unpacked (absolute form) Use an absolute URL. Where unpacked (an example name) is a directory where ${archisobasedir} ("arch" by default) is located with all *.sfs. This fetch only needed files. archiso_curl_url=ftp://${pxeserver}/archlinux/iso/unpacked (relative form) Use the as IP the same server where PXE reside. Where unpacked (an example name) is a directory where ${archisobasedir} ("arch" by default) is located with all *.sfs. This fetch only needed files. * curlspace_size= Set a size for a tmpfs filesystem where files are downloaded. Signed-off-by: Gerardo Exequiel Pozzi <vmlinuz386@yahoo.com.ar> --- README | 15 ++++++++ archiso/Makefile | 4 ++ archiso/hooks/archiso_pxe_curl | 75 ++++++++++++++++++++++++++++++++++++++ archiso/install/archiso_pxe_curl | 18 +++++++++ configs/releng/build.sh | 2 +- configs/releng/mkinitcpio.conf | 2 +- 6 files changed, 114 insertions(+), 2 deletions(-) create mode 100644 archiso/hooks/archiso_pxe_curl create mode 100644 archiso/install/archiso_pxe_curl diff --git a/README b/README index 0dd2179..dede082 100644 --- a/README +++ b/README @@ -74,6 +74,7 @@ INDEX BOOTIF=<hardware-address-of-boot-interface> Default: (set via PXELINUX) + ** hooks/archiso_pxe_nbd * archiso_nbd_name= Set NBD export name used by the server. @@ -82,6 +83,18 @@ INDEX Default: "${pxeserver}" (The <boot-server-ip from ip=) +** hooks/archiso_pxe_curl + +* archiso_curl_url= Set an HTTP/FTP URL of an ISO file (first form), + or a path (must end with /) where ${archisobasedir} + is found with all *.sfs files (second form). + In the IP part if ${pxeserver} is used, PXE IP will be used. + Default: (unset) +* curlspace_size= Set the size of tmpfs of "curlspace" where the specified + ISO image or *.sfs files are downloaded. + Default: "75%" + + ** hooks/archiso_loop_mnt * img_label= Set the filesystem label where archiso-image.iso. @@ -127,6 +140,8 @@ if nothing is specified on command line. + mkinitcpio-nfs-utils for ipconfig * archiso_pxe_nbd + nbd for nbd-client +* archiso_pxe_curl + + curl for curl * archiso_shutdown + (none) diff --git a/archiso/Makefile b/archiso/Makefile index 115a15b..0dbbbf8 100644 --- a/archiso/Makefile +++ b/archiso/Makefile @@ -20,6 +20,8 @@ install-hooks: install -D -m 644 install/archiso_pxe_common $(DESTDIR)/lib/initcpio/install/archiso_pxe_common install -D -m 644 hooks/archiso_pxe_nbd $(DESTDIR)/lib/initcpio/hooks/archiso_pxe_nbd install -D -m 644 install/archiso_pxe_nbd $(DESTDIR)/lib/initcpio/install/archiso_pxe_nbd + install -D -m 644 hooks/archiso_pxe_curl $(DESTDIR)/lib/initcpio/hooks/archiso_pxe_curl + install -D -m 644 install/archiso_pxe_curl $(DESTDIR)/lib/initcpio/install/archiso_pxe_curl install -D -m 644 hooks/archiso_loop_mnt $(DESTDIR)/lib/initcpio/hooks/archiso_loop_mnt install -D -m 644 install/archiso_loop_mnt $(DESTDIR)/lib/initcpio/install/archiso_loop_mnt @@ -45,6 +47,8 @@ uninstall: rm -f $(DESTDIR)/lib/initcpio/install/archiso_pxe_common rm -f $(DESTDIR)/lib/initcpio/hooks/archiso_pxe_nbd rm -f $(DESTDIR)/lib/initcpio/install/archiso_pxe_nbd + rm -f $(DESTDIR)/lib/initcpio/hooks/archiso_pxe_curl + rm -f $(DESTDIR)/lib/initcpio/install/archiso_pxe_curl rm -f $(DESTDIR)/lib/initcpio/hooks/archiso_loop_mnt rm -f $(DESTDIR)/lib/initcpio/install/archiso_loop_mnt rm -rf $(DESTDIR)/usr/share/archiso/ diff --git a/archiso/hooks/archiso_pxe_curl b/archiso/hooks/archiso_pxe_curl new file mode 100644 index 0000000..bf05983 --- /dev/null +++ b/archiso/hooks/archiso_pxe_curl @@ -0,0 +1,75 @@ +# vim: set ft=sh: + +run_hook() { + if [[ -n "${ip}" && -n "${archiso_curl_url}" ]]; then + archiso_curl_url=$(eval echo ${archiso_curl_url}) + [[ -z "${curlspace_size}" ]] && curlspace_size="75%" + mount_handler="archiso_pxe_curl_mount_handler" + fi +} + +# Fetch a file with CURL +# +# $1 URL +# $2 Destination directory inside curlspace +_curl_get() { + local _url="${1}" + local _dst="${2}" + + msg ":: Downloading '${_url}'" + if ! curl -f -o "/run/archiso/curlspace/${_dst}/${_url##*/}" --create-dirs "${_url}"; then + echo "ERROR: Downloading '${_url}'" + echo " Falling back to interactive prompt" + echo " You can try to fix the problem manually, log out when you are finished" + launch_interactive_shell + fi +} + +archiso_pxe_curl_mount_handler () { + newroot="${1}" + + msg ":: Mounting /run/archiso/curlspace (tmpfs) filesystem, size='${curlspace_size}'" + mkdir -p "/run/archiso/curlspace" + mount -t tmpfs -o size="${curlspace_size}",mode=0755 curlspace "/run/archiso/curlspace" + + # Check if URL ends in / + if [[ "${archiso_curl_url}" != "${archiso_curl_url%/}" ]]; then + + local _aitab_url="${archiso_curl_url}${aitab#/run/archiso/bootmnt/}" + local _aitab_file="/run/archiso/curlspace/${aitab#/run/archiso/bootmnt/}" + + _curl_get "${_aitab_url}" "${archisobasedir}" + + local aitab_img aitab_mnt aitab_arch aitab_sfs_comp aitab_fs_type aitab_fs_size + while read aitab_img aitab_mnt aitab_arch aitab_sfs_comp aitab_fs_type aitab_fs_size; do + [[ "${aitab_img#\#}" != "${aitab_img}" ]] && continue + [[ "${aitab_arch}" != "any" && "${aitab_arch}" != "${arch}" ]] && continue + if [[ "${aitab_fs_type}" != "none" ]]; then + if [[ "${aitab_sfs_comp}" != "none" ]]; then + _curl_get "${archiso_curl_url}${archisobasedir}/${aitab_arch}/${aitab_img}.fs.sfs" "${archisobasedir}/${aitab_arch}" + else + _curl_get "${archiso_curl_url}/${archisobasedir}/${aitab_arch}/${aitab_img}.fs" "${archisobasedir}/${aitab_arch}" + fi + else + _curl_get "${archiso_curl_url}/${archisobasedir}/${aitab_arch}/${aitab_img}.sfs" "${archisobasedir}/${aitab_arch}" + fi + done < "${_aitab_file}" + + mkdir -p "/run/archiso/bootmnt" + mount -o bind /run/archiso/curlspace /run/archiso/bootmnt + else + local _dev_loop + + _curl_get "${archiso_curl_url}" "/" + + _dev_loop=$(losetup -f) + if ! losetup "${_dev_loop}" "/run/archiso/curlspace/${archiso_curl_url##*/}"; then + echo "ERROR: Setting loopback device '${_dev_loop}'" + echo " for file '/run/archiso/curlspace/${archiso_curl_url##*/}'" + echo " Falling back to interactive prompt" + echo " You can try to fix the problem manually, log out when you are finished" + launch_interactive_shell + fi + fi + archiso_mount_handler ${newroot} +} diff --git a/archiso/install/archiso_pxe_curl b/archiso/install/archiso_pxe_curl new file mode 100644 index 0000000..62fc4b8 --- /dev/null +++ b/archiso/install/archiso_pxe_curl @@ -0,0 +1,18 @@ +# vim: set ft=sh: + +build () +{ + MODULES="" + BINARIES="" + FILES="" + SCRIPT="archiso_pxe_curl" + + add_binary "/usr/bin/curl" "/usr/bin/curl" +} + +help () +{ +cat<<HELPEOF + This hook loads the necessary modules for boot via PXE and curl. +HELPEOF +} diff --git a/configs/releng/build.sh b/configs/releng/build.sh index f6e0070..101a56f 100755 --- a/configs/releng/build.sh +++ b/configs/releng/build.sh @@ -43,7 +43,7 @@ make_customize_root_image() { make_setup_mkinitcpio() { if [[ ! -e ${work_dir}/build.${FUNCNAME} ]]; then local _hook - for _hook in archiso archiso_shutdown archiso_pxe_common archiso_pxe_nbd archiso_loop_mnt; do + for _hook in archiso archiso_shutdown archiso_pxe_common archiso_pxe_nbd archiso_pxe_curl archiso_loop_mnt; do cp /lib/initcpio/hooks/${_hook} ${work_dir}/root-image/lib/initcpio/hooks cp /lib/initcpio/install/${_hook} ${work_dir}/root-image/lib/initcpio/install done diff --git a/configs/releng/mkinitcpio.conf b/configs/releng/mkinitcpio.conf index eb0a68a..6387982 100644 --- a/configs/releng/mkinitcpio.conf +++ b/configs/releng/mkinitcpio.conf @@ -1,2 +1,2 @@ -HOOKS="base udev memdisk archiso_shutdown archiso archiso_loop_mnt archiso_pxe_common archiso_pxe_nbd pata scsi sata virtio usb fw pcmcia filesystems usbinput" +HOOKS="base udev memdisk archiso_shutdown archiso archiso_loop_mnt archiso_pxe_common archiso_pxe_nbd archiso_pxe_curl pata scsi sata virtio usb fw pcmcia filesystems usbinput" COMPRESSION="xz" -- 1.7.7.4
Am 23.11.2011 01:00, schrieb Gerardo Exequiel Pozzi:
Its add support for downloading the ISO image via HTTP/FTP with curl or download *.sfs files. The files are downloaded on /run/archiso/curlspace (tmpfs) and setup a loop dev from it (like in archiso_loop_mnt) in the ISO case, or just do a bind mount to /run/archiso/bootmnt in the other case.
This hook must be located after archiso_pxe_nbd in mkinitcpio.conf, in that way NBD is the default.
This is extremely useful. Can we get a resolver into initramfs, so we can use URLs instead of IPs? Can we also have an extracted version of the netinstall-dual image on releng, instead of only the ISO? We can use this in the following way: Either load gpxelinux via network boot, or boot gpxe from floppy/USB/CD/hard drive/a "kernel" image. Then, fetch syslinux configuration file and .c32 files via http. The configuration file can contain a menu that allows to choose an architecture and a mirror, allowing to boot the kernel and initrd via http or ftp, then using the archiso_pxe_curl hook to download and mount the .sfs files. Also: when can we get an image on releng that contains this hook?
Am 23.11.2011 21:02, schrieb Thomas Bächler:
Am 23.11.2011 01:00, schrieb Gerardo Exequiel Pozzi:
Its add support for downloading the ISO image via HTTP/FTP with curl or download *.sfs files. The files are downloaded on /run/archiso/curlspace (tmpfs) and setup a loop dev from it (like in archiso_loop_mnt) in the ISO case, or just do a bind mount to /run/archiso/bootmnt in the other case.
This hook must be located after archiso_pxe_nbd in mkinitcpio.conf, in that way NBD is the default.
This is extremely useful. Can we get a resolver into initramfs, so we can use URLs instead of IPs? Can we also have an extracted version of the netinstall-dual image on releng, instead of only the ISO?
We can use this in the following way: Either load gpxelinux via network boot, or boot gpxe from floppy/USB/CD/hard drive/a "kernel" image. Then, fetch syslinux configuration file and .c32 files via http. The configuration file can contain a menu that allows to choose an architecture and a mirror, allowing to boot the kernel and initrd via http or ftp, then using the archiso_pxe_curl hook to download and mount the .sfs files.
Also: when can we get an image on releng that contains this hook?
This is what I have in mind (test it, it's fun). https://releng.archlinux.org/pxeboot/ https://releng.archlinux.org/pxeboot/README As I said, all we need to finalize this is 1) the extracted ISO image (aitab, .sfs) on releng (or, for a release, on all mirrors), 2) preferably a resolver in the initramfs.
On 11/23/2011 05:45 PM, Thomas Bächler wrote:
Am 23.11.2011 21:02, schrieb Thomas Bächler:
Its add support for downloading the ISO image via HTTP/FTP with curl or download *.sfs files. The files are downloaded on /run/archiso/curlspace (tmpfs) and setup a loop dev from it (like in archiso_loop_mnt) in the ISO case, or just do a bind mount to /run/archiso/bootmnt in the other case.
This hook must be located after archiso_pxe_nbd in mkinitcpio.conf, in that way NBD is the default. This is extremely useful. Can we get a resolver into initramfs, so we can use URLs instead of IPs? Can we also have an extracted version of
Am 23.11.2011 01:00, schrieb Gerardo Exequiel Pozzi: the netinstall-dual image on releng, instead of only the ISO?
We can use this in the following way: Either load gpxelinux via network boot, or boot gpxe from floppy/USB/CD/hard drive/a "kernel" image. Then, fetch syslinux configuration file and .c32 files via http. The configuration file can contain a menu that allows to choose an architecture and a mirror, allowing to boot the kernel and initrd via http or ftp, then using the archiso_pxe_curl hook to download and mount the .sfs files.
Also: when can we get an image on releng that contains this hook? This is what I have in mind (test it, it's fun).
https://releng.archlinux.org/pxeboot/ https://releng.archlinux.org/pxeboot/README I read the README, I guess you want to say HTTP server instead of TFTP server ;)
As I said, all we need to finalize this is 1) the extracted ISO image (aitab, .sfs) on releng (or, for a release, on all mirrors), 2) preferably a resolver in the initramfs.
-- Gerardo Exequiel Pozzi \cos^2\alpha + \sin^2\alpha = 1
On 11/24/2011 12:49 AM, Gerardo Exequiel Pozzi wrote:
On 11/23/2011 05:45 PM, Thomas Bächler wrote:
Am 23.11.2011 21:02, schrieb Thomas Bächler:
Its add support for downloading the ISO image via HTTP/FTP with curl or download *.sfs files. The files are downloaded on /run/archiso/curlspace (tmpfs) and setup a loop dev from it (like in archiso_loop_mnt) in the ISO case, or just do a bind mount to /run/archiso/bootmnt in the other case.
This hook must be located after archiso_pxe_nbd in mkinitcpio.conf, in that way NBD is the default. This is extremely useful. Can we get a resolver into initramfs, so we can use URLs instead of IPs? Can we also have an extracted version of
Am 23.11.2011 01:00, schrieb Gerardo Exequiel Pozzi: the netinstall-dual image on releng, instead of only the ISO?
We can use this in the following way: Either load gpxelinux via network boot, or boot gpxe from floppy/USB/CD/hard drive/a "kernel" image. Then, fetch syslinux configuration file and .c32 files via http. The configuration file can contain a menu that allows to choose an architecture and a mirror, allowing to boot the kernel and initrd via http or ftp, then using the archiso_pxe_curl hook to download and mount the .sfs files.
Also: when can we get an image on releng that contains this hook? This is what I have in mind (test it, it's fun).
https://releng.archlinux.org/pxeboot/ https://releng.archlinux.org/pxeboot/README I read the README, I guess you want to say HTTP server instead of TFTP server ;)
Ignore me. I thinking in a system with gpxe enabled :P
As I said, all we need to finalize this is 1) the extracted ISO image (aitab, .sfs) on releng (or, for a release, on all mirrors), 2) preferably a resolver in the initramfs.
-- Gerardo Exequiel Pozzi \cos^2\alpha + \sin^2\alpha = 1
On Wed, 23 Nov 2011 21:02:55 +0100 Thomas Bächler <thomas@archlinux.org> wrote:
Can we also have an extracted version of the netinstall-dual image on releng, instead of only the ISO?
Sure, if archiso delivers an "extracted variant", it's fairly trivial for me to track them like I track iso's right now, as long as the naming is somewhat similar
Also: when can we get an image on releng that contains this hook?
The releng env is currently set to build every 4 days. And it should auto-update archiso from git IIRC Dieter
Am 23.11.2011 22:01, schrieb Dieter Plaetinck:
On Wed, 23 Nov 2011 21:02:55 +0100 Thomas Bächler <thomas@archlinux.org> wrote:
Can we also have an extracted version of the netinstall-dual image on releng, instead of only the ISO?
Sure, if archiso delivers an "extracted variant", it's fairly trivial for me to track them like I track iso's right now, as long as the naming is somewhat similar
Also: when can we get an image on releng that contains this hook?
The releng env is currently set to build every 4 days. And it should auto-update archiso from git IIRC
Gerardo didn't push these patches to archiso.git yet. That's why I asked.
On 11/23/2011 06:54 PM, Thomas Bächler wrote:
Am 23.11.2011 22:01, schrieb Dieter Plaetinck:
On Wed, 23 Nov 2011 21:02:55 +0100 Thomas Bächler<thomas@archlinux.org> wrote:
Can we also have an extracted version of the netinstall-dual image on releng, instead of only the ISO? Sure, if archiso delivers an "extracted variant", it's fairly trivial for me to track them like I track iso's right now, as long as the naming is somewhat similar
Also: when can we get an image on releng that contains this hook? The releng env is currently set to build every 4 days. And it should auto-update archiso from git IIRC Gerardo didn't push these patches to archiso.git yet. That's why I asked.
These patches are waiting for next mkinitcpio + pending RFC patch to fix BASEDIR (Or I can just say "good-bye" to -b BASEDIR and use chroot mkinitcpio") Also I want to change a bit the directory structure of the ISO to be more pxelinux friendly. This is a change that I want to do long long time ago, but breaks "the nice scheme" (syslinux only files in one place, kernel/initramfs in other...). Basically now we have this inside install_dir (arch) ($archisobasedir) ├── aitab ├── any │ └── *.sfs ├── boot │ ├── i686 │ │ ├── archiso.img │ │ └── vmlinuz │ ├── memtest │ ├── memtest.COPYING │ └── syslinux │ ├── hdt │ │ ├── modalias.gz │ │ └── pciids.gz │ ├── memdisk │ ├── splash.png │ ├── syslinux.cfg │ ├── *.0 │ ├── *.com │ └── *.c32 ├── checksum.md5 └── i686 └── *.sfs And make it PXE friendly is: ├── aitab ├── any │ └── *.sfs ├── syslinux │ ├── arch │ │ └── boot │ │ ├── i686 │ │ │ ├── archiso.img │ │ │ └── vmlinuz │ │ ├── memtest │ │ └── memtest.COPYING │ ├── hdt │ │ ├── modalias.gz │ │ └── pciids.gz │ ├── pxelinux.cfg │ │ └── default │ ├── memdisk │ ├── splash.png │ ├── syslinux.cfg │ ├── *.0 │ ├── *.com │ └── *.c32 ├── checksum.md5 └── i686 └── *.sfs bah, is still nice... is there any objections? We can start a TFTP/FTP/HTTP/NFS that points directly to /run/archiso/bootmnt from the live-enviroment. Or just extract/mount the ISO in a running system ;) -- Gerardo Exequiel Pozzi \cos^2\alpha + \sin^2\alpha = 1
Am 23.11.2011 23:36, schrieb Gerardo Exequiel Pozzi:
The releng env is currently set to build every 4 days. And it should auto-update archiso from git IIRC Gerardo didn't push these patches to archiso.git yet. That's why I asked.
These patches are waiting for next mkinitcpio + pending RFC patch to fix BASEDIR (Or I can just say "good-bye" to -b BASEDIR and use chroot mkinitcpio")
We could install mkinitcpio-git in the releng environments. Dieter?
Also I want to change a bit the directory structure of the ISO to be more pxelinux friendly. This is a change that I want to do long long time ago, but breaks "the nice scheme" (syslinux only files in one place, kernel/initramfs in other...).
[...]
bah, is still nice...
is there any objections?
I like the old scheme better. I like how the bootloader files are separate from the linux files.
We can start a TFTP/FTP/HTTP/NFS that points directly to /run/archiso/bootmnt from the live-enviroment. Or just extract/mount the ISO in a running system ;)
Why can't we do that with the current scheme?
On 11/23/2011 07:52 PM, Thomas Bächler wrote:
Am 23.11.2011 23:36, schrieb Gerardo Exequiel Pozzi:
The releng env is currently set to build every 4 days. And it should auto-update archiso from git IIRC Gerardo didn't push these patches to archiso.git yet. That's why I asked.
These patches are waiting for next mkinitcpio + pending RFC patch to fix BASEDIR (Or I can just say "good-bye" to -b BASEDIR and use chroot mkinitcpio") We could install mkinitcpio-git in the releng environments. Dieter?
Also I want to change a bit the directory structure of the ISO to be more pxelinux friendly. This is a change that I want to do long long time ago, but breaks "the nice scheme" (syslinux only files in one place, kernel/initramfs in other...).
[...]
bah, is still nice...
is there any objections? I like the old scheme better. I like how the bootloader files are separate from the linux files. Yes current scheme is nice.
We can start a TFTP/FTP/HTTP/NFS that points directly to /run/archiso/bootmnt from the live-enviroment. Or just extract/mount the ISO in a running system ;) Why can't we do that with the current scheme?
Because pxelinux works different compared to other bootloaders (in terms of filesystem). While syslinux/isolinux/extlinux you can see all files from the filesystem where is installed. For example extlinux installed on /boot/syslinux/ld-linux.sys from the bootloader you can access to /anyotherdir. In pxelinux things works differently, You have it installed on /tftpboot/syslinux/pxelinux.0 and you can not access to files outside "syslinux/" when you reference a file as "/pepito.txt" you are trying to access to /tftpboot/syslinux/pepito.txt (even if tftpd root is set to /tftpboot/). Yes sure still can be referenced as tftp:// (or http://) I tried with change the tftp-prefix with dhcp-option 210 (pxelinux.pathprefix) "dnsmasq --dhcp-option-force=210,/" then having this scheme (by default will take syslinux/ as prefix, the directory where pxelinux.0 is found) /tftpboot/arch/boot/i686/vmlinuz /tftpboot/syslinux/with-all-files-here /tftpboot/pxelinux.cfg/default And default file with this content (the same method used in ISO under /isolinux/isolinux.cfg) --- DEFAULT loadconfig LABEL loadconfig CONFIG /syslinux/syslinux.cfg APPEND /syslinux/ --- All things related to syslinux loads OK (c32, splash.png, etc...) but when you try to access to /arch/boot/i686/vmlinuz is not found boot: cat /arch/pepe.txt cat.c32: /arch/pepe.txt: file not found boot: cat tftp://192.168.0.77/arch/pepe.txt I Am /tftpboot/arch/pepe.txt boot: cat pepe.txt I Am /tftpboot/syslinux/pepe.txt boot: cat /pepe.txt I Am /tftpboot/syslinux/pepe.txt boot: ------------- Another alternative directory scheme is /arch/boot/{i686,x86_64}/{vmlinuz,archiso.img,...} /arch/boot/syslinux/*.{cfg,c32,...} /arch/boot/pxelinux.cfg/default <- this is a copy of syslinux.cfg (not a "loadconfig" hack) So in the case of dual-iso, we need *.cfg files here. /arch/boot/{g,p}xelinux.0 -- Gerardo Exequiel Pozzi \cos^2\alpha + \sin^2\alpha = 1
On 11/23/2011 09:30 PM, Gerardo Exequiel Pozzi wrote:
On 11/23/2011 07:52 PM, Thomas Bächler wrote:
Am 23.11.2011 23:36, schrieb Gerardo Exequiel Pozzi:
The releng env is currently set to build every 4 days. And it should auto-update archiso from git IIRC Gerardo didn't push these patches to archiso.git yet. That's why I asked.
These patches are waiting for next mkinitcpio + pending RFC patch to fix BASEDIR (Or I can just say "good-bye" to -b BASEDIR and use chroot mkinitcpio") We could install mkinitcpio-git in the releng environments. Dieter?
Also I want to change a bit the directory structure of the ISO to be more pxelinux friendly. This is a change that I want to do long long time ago, but breaks "the nice scheme" (syslinux only files in one place, kernel/initramfs in other...).
[...]
bah, is still nice...
is there any objections? I like the old scheme better. I like how the bootloader files are separate from the linux files. Yes current scheme is nice.
We can start a TFTP/FTP/HTTP/NFS that points directly to /run/archiso/bootmnt from the live-enviroment. Or just extract/mount the ISO in a running system ;) Why can't we do that with the current scheme?
Because pxelinux works different compared to other bootloaders (in terms of filesystem). While syslinux/isolinux/extlinux you can see all files from the filesystem where is installed. For example extlinux installed on /boot/syslinux/ld-linux.sys from the bootloader you can access to /anyotherdir. In pxelinux things works differently, You have it installed on /tftpboot/syslinux/pxelinux.0 and you can not access to files outside "syslinux/" when you reference a file as "/pepito.txt" you are trying to access to /tftpboot/syslinux/pepito.txt (even if tftpd root is set to /tftpboot/). Yes sure still can be referenced as tftp:// (or http://)
I tried with change the tftp-prefix with dhcp-option 210 (pxelinux.pathprefix) "dnsmasq --dhcp-option-force=210,/" then having this scheme (by default will take syslinux/ as prefix, the directory where pxelinux.0 is found)
/tftpboot/arch/boot/i686/vmlinuz /tftpboot/syslinux/with-all-files-here /tftpboot/pxelinux.cfg/default
And default file with this content (the same method used in ISO under /isolinux/isolinux.cfg)
--- DEFAULT loadconfig
LABEL loadconfig CONFIG /syslinux/syslinux.cfg APPEND /syslinux/ ---
All things related to syslinux loads OK (c32, splash.png, etc...) but when you try to access to /arch/boot/i686/vmlinuz is not found
boot: cat /arch/pepe.txt cat.c32: /arch/pepe.txt: file not found boot: cat tftp://192.168.0.77/arch/pepe.txt I Am /tftpboot/arch/pepe.txt boot: cat pepe.txt I Am /tftpboot/syslinux/pepe.txt boot: cat /pepe.txt I Am /tftpboot/syslinux/pepe.txt boot:
-------------
Another alternative directory scheme is
/arch/boot/{i686,x86_64}/{vmlinuz,archiso.img,...} /arch/boot/syslinux/*.{cfg,c32,...} /arch/boot/pxelinux.cfg/default <- this is a copy of syslinux.cfg (not a "loadconfig" hack) So in the case of dual-iso, we need *.cfg files here. /arch/boot/{g,p}xelinux.0
What do you think about keep scheme like now but... * Set the dhcp to send that pxeconfig is syslinux/syslinux.cfg (same path for all *.cfg) * Convert all relative path in syslinux.cfg to absolute paths * Enable IPAPPEND even for non-pxelinux (you only see a non-harm warning mesage if you look quickly at screen (I can not see now, maybe newer syslinux versions...)). In this way avoid two same files that differs on in this directive. (anyway we can have a nice menu for differents boot options) In that way we have unified the same directory scheme / config files for all syslinux bootloaders. # mount -o ro /tmp/out/archlinux-2011.11.24-netinstall-i686.iso /mnt/iso # dnsmasq --port=0 --enable-tftp --tftp-root=/mnt/iso --dhcp-boot=/arch/boot/syslinux/gpxelinux.0,192.168.0.77 --dhcp-range=192.168.0.2,192.168.0.254,8640 --dhcp-option-force=209,/arch/boot/syslinux/syslinux.cfg --dhcp-option-force=210,http://192.168.0.77 or using an embbeded gpxe script ------ #!gpxe dhcp net0 set 209:string /arch/boot/syslinux/syslinux.cfg set 210:string http://192.168.0.77 chain ${210:string}/arch/boot/syslinux/gpxelinux.0 boot ------- Then adding archiso_curl_url=http://192.168.0.77/ # darkhttpd /mnt/iso darkhttpd/1.8, copyright (c) 2003-2011 Emil Mikulic. listening on: http://0.0.0.0:80/ 1322117543 192.168.0.219 "GET /arch/boot/syslinux/gpxelinux.0" 200 89814 "" "gPXE/1.0.1+" 1322117545 192.168.0.219 "GET /arch/boot/syslinux/syslinux.cfg" 200 2112 "" "gPXE/1.0.0" 1322117545 192.168.0.219 "GET /arch/boot/syslinux/vesamenu.c32" 200 155894 "" "gPXE/1.0.0" 1322117545 192.168.0.219 "GET /arch/boot/syslinux/syslinux.cfg" 200 2112 "" "gPXE/1.0.0" 1322117546 192.168.0.219 "GET /arch/boot/syslinux/splash.png" 200 45614 "" "gPXE/1.0.0" 1322117580 192.168.0.219 "GET /arch/boot/i686/vmlinuz" 200 2947463 "" "gPXE/1.0.0" 1322117583 192.168.0.219 "GET /arch/boot/i686/archiso.img" 200 16698712 "" "gPXE/1.0.0" 1322117586 192.168.0.219 "GET /arch/aitab" 200 668 "" "curl/7.23.1 (i686-pc-linux-gnu) libcurl/7.23.1 OpenSSL/1.0.0e zlib/1.2.5 libssh2/1.3.0" 1322117586 192.168.0.219 "GET /arch/i686/root-image.fs.sfs" 200 88629485 "" "curl/7.23.1 (i686-pc-linux-gnu) libcurl/7.23.1 OpenSSL/1.0.0e zlib/1.2.5 libssh2/1.3.0" 1322117587 192.168.0.219 "GET /arch/i686/lib-modules.fs.sfs" 200 35356909 "" "curl/7.23.1 (i686-pc-linux-gnu) libcurl/7.23.1 OpenSSL/1.0.0e zlib/1.2.5 libssh2/1.3.0" 1322117587 192.168.0.219 "GET /arch/any/usr-share.fs.sfs" 200 55234797 "" "curl/7.23.1 (i686-pc-linux-gnu) libcurl/7.23.1 OpenSSL/1.0.0e zlib/1.2.5 libssh2/1.3.0" I need to sleep, see later! -- Gerardo Exequiel Pozzi \cos^2\alpha + \sin^2\alpha = 1
Am 24.11.2011 08:26, schrieb Gerardo Exequiel Pozzi:
On 11/23/2011 09:30 PM, Gerardo Exequiel Pozzi wrote:
Am 23.11.2011 23:36, schrieb Gerardo Exequiel Pozzi: I tried with change the tftp-prefix with dhcp-option 210 (pxelinux.pathprefix) "dnsmasq --dhcp-option-force=210,/" then having
On 11/23/2011 07:52 PM, Thomas Bächler wrote: this scheme (by default will take syslinux/ as prefix, the directory where pxelinux.0 is found)
If you change option 210 to the /arch folder and set the filename to "boot/syslinux/pxelinux.0", everything should work fine. You also need to set option 209 to "boot/syslinux/syslinux.cfg". Then, you should be able to access everything under /arch via paths relative to /arch.
Another alternative directory scheme is
/arch/boot/{i686,x86_64}/{vmlinuz,archiso.img,...} /arch/boot/syslinux/*.{cfg,c32,...} /arch/boot/pxelinux.cfg/default <- this is a copy of syslinux.cfg (not a "loadconfig" hack) So in the case of dual-iso, we need *.cfg files here. /arch/boot/{g,p}xelinux.0
We can do the same with the old scheme, when we do as I outlined above.
What do you think about keep scheme like now but...
* Set the dhcp to send that pxeconfig is syslinux/syslinux.cfg (same path for all *.cfg)
"boot/syslinux/syslinux.cfg" (as outlined above)
* Convert all relative path in syslinux.cfg to absolute paths
This will give problems with isolinux and syslinux, because / is the root of the file system, not /arch. I'm a bit confused. Paths relative to /arch would work, except when you put it on USB and put ldlinux.sys into /arch/boot/syslinux, all paths will be considered relative to that, not to /arch. This is indeed a problem. It would be better if all paths were interpreted relative to the config file, rather than the PXE prefix / current directory.
* Enable IPAPPEND even for non-pxelinux (you only see a non-harm warning mesage if you look quickly at screen (I can not see now, maybe newer syslinux versions...)). In this way avoid two same files that differs on in this directive. (anyway we can have a nice menu for differents boot options)
Well, okay.
In that way we have unified the same directory scheme / config files for all syslinux bootloaders.
# mount -o ro /tmp/out/archlinux-2011.11.24-netinstall-i686.iso /mnt/iso
# dnsmasq --port=0 --enable-tftp --tftp-root=/mnt/iso --dhcp-boot=/arch/boot/syslinux/gpxelinux.0,192.168.0.77 --dhcp-range=192.168.0.2,192.168.0.254,8640 --dhcp-option-force=209,/arch/boot/syslinux/syslinux.cfg --dhcp-option-force=210,http://192.168.0.77
I don't like that: I want the possibility to copy /arch from any ISO to an arbitrary path on a USB drive, install syslinux, and make it boot. If you have absolute paths, this will force people to use the /arch folder everywhere. That's why absolute paths are evil.
On Wed, 23 Nov 2011 23:52:53 +0100 Thomas Bächler <thomas@archlinux.org> wrote:
We could install mkinitcpio-git in the releng environments. Dieter?
This is technically very possible, of course. OTOH that seems messy: the goal of the releng environment is to build isos based on stable/known packages. testing isos which can turn into stable isos after testing. (stable isos definitely need to be built from packages) archiso is the exception because it's only used for iso building anyway. and sometimes i enable testing packages and then disable it again, to test specific packages. but git packages are a step further (too far imho) I would like to keep a clean env to build "testing->stable" images, is using testing packages not enough? or can't you locally build isos with mkinitcpio from git? if "no and no", i would rather setup a 2nd env rather then "unstabilizing" the current one. Dieter
Am 24.11.2011 10:24, schrieb Dieter Plaetinck:
On Wed, 23 Nov 2011 23:52:53 +0100 Thomas Bächler <thomas@archlinux.org> wrote:
We could install mkinitcpio-git in the releng environments. Dieter?
This is technically very possible, of course. OTOH that seems messy: the goal of the releng environment is to build isos based on stable/known packages. testing isos which can turn into stable isos after testing. (stable isos definitely need to be built from packages) archiso is the exception because it's only used for iso building anyway. and sometimes i enable testing packages and then disable it again, to test specific packages. but git packages are a step further (too far imho)
I would like to keep a clean env to build "testing->stable" images, is using testing packages not enough? or can't you locally build isos with mkinitcpio from git? if "no and no", i would rather setup a 2nd env rather then "unstabilizing" the current one.
The packages that are installed on the ISO are standard core packages regardless. This is only about the mkinitcpio version used to build the image. If we want to test new features now, but mkinitcpio releases take too long, this is the only way. This is not about destabilizing, this is about fixing bugs that prevent us from moving the archiso development ahead. The 'mkinitcpio' package is only used for building archiso's initramfs, so there is no way to do harm, other than having a broken test ISO.
On Thu, 24 Nov 2011 10:41:59 +0100 Thomas Bächler <thomas@archlinux.org> wrote:
Am 24.11.2011 10:24, schrieb Dieter Plaetinck:
On Wed, 23 Nov 2011 23:52:53 +0100 Thomas Bächler <thomas@archlinux.org> wrote:
We could install mkinitcpio-git in the releng environments. Dieter?
This is technically very possible, of course. OTOH that seems messy: the goal of the releng environment is to build isos based on stable/known packages. testing isos which can turn into stable isos after testing. (stable isos definitely need to be built from packages) archiso is the exception because it's only used for iso building anyway. and sometimes i enable testing packages and then disable it again, to test specific packages. but git packages are a step further (too far imho)
I would like to keep a clean env to build "testing->stable" images, is using testing packages not enough? or can't you locally build isos with mkinitcpio from git? if "no and no", i would rather setup a 2nd env rather then "unstabilizing" the current one.
The packages that are installed on the ISO are standard core packages regardless. This is only about the mkinitcpio version used to build the image. If we want to test new features now, but mkinitcpio releases take too long, this is the only way. This is not about destabilizing, this is about fixing bugs that prevent us from moving the archiso development ahead.
The 'mkinitcpio' package is only used for building archiso's initramfs, so there is no way to do harm, other than having a broken test ISO.
mkinitcpio is also used to build users' initramfs and users' isos. You may say "there is no way to do harm, other than having a broken test ISO.", but I find that hard to believe, IMHO this would open the gate for subtle bugs in our isos which may not be known because we used a newer version then what's in the testing repository, for example. Having an opening for sneaky bugs is not good for isos that get promoted to official media, unless by the time we promote isos, the changes that were in git ended up in stable mkinitcpio packages, or if we can switch back and forth between mkinitcpio-git and the packaged one. personally, i think the cleanest solution is just to have 2 environments. one with only stable packages (and archiso-git), and another one with archiso-git, mkinitcpio-git and whatever else from git. Dieter
Am 24.11.2011 11:45, schrieb Dieter Plaetinck:
mkinitcpio is also used to build users' initramfs and users' isos.
No. Right now, the mkinitcpio version in the releng environment is only used for building the initramfs used in the ISO. The version installed into the ISO environment itself can be a different one.
On 11/24/2011 09:06 AM, Thomas Bächler wrote:
Am 24.11.2011 11:45, schrieb Dieter Plaetinck:
mkinitcpio is also used to build users' initramfs and users' isos. No. Right now, the mkinitcpio version in the releng environment is only used for building the initramfs used in the ISO. The version installed into the ISO environment itself can be a different one.
mkinitcpio must match in build env and in "root-image". But now, I changed this because mkinitcpio -b is broken, so only needs mkinitcpio to be installed on "root-image". (see recent RFC patch) -- Gerardo Exequiel Pozzi \cos^2\alpha + \sin^2\alpha = 1
On Thu, 24 Nov 2011 09:52:03 -0300 Gerardo Exequiel Pozzi <vmlinuz386@yahoo.com.ar> wrote:
On 11/24/2011 09:06 AM, Thomas Bächler wrote:
Am 24.11.2011 11:45, schrieb Dieter Plaetinck:
mkinitcpio is also used to build users' initramfs and users' isos. No. Right now, the mkinitcpio version in the releng environment is only used for building the initramfs used in the ISO. The version installed into the ISO environment itself can be a different one.
mkinitcpio must match in build env and in "root-image".
But now, I changed this because mkinitcpio -b is broken, so only needs mkinitcpio to be installed on "root-image". (see recent RFC patch)
Okay, thanks. Btw Thomas, all I'm saying is I think it makes more sense to have either: A) a "stable" build env (like we have now) + building very early prototypes (i.e. using very bleeding edge mkinitcpio stuff) on your own pc B) a "stable" build env + a new build env using mkinitcpio from git. You don't like those solutions? If not, why? Dieter
On 11/23/2011 05:02 PM, Thomas Bächler wrote: > Am 23.11.2011 01:00, schrieb Gerardo Exequiel Pozzi: >> Its add support for downloading the ISO image via HTTP/FTP with curl or download *.sfs files. >> The files are downloaded on /run/archiso/curlspace (tmpfs) and setup a loop dev from it >> (like in archiso_loop_mnt) in the ISO case, or just do a bind mount to >> /run/archiso/bootmnt in the other case. >> >> This hook must be located after archiso_pxe_nbd in mkinitcpio.conf, in that way >> NBD is the default. > This is extremely useful. Can we get a resolver into initramfs, so we > can use URLs instead of IPs? Can we also have an extracted version of > the netinstall-dual image on releng, instead of only the ISO? Resolver, yes sure I can add libnss_dns + needed .conf. About releng.archlinux.org, ask to Dieter. > > We can use this in the following way: Either load gpxelinux via network > boot, or boot gpxe from floppy/USB/CD/hard drive/a "kernel" image. Then, > fetch syslinux configuration file and .c32 files via http. The > configuration file can contain a menu that allows to choose an > architecture and a mirror, allowing to boot the kernel and initrd via > http or ftp, then using the archiso_pxe_curl hook to download and mount > the .sfs files. + bootrom Flashing :P > > Also: when can we get an image on releng that contains this hook? > -- Gerardo Exequiel Pozzi \cos^2\alpha + \sin^2\alpha = 1
Am 23.11.2011 23:34, schrieb Gerardo Exequiel Pozzi: > On 11/23/2011 05:02 PM, Thomas Bächler wrote: >> This is extremely useful. Can we get a resolver into initramfs, so we >> can use URLs instead of IPs? Can we also have an extracted version of >> the netinstall-dual image on releng, instead of only the ISO? > Resolver, yes sure I can add libnss_dns + needed .conf. Good. > About releng.archlinux.org, ask to Dieter. Archiso needs to prepare a folder with the needed files (kernel, initramfs, aitab, checksum.md5 and *.sfs, I think) in addition to the .iso files. These files could be added as an uncompressed .tar archive. >> We can use this in the following way: Either load gpxelinux via network >> boot, or boot gpxe from floppy/USB/CD/hard drive/a "kernel" image. Then, >> fetch syslinux configuration file and .c32 files via http. The >> configuration file can contain a menu that allows to choose an >> architecture and a mirror, allowing to boot the kernel and initrd via >> http or ftp, then using the archiso_pxe_curl hook to download and mount >> the .sfs files. > + bootrom Flashing :P A little extreme, but possible.
On 11/23/2011 07:46 PM, Thomas Bächler wrote: > Am 23.11.2011 23:34, schrieb Gerardo Exequiel Pozzi: >> On 11/23/2011 05:02 PM, Thomas Bächler wrote: >>> This is extremely useful. Can we get a resolver into initramfs, so we >>> can use URLs instead of IPs? Can we also have an extracted version of >>> the netinstall-dual image on releng, instead of only the ISO? >> Resolver, yes sure I can add libnss_dns + needed .conf. > Good. > >> About releng.archlinux.org, ask to Dieter. > Archiso needs to prepare a folder with the needed files (kernel, > initramfs, aitab, checksum.md5 and *.sfs, I think) in addition to the > .iso files. These files could be added as an uncompressed .tar archive. No problem. > >>> We can use this in the following way: Either load gpxelinux via network >>> boot, or boot gpxe from floppy/USB/CD/hard drive/a "kernel" image. Then, >>> fetch syslinux configuration file and .c32 files via http. The >>> configuration file can contain a menu that allows to choose an >>> architecture and a mirror, allowing to boot the kernel and initrd via >>> http or ftp, then using the archiso_pxe_curl hook to download and mount >>> the .sfs files. >> + bootrom Flashing :P > A little extreme, but possible. > I did it some time ago in a old motherboard replacing the native pxe with gpxe, it boots from BKO (boot.kernel.org) (Arch Linux is here, not available now) and also you have the nice shell of gpxe :P -- Gerardo Exequiel Pozzi \cos^2\alpha + \sin^2\alpha = 1
On 11/22/2011 09:00 PM, Gerardo Exequiel Pozzi wrote:
In this second version also allow to download only needed *.sfs files from unpacked ISO image, instead of the whole ISO.
For example running an small http daemon (darkhttp) inside live enviroment: (plus tftpd and dhcpd)
[root@archiso ~]# darkhttpd /run/archiso/bootmnt/ darkhttpd/1.8, copyright (c) 2003-2011 Emil Mikulic. listening on: http://0.0.0.0:80/
Then booting from PXE in another machine with boot parameter archiso_curl_url=http://${pxeserver}/ results in:
:: Running Hook [archiso_pxe_common] IP-Config: eth0 hardware address 52:54:00:12:34:57 mtu 1500 IP-Config: eth0 guessed broadcast address 192.168.0.255 IP-Config: eth0 guessed nameserver address 192.168.0.218 IP-Config: eth0 complete (from 192.168.0.218): address: 192.168.0.90 broadcast: 192.168.0.255 netmask: 255.255.255.0 gateway: 192.168.0.218 dns0 : 192.168.0.218 dns1 : 0.0.0.0 rootserver: 192.168.0.218 rootpath: filename : :: Running Hook [archiso_pxe_nbd] :: Running Hook [archiso_pxe_curl] :: Running Hook [archiso_pxe_nfs] :: Mounting /run/archiso/curlspace (tmpfs) filesystem, size='75%' :: Downloading 'http://192.168.0.218/arch/aitab' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 298 100 298 0 0 15051 0 --:--:-- --:--:-- --:--:-- 17529 :: Downloading 'http://192.168.0.218/arch/i686/root-image.fs.sfs' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 84.5M 100 84.5M 0 0 106M 0 --:--:-- --:--:-- --:--:-- 107M :: Downloading 'http://192.168.0.218/arch/i686/lib-modules.fs.sfs' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 33.7M 100 33.7M 0 0 97.9M 0 --:--:-- --:--:-- --:--:-- 98.3M :: Downloading 'http://192.168.0.218/arch/any/usr-share.fs.sfs' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 52.6M 100 52.6M 0 0 100M 0 --:--:-- --:--:-- --:--:-- 100M :: Mounting /run/archiso/cowspace (tmpfs) filesystem, size=75%...
TODO: I need to refactor checksum code a bit to work propertly here for only checksum downloaded *.sfs files. (current checksum is for all files inside $archisobasedir) -- Gerardo Exequiel Pozzi \cos^2\alpha + \sin^2\alpha = 1
participants (3)
-
Dieter Plaetinck
-
Gerardo Exequiel Pozzi
-
Thomas Bächler