[arch-releng] reduce image size
Hello everybody,
now that we support root filesystem on squashfs/overlayfs... Do we want to reduce image size some more?
Currently we generate a squashfs for each architecture, one for x86_64 and one for i686. How about merging these into one filesystem? Only extra thing to do is a bind mount for the subdirectory to /new_root.
The benefit is the deduplication support in squashfs, identical files are added to the filesystem just once. I did not yet test with the official media, but I'd expect the size to shrink about 120MB...
This has a drawback, though: Copy-to-RAM would copy the files for both architectures, not just the needed one.
Any comments?
Christian Hesse list@eworm.de on Thu, 2015/10/01 16:53:
Hello everybody,
now that we support root filesystem on squashfs/overlayfs... Do we want to reduce image size some more?
Currently we generate a squashfs for each architecture, one for x86_64 and one for i686. How about merging these into one filesystem? Only extra thing to do is a bind mount for the subdirectory to /new_root.
That's not true... We do not have to bind mount but add the arch to path overlay mount options.
The benefit is the deduplication support in squashfs, identical files are added to the filesystem just once. I did not yet test with the official media, but I'd expect the size to shrink about 120MB...
I've build my own media and reduced size from 892M to 738M.
This has a drawback, though: Copy-to-RAM would copy the files for both architectures, not just the needed one.
To compare... To split airootfs.sfs image were about 400MB each... The single one with both archs is 641MB.
Any comments?
Did some quick and dirty hacks:
https://github.com/eworm-de/archiso/commits/single-squashfs
This needs some polishing... Would be great to have a single switch to decide whether or not the build a single airootfs.
Comments are still welcome. ;)
On 10/02/2015 06:02 AM, Christian Hesse wrote:
Christian Hesse list@eworm.de on Thu, 2015/10/01 16:53:
Hello everybody,
now that we support root filesystem on squashfs/overlayfs... Do we want to reduce image size some more?
Currently we generate a squashfs for each architecture, one for x86_64 and one for i686. How about merging these into one filesystem? Only extra thing to do is a bind mount for the subdirectory to /new_root.
That's not true... We do not have to bind mount but add the arch to path overlay mount options.
The benefit is the deduplication support in squashfs, identical files are added to the filesystem just once. I did not yet test with the official media, but I'd expect the size to shrink about 120MB...
I've build my own media and reduced size from 892M to 738M.
This has a drawback, though: Copy-to-RAM would copy the files for both architectures, not just the needed one.
To compare... To split airootfs.sfs image were about 400MB each... The single one with both archs is 641MB.
Any comments?
Did some quick and dirty hacks:
https://github.com/eworm-de/archiso/commits/single-squashfs
This needs some polishing... Would be great to have a single switch to decide whether or not the build a single airootfs.
Comments are still welcome. ;)
This is really great, the only downside is I am not sure how good for "releng" iso: for example for PXE setups, more bw needed+more memory, more memory needed on copytoram, and on loopback boot.
I agree in adding as optional feature, but not by default for "releng".
Thanks for your work!
Gerardo Exequiel Pozzi vmlinuz386@gmail.com on Fri, 2015/10/02 16:22:
On 10/02/2015 06:02 AM, Christian Hesse wrote:
Christian Hesse list@eworm.de on Thu, 2015/10/01 16:53:
Hello everybody,
now that we support root filesystem on squashfs/overlayfs... Do we want to reduce image size some more?
Currently we generate a squashfs for each architecture, one for x86_64 and one for i686. How about merging these into one filesystem? Only extra thing to do is a bind mount for the subdirectory to /new_root.
That's not true... We do not have to bind mount but add the arch to path overlay mount options.
The benefit is the deduplication support in squashfs, identical files are added to the filesystem just once. I did not yet test with the official media, but I'd expect the size to shrink about 120MB...
I've build my own media and reduced size from 892M to 738M.
This has a drawback, though: Copy-to-RAM would copy the files for both architectures, not just the needed one.
To compare... To split airootfs.sfs image were about 400MB each... The single one with both archs is 641MB.
Any comments?
Did some quick and dirty hacks:
https://github.com/eworm-de/archiso/commits/single-squashfs
This needs some polishing... Would be great to have a single switch to decide whether or not the build a single airootfs.
Comments are still welcome. ;)
This is really great, the only downside is I am not sure how good for "releng" iso: for example for PXE setups,
I do use a modified nbd hook that allows to boot without copytoram... :-p
more bw needed+more memory, more memory needed on copytoram,
That's what I was speaking about. No benefit without downside. :D
and on loopback boot.
Really? IMHO you can boot in loopback mode without copytoram just fine.
I agree in adding as optional feature, but not by default for "releng".
ACK. ;) Wondering whether or not I will use it by default for my own media...
So do you want me to polish this thing or do you want to implement a clean solution yourself? In first case I would like to discuss some details before.
Some general thoughts... In make_prepare() we do a 'cp -a ...' - does that make sense? We can skip that step when bootstrapping to the correct directory, no? Moreover, sometimes we call 'mkarchiso -w "${work_dir}" ...' and add the subdirectories later in function, sometimes we call 'mkarchiso -w "${work_dir}/${arch}" ...' or similar. Is there a reason for that? To make that path handling unique should simplify implementing my feature. This would break existing configs, though.
Thanks for your work!
No bit deal. I want to have the best live media myself. ;)
On 10/02/2015 04:44 PM, Christian Hesse wrote:
Gerardo Exequiel Pozzi vmlinuz386@gmail.com on Fri, 2015/10/02 16:22:
On 10/02/2015 06:02 AM, Christian Hesse wrote:
Christian Hesse list@eworm.de on Thu, 2015/10/01 16:53:
Hello everybody,
now that we support root filesystem on squashfs/overlayfs... Do we want to reduce image size some more?
Currently we generate a squashfs for each architecture, one for x86_64 and one for i686. How about merging these into one filesystem? Only extra thing to do is a bind mount for the subdirectory to /new_root.
That's not true... We do not have to bind mount but add the arch to path overlay mount options.
The benefit is the deduplication support in squashfs, identical files are added to the filesystem just once. I did not yet test with the official media, but I'd expect the size to shrink about 120MB...
I've build my own media and reduced size from 892M to 738M.
This has a drawback, though: Copy-to-RAM would copy the files for both architectures, not just the needed one.
To compare... To split airootfs.sfs image were about 400MB each... The single one with both archs is 641MB.
Any comments?
Did some quick and dirty hacks:
https://github.com/eworm-de/archiso/commits/single-squashfs
This needs some polishing... Would be great to have a single switch to decide whether or not the build a single airootfs.
Comments are still welcome. ;)
This is really great, the only downside is I am not sure how good for "releng" iso: for example for PXE setups,
I do use a modified nbd hook that allows to boot without copytoram... :-p
ndb was good sometimes, was bad sometimes... I do not know current status. anyway ndb was just the first PXE mode in archiso, other modes are NFS, HTTP. To avoid problems with dhcp and with systemd both nfs and ndb are always copytoram.
more bw needed+more memory, more memory needed on copytoram,
That's what I was speaking about. No benefit without downside. :D
yeap
and on loopback boot.
Really? IMHO you can boot in loopback mode without copytoram just fine.
oops, I want to say memdisk (syslinux), anyway this mode is a bit problematic with crazy bios.
I agree in adding as optional feature, but not by default for "releng".
ACK. ;) Wondering whether or not I will use it by default for my own media...
So do you want me to polish this thing or do you want to implement a clean solution yourself?
Go ahead. I like it (I did not look too deep at the code), I can review when I get some time or I can think "in my way" and compare both options.
The new image should be placed in a "any" (non-$arch) directory on isofs.
In first case I would like to discuss some details before.
Some general thoughts... In make_prepare() we do a 'cp -a ...' - does that make sense? We can skip that step when bootstrapping to the correct directory, no? Moreover, sometimes we call 'mkarchiso -w "${work_dir}" ...' and add the subdirectories later in function, sometimes we call 'mkarchiso -w "${work_dir}/${arch}" ...' or similar. Is there a reason for that? To make that path handling unique should simplify implementing my feature. This would break existing configs, though.
"cp .. -l ..." all hardlinks, if my memory does not fail....this is done to make, the build process of dual iso much more easy, I do not remember exactly why in this way. The idea was, build each $arch at time, then merge in last step, also easy to "re-exec build.sh" at intermediate points.
Thanks for your work!
No bit deal. I want to have the best live media myself. ;)
nice!
On 10/02/2015 08:10 PM, Gerardo Exequiel Pozzi wrote:
Some general thoughts... In make_prepare() we do a 'cp -a ...' - does that make sense? We can skip that step when bootstrapping to the correct directory, no? Moreover, sometimes we call 'mkarchiso -w "${work_dir}" ...' and add the subdirectories later in function, sometimes we call 'mkarchiso -w "${work_dir}/${arch}" ...' or similar. Is there a reason for that? To make that path handling unique should simplify implementing my feature. This would break existing configs, though.
"cp .. -l ..." all hardlinks, if my memory does not fail....this is done to make, the build process of dual iso much more easy, I do not remember exactly why in this way. The idea was, build each $arch at time, then merge in last step, also easy to "re-exec build.sh" at intermediate points.
correction: The point is, if you want to re-execute make_prepare(), (deleting the lock file made by run_once()), do it on the same "state" of untouched workdir, since _cleanup removes files and maybe you want to re-exec pacman, was good for testing purposes... can be removed, indeed # rm ... /airootfs can be uncommented....
I am currently working on this idea of single sfs ;)
On 10/03/2015 05:25 PM, Gerardo Exequiel Pozzi wrote:
On 10/02/2015 08:10 PM, Gerardo Exequiel Pozzi wrote:
Some general thoughts... In make_prepare() we do a 'cp -a ...' - does that make sense? We can skip that step when bootstrapping to the correct directory, no? Moreover, sometimes we call 'mkarchiso -w "${work_dir}" ...' and add the subdirectories later in function, sometimes we call 'mkarchiso -w "${work_dir}/${arch}" ...' or similar. Is there a reason for that? To make that path handling unique should simplify implementing my feature. This would break existing configs, though.
"cp .. -l ..." all hardlinks, if my memory does not fail....this is done to make, the build process of dual iso much more easy, I do not remember exactly why in this way. The idea was, build each $arch at time, then merge in last step, also easy to "re-exec build.sh" at intermediate points.
correction: The point is, if you want to re-execute make_prepare(), (deleting the lock file made by run_once()), do it on the same "state" of untouched workdir, since _cleanup removes files and maybe you want to re-exec pacman, was good for testing purposes... can be removed, indeed # rm ... /airootfs can be uncommented....
I am currently working on this idea of single sfs ;)
some drafts [#1] (non-working). Apart from this, seems to be a good time do to some cleanups of unused things, to make scripts more simple.
Gerardo Exequiel Pozzi vmlinuz386@gmail.com on Sun, 2015/10/04 16:51:
On 10/03/2015 05:25 PM, Gerardo Exequiel Pozzi wrote:
On 10/02/2015 08:10 PM, Gerardo Exequiel Pozzi wrote:
Some general thoughts... In make_prepare() we do a 'cp -a ...' - does that make sense? We can skip that step when bootstrapping to the correct directory, no? Moreover, sometimes we call 'mkarchiso -w "${work_dir}" ...' and add the subdirectories later in function, sometimes we call 'mkarchiso -w "${work_dir}/${arch}" ...' or similar. Is there a reason for that? To make that path handling unique should simplify implementing my feature. This would break existing configs, though.
"cp .. -l ..." all hardlinks, if my memory does not fail....this is done to make, the build process of dual iso much more easy, I do not remember exactly why in this way. The idea was, build each $arch at time, then merge in last step, also easy to "re-exec build.sh" at intermediate points.
correction: The point is, if you want to re-execute make_prepare(), (deleting the lock file made by run_once()), do it on the same "state" of untouched workdir, since _cleanup removes files and maybe you want to re-exec pacman, was good for testing purposes... can be removed, indeed # rm ... /airootfs can be uncommented....
Ah, I missing the flag about hardlinks... I've never made any use of the copy, though.
I am currently working on this idea of single sfs ;)
Great!
some drafts [#1] (non-working). Apart from this, seems to be a good time do to some cleanups of unused things, to make scripts more simple.
Let me know when this is supposed to work and I will give it a try. Thanks a lot!
participants (2)
-
Christian Hesse
-
Gerardo Exequiel Pozzi