[arch-releng] archiso with cowfile_size > 100
Hello everybody, actually cowfile_size is limited to values smaller or equal to 100. That way you have to decide at iso build time what size your filesystems should have. I have added a feature to allow values bigger than 100: In this case the read only device is linearly assembled with a sparse file (we do not have to care about size on disk as no write go to this file), this device is used for read only device then. After chroot (or when the system has bootet up) you can grow your filesystem to the new size. Growing the filesystem in initramfs would be possible, but we would have to add userspace utilities (resize2fs, xfs_growfs, ...) there... The patch is attached. What do you think? -- main(a){char*c=/* Schoene Gruesse */"B?IJj;MEH" "CX:;",b;for(a/* Chris get my mail address: */=0;b=c[a++];) putchar(b-1/(/* gcc -o sig sig.c && ./sig */b/42*2-3)*42);}
Christian Hesse <list@eworm.de> on Thu, 2013/08/29 15:22:
Hello everybody,
actually cowfile_size is limited to values smaller or equal to 100. That way you have to decide at iso build time what size your filesystems should have.
I have added a feature to allow values bigger than 100: In this case the read only device is linearly assembled with a sparse file (we do not have to care about size on disk as no write go to this file), this device is used for read only device then. After chroot (or when the system has bootet up) you can grow your filesystem to the new size.
Growing the filesystem in initramfs would be possible, but we would have to add userspace utilities (resize2fs, xfs_growfs, ...) there...
The patch is attached. What do you think?
Hmm, anything stripped my patch... (That's why the signature in invalid...) So I simply uploaded it: http://www.eworm.de/download/linux/archiso_growfs.patch -- main(a){char*c=/* Schoene Gruesse */"B?IJj;MEH" "CX:;",b;for(a/* Chris get my mail address: */=0;b=c[a++];) putchar(b-1/(/* gcc -o sig sig.c && ./sig */b/42*2-3)*42);}
On 29.08.2013 15:27, Christian Hesse wrote:
Hmm, anything stripped my patch... (That's why the signature in invalid...)
So I simply uploaded it: http://www.eworm.de/download/linux/archiso_growfs.patch
Please use "git send-email" since that allows for easy inline commenting and we get a full commit including subject and probably commit message which you didn't put into that linked patch file.
From: Christian Hesse <mail@eworm.de> actually cowfile_size is limited to values smaller or equal to 100. That way you have to decide at iso build time what size your filesystems should have. I have added a feature to allow values bigger than 100: In this case the read only device is linearly assembled with a sparse file (we do not have to care about size on disk as no write go to this file), this device is used for read only device then. After chroot (or when the system has bootet up) you can grow your filesystem to the new size. Growing the filesystem in initramfs would be possible, but we would have to add userspace utilities (resize2fs, xfs_growfs, ...) there... --- archiso/initcpio/hooks/archiso | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/archiso/initcpio/hooks/archiso b/archiso/initcpio/hooks/archiso index 66a6273..94d5a99 100644 --- a/archiso/initcpio/hooks/archiso +++ b/archiso/initcpio/hooks/archiso @@ -6,13 +6,28 @@ _mnt_fs() { local img_fullname="${img##*/}"; local img_name="${img_fullname%%.*}" local dm_snap_name="${dm_snap_prefix}_${img_name}" - local ro_dev ro_dev_size rw_dev + local ro_dev ro_dev_size rw_dev rw_dev_size pad_dev pad_dev_size ro_dev=$(losetup --find --show --read-only "${img}") echo ${ro_dev} >> /run/archiso/used_block_devices ro_dev_size=$(blockdev --getsz ${ro_dev}) + if [[ "${cowfile_size}" == "100" ]]; then rw_dev_size=${ro_dev_size} + elif [[ "${cowfile_size}" -gt 100 ]]; then + pad_dev_size=$((ro_dev_size/100*cowfile_size - ro_dev_size)) + dd of="/run/archiso/cowspace/${cow_directory}/${img_name}.pad" count=0 seek=${pad_dev_size} &> /dev/null + pad_dev=$(losetup --find --show --read-only "/run/archiso/cowspace/${cow_directory}/${img_name}.pad") + echo ${pad_dev} >> /run/archiso/used_block_devices + # this is just a sparse file that never gets written to, so just delete it + rm "/run/archiso/cowspace/${cow_directory}/${img_name}.pad" + echo -e "0 ${ro_dev_size} linear ${ro_dev} 0\n${ro_dev_size} ${pad_dev_size} linear ${pad_dev} 0" | dmsetup create "ro_${dm_snap_name}" + + ro_dev="/dev/mapper/ro_${dm_snap_name}" + ro_dev_size="$((ro_dev_size + pad_dev_size))" + rw_dev_size="${ro_dev_size}" + + msg ":: Device ${dm_snap_name} has been grown, remember to grow your filesystem!" else # size calculation done in this way to avoid integer overflow when ro_dev_size is > 10.2G rw_dev_size=$((ro_dev_size/100*cowfile_size)) -- 1.8.4
On 08/29/2013 10:55 AM, Christian Hesse wrote:
From: Christian Hesse <mail@eworm.de>
actually cowfile_size is limited to values smaller or equal to 100. That way you have to decide at iso build time what size your filesystems should have.
I have added a feature to allow values bigger than 100: In this case the read only device is linearly assembled with a sparse file (we do not have to care about size on disk as no write go to this file), this device is used for read only device then. After chroot (or when the system has bootet up) you can grow your filesystem to the new size.
Growing the filesystem in initramfs would be possible, but we would have to add userspace utilities (resize2fs, xfs_growfs, ...) there... --- archiso/initcpio/hooks/archiso | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)
diff --git a/archiso/initcpio/hooks/archiso b/archiso/initcpio/hooks/archiso index 66a6273..94d5a99 100644 --- a/archiso/initcpio/hooks/archiso +++ b/archiso/initcpio/hooks/archiso @@ -6,13 +6,28 @@ _mnt_fs() { local img_fullname="${img##*/}"; local img_name="${img_fullname%%.*}" local dm_snap_name="${dm_snap_prefix}_${img_name}" - local ro_dev ro_dev_size rw_dev + local ro_dev ro_dev_size rw_dev rw_dev_size pad_dev pad_dev_size
ro_dev=$(losetup --find --show --read-only "${img}") echo ${ro_dev} >> /run/archiso/used_block_devices ro_dev_size=$(blockdev --getsz ${ro_dev}) + if [[ "${cowfile_size}" == "100" ]]; then rw_dev_size=${ro_dev_size} + elif [[ "${cowfile_size}" -gt 100 ]]; then + pad_dev_size=$((ro_dev_size/100*cowfile_size - ro_dev_size)) + dd of="/run/archiso/cowspace/${cow_directory}/${img_name}.pad" count=0 seek=${pad_dev_size} &> /dev/null + pad_dev=$(losetup --find --show --read-only "/run/archiso/cowspace/${cow_directory}/${img_name}.pad") + echo ${pad_dev} >> /run/archiso/used_block_devices + # this is just a sparse file that never gets written to, so just delete it + rm "/run/archiso/cowspace/${cow_directory}/${img_name}.pad" + echo -e "0 ${ro_dev_size} linear ${ro_dev} 0\n${ro_dev_size} ${pad_dev_size} linear ${pad_dev} 0" | dmsetup create "ro_${dm_snap_name}" + + ro_dev="/dev/mapper/ro_${dm_snap_name}" + ro_dev_size="$((ro_dev_size + pad_dev_size))" + rw_dev_size="${ro_dev_size}" + + msg ":: Device ${dm_snap_name} has been grown, remember to grow your filesystem!" else # size calculation done in this way to avoid integer overflow when ro_dev_size is > 10.2G rw_dev_size=$((ro_dev_size/100*cowfile_size))
Too much complex, and some sematics broken (for example, you are mixing terms for setting non-cowdevices or storing non-cow file inside cowspace, missing proper device shutdown, ...) Why not just use a value above 100%? ;) Thanks for your interest on improving archiso :) -- Gerardo Exequiel Pozzi \cos^2\alpha + \sin^2\alpha = 1
On 08/29/2013 07:31 PM, Gerardo Exequiel Pozzi wrote:
On 08/29/2013 10:55 AM, Christian Hesse wrote:
From: Christian Hesse <mail@eworm.de>
actually cowfile_size is limited to values smaller or equal to 100. That way you have to decide at iso build time what size your filesystems should have.
I have added a feature to allow values bigger than 100: In this case the read only device is linearly assembled with a sparse file (we do not have to care about size on disk as no write go to this file), this device is used for read only device then. After chroot (or when the system has bootet up) you can grow your filesystem to the new size.
Growing the filesystem in initramfs would be possible, but we would have to add userspace utilities (resize2fs, xfs_growfs, ...) there... --- archiso/initcpio/hooks/archiso | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)
diff --git a/archiso/initcpio/hooks/archiso b/archiso/initcpio/hooks/archiso index 66a6273..94d5a99 100644 --- a/archiso/initcpio/hooks/archiso +++ b/archiso/initcpio/hooks/archiso @@ -6,13 +6,28 @@ _mnt_fs() { local img_fullname="${img##*/}"; local img_name="${img_fullname%%.*}" local dm_snap_name="${dm_snap_prefix}_${img_name}" - local ro_dev ro_dev_size rw_dev + local ro_dev ro_dev_size rw_dev rw_dev_size pad_dev pad_dev_size
ro_dev=$(losetup --find --show --read-only "${img}") echo ${ro_dev} >> /run/archiso/used_block_devices ro_dev_size=$(blockdev --getsz ${ro_dev}) + if [[ "${cowfile_size}" == "100" ]]; then rw_dev_size=${ro_dev_size} + elif [[ "${cowfile_size}" -gt 100 ]]; then + pad_dev_size=$((ro_dev_size/100*cowfile_size - ro_dev_size)) + dd of="/run/archiso/cowspace/${cow_directory}/${img_name}.pad" count=0 seek=${pad_dev_size} &> /dev/null + pad_dev=$(losetup --find --show --read-only "/run/archiso/cowspace/${cow_directory}/${img_name}.pad") + echo ${pad_dev} >> /run/archiso/used_block_devices + # this is just a sparse file that never gets written to, so just delete it + rm "/run/archiso/cowspace/${cow_directory}/${img_name}.pad" + echo -e "0 ${ro_dev_size} linear ${ro_dev} 0\n${ro_dev_size} ${pad_dev_size} linear ${pad_dev} 0" | dmsetup create "ro_${dm_snap_name}" + + ro_dev="/dev/mapper/ro_${dm_snap_name}" + ro_dev_size="$((ro_dev_size + pad_dev_size))" + rw_dev_size="${ro_dev_size}" + + msg ":: Device ${dm_snap_name} has been grown, remember to grow your filesystem!" else # size calculation done in this way to avoid integer overflow when ro_dev_size is > 10.2G rw_dev_size=$((ro_dev_size/100*cowfile_size))
Too much complex, and some sematics broken (for example, you are mixing terms for setting non-cowdevices or storing non-cow file inside cowspace, missing proper device shutdown, ...)
Why not just use a value above 100%? ;)
Thanks for your interest on improving archiso :)
Ignore me, was a bad smart answer, since you want to increse the size of the device! In any case, the semantics of cowfile_size should not be changed. You can add another parameter like grow_ro_dev_size=, also should be ensure the proper device shutdown and persistence. We are reaching 3.11 and still no "overlayfs" :( (was promised for 3.10, 3.2, ...) But before adding this I want an opinion from Thomas. -- Gerardo Exequiel Pozzi \cos^2\alpha + \sin^2\alpha = 1
Gerardo Exequiel Pozzi <vmlinuz386@yahoo.com.ar> on Thu, 2013/08/29 20:01:
On 08/29/2013 07:31 PM, Gerardo Exequiel Pozzi wrote:
On 08/29/2013 10:55 AM, Christian Hesse wrote:
From: Christian Hesse <mail@eworm.de>
actually cowfile_size is limited to values smaller or equal to 100. That way you have to decide at iso build time what size your filesystems should have.
I have added a feature to allow values bigger than 100: In this case the read only device is linearly assembled with a sparse file (we do not have to care about size on disk as no write go to this file), this device is used for read only device then. After chroot (or when the system has bootet up) you can grow your filesystem to the new size.
Growing the filesystem in initramfs would be possible, but we would have to add userspace utilities (resize2fs, xfs_growfs, ...) there... --- archiso/initcpio/hooks/archiso | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)
diff --git a/archiso/initcpio/hooks/archiso b/archiso/initcpio/hooks/archiso index 66a6273..94d5a99 100644 --- a/archiso/initcpio/hooks/archiso +++ b/archiso/initcpio/hooks/archiso @@ -6,13 +6,28 @@ _mnt_fs() { local img_fullname="${img##*/}"; local img_name="${img_fullname%%.*}" local dm_snap_name="${dm_snap_prefix}_${img_name}" - local ro_dev ro_dev_size rw_dev + local ro_dev ro_dev_size rw_dev rw_dev_size pad_dev pad_dev_size
ro_dev=$(losetup --find --show --read-only "${img}") echo ${ro_dev} >> /run/archiso/used_block_devices ro_dev_size=$(blockdev --getsz ${ro_dev}) + if [[ "${cowfile_size}" == "100" ]]; then rw_dev_size=${ro_dev_size} + elif [[ "${cowfile_size}" -gt 100 ]]; then + pad_dev_size=$((ro_dev_size/100*cowfile_size - ro_dev_size)) + dd of="/run/archiso/cowspace/${cow_directory}/${img_name}.pad" count=0 seek=${pad_dev_size} &> /dev/null + pad_dev=$(losetup --find --show --read-only "/run/archiso/cowspace/${cow_directory}/${img_name}.pad") + echo ${pad_dev} >> /run/archiso/used_block_devices + # this is just a sparse file that never gets written to, so just delete it + rm "/run/archiso/cowspace/${cow_directory}/${img_name}.pad" + echo -e "0 ${ro_dev_size} linear ${ro_dev} 0\n${ro_dev_size} ${pad_dev_size} linear ${pad_dev} 0" | dmsetup create "ro_${dm_snap_name}" + + ro_dev="/dev/mapper/ro_${dm_snap_name}" + ro_dev_size="$((ro_dev_size + pad_dev_size))" + rw_dev_size="${ro_dev_size}" + + msg ":: Device ${dm_snap_name} has been grown, remember to grow your filesystem!" else # size calculation done in this way to avoid integer overflow when ro_dev_size is > 10.2G rw_dev_size=$((ro_dev_size/100*cowfile_size))
Too much complex, and some sematics broken (for example, you are mixing terms for setting non-cowdevices or storing non-cow file inside cowspace, missing proper device shutdown, ...)
Why not just use a value above 100%? ;)
Thanks for your interest on improving archiso :)
Ignore me, was a bad smart answer, since you want to increse the size of the device!
In any case, the semantics of cowfile_size should not be changed. You can add another parameter like grow_ro_dev_size=,
I add a use case for what was completely useless before. (In fact the cow files got bigger, but the resulting device size stayed the same.) Do you really want an new parameter for that? I do not mind, I can change that if desired.
also should be ensure the proper device shutdown and persistence.
systemd should disassemble the devices, no? What is needed more? And what do you refer with persistance? Everything should work just fine given a persistant system is booted with the same parameters.
We are reaching 3.11 and still no "overlayfs" :( (was promised for 3.10, 3.2, ...)
Well, yes... On the other hand I really like the idea to solve this with device mapper. ;) Just thinking about how to integrate the userspace tools and handling for this. I would like to add a new hook that adds the binary files (resize2fs, xfs_growfs, btrfs), so anybody can decide whether or not to include. But where do we call this? -- main(a){char*c=/* Schoene Gruesse */"B?IJj;MEH" "CX:;",b;for(a/* Chris get my mail address: */=0;b=c[a++];) putchar(b-1/(/* gcc -o sig sig.c && ./sig */b/42*2-3)*42);}
On 08/30/2013 06:12 AM, Christian Hesse wrote:
Gerardo Exequiel Pozzi <vmlinuz386@yahoo.com.ar> on Thu, 2013/08/29 20:01:
On 08/29/2013 07:31 PM, Gerardo Exequiel Pozzi wrote:
On 08/29/2013 10:55 AM, Christian Hesse wrote:
From: Christian Hesse <mail@eworm.de>
actually cowfile_size is limited to values smaller or equal to 100. That way you have to decide at iso build time what size your filesystems should have.
I have added a feature to allow values bigger than 100: In this case the read only device is linearly assembled with a sparse file (we do not have to care about size on disk as no write go to this file), this device is used for read only device then. After chroot (or when the system has bootet up) you can grow your filesystem to the new size.
Growing the filesystem in initramfs would be possible, but we would have to add userspace utilities (resize2fs, xfs_growfs, ...) there... --- archiso/initcpio/hooks/archiso | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)
diff --git a/archiso/initcpio/hooks/archiso b/archiso/initcpio/hooks/archiso index 66a6273..94d5a99 100644 --- a/archiso/initcpio/hooks/archiso +++ b/archiso/initcpio/hooks/archiso @@ -6,13 +6,28 @@ _mnt_fs() { local img_fullname="${img##*/}"; local img_name="${img_fullname%%.*}" local dm_snap_name="${dm_snap_prefix}_${img_name}" - local ro_dev ro_dev_size rw_dev + local ro_dev ro_dev_size rw_dev rw_dev_size pad_dev pad_dev_size
ro_dev=$(losetup --find --show --read-only "${img}") echo ${ro_dev} >> /run/archiso/used_block_devices ro_dev_size=$(blockdev --getsz ${ro_dev}) + if [[ "${cowfile_size}" == "100" ]]; then rw_dev_size=${ro_dev_size} + elif [[ "${cowfile_size}" -gt 100 ]]; then + pad_dev_size=$((ro_dev_size/100*cowfile_size - ro_dev_size)) + dd of="/run/archiso/cowspace/${cow_directory}/${img_name}.pad" count=0 seek=${pad_dev_size} &> /dev/null + pad_dev=$(losetup --find --show --read-only "/run/archiso/cowspace/${cow_directory}/${img_name}.pad") + echo ${pad_dev} >> /run/archiso/used_block_devices + # this is just a sparse file that never gets written to, so just delete it + rm "/run/archiso/cowspace/${cow_directory}/${img_name}.pad" + echo -e "0 ${ro_dev_size} linear ${ro_dev} 0\n${ro_dev_size} ${pad_dev_size} linear ${pad_dev} 0" | dmsetup create "ro_${dm_snap_name}" + + ro_dev="/dev/mapper/ro_${dm_snap_name}" + ro_dev_size="$((ro_dev_size + pad_dev_size))" + rw_dev_size="${ro_dev_size}" + + msg ":: Device ${dm_snap_name} has been grown, remember to grow your filesystem!" else # size calculation done in this way to avoid integer overflow when ro_dev_size is > 10.2G rw_dev_size=$((ro_dev_size/100*cowfile_size))
Too much complex, and some sematics broken (for example, you are mixing terms for setting non-cowdevices or storing non-cow file inside cowspace, missing proper device shutdown, ...)
Why not just use a value above 100%? ;)
Thanks for your interest on improving archiso :)
Ignore me, was a bad smart answer, since you want to increse the size of the device!
In any case, the semantics of cowfile_size should not be changed. You can add another parameter like grow_ro_dev_size=,
I add a use case for what was completely useless before. (In fact the cow files got bigger, but the resulting device size stayed the same.) Do you really want an new parameter for that? I do not mind, I can change that if desired.
Yes using a new parameter should be more clear.
also should be ensure the proper device shutdown and persistence.
systemd should disassemble the devices, no? What is needed more?
No, systemd can not stop this complex setup, indeed we can not use the provided shutdown script by mkinitcpio, we are using a own shutdown script.
And what do you refer with persistance? Everything should work just fine given a persistant system is booted with the same parameters.
cow_persistent=P | cow_persistent=N For this extension, should not use the same directory. Of course, this "extension" file act as ro-device, will never be modified during runtime.
We are reaching 3.11 and still no "overlayfs" :( (was promised for 3.10, 3.2, ...)
Well, yes... On the other hand I really like the idea to solve this with device mapper. ;)
If at some point linux implements an overlayfs, and I still alive ;), I want to move back to filesystem solution instead of block-based like now.
Just thinking about how to integrate the userspace tools and handling for this. I would like to add a new hook that adds the binary files (resize2fs, xfs_growfs, btrfs), so anybody can decide whether or not to include. But where do we call this?
I did not see any real usage of this "extension" inside official image. If you are building a custom image build, why not just create a big filesystem from the start? -- Gerardo Exequiel Pozzi \cos^2\alpha + \sin^2\alpha = 1
Gerardo Exequiel Pozzi <vmlinuz386@yahoo.com.ar> on Fri, 2013/08/30 13:01: > On 08/30/2013 06:12 AM, Christian Hesse wrote: > > Gerardo Exequiel Pozzi <vmlinuz386@yahoo.com.ar> on Thu, 2013/08/29 20:01: > >> Ignore me, was a bad smart answer, since you want to increse the size of > >> the device! > >> > >> In any case, the semantics of cowfile_size should not be changed. You > >> can add another parameter like grow_ro_dev_size=, > > > > I add a use case for what was completely useless before. (In fact the cow > > files got bigger, but the resulting device size stayed the same.) Do you > > really want an new parameter for that? > > I do not mind, I can change that if desired. > > Yes using a new parameter should be more clear. Done. This parameter only takes affect with values greater than 100. > >> also should be ensure > >> the proper device shutdown and persistence. > > > > systemd should disassemble the devices, no? What is needed more? > > No, systemd can not stop this complex setup, indeed we can not use the > provided shutdown script by mkinitcpio, we are using a own shutdown script. Ah, looks like I missed to add a file to used_block_devices. Is it ok now? Hope so... Otherwise I have to take a deeper look at the shutdown script. > > And what do you refer with persistance? Everything should work just fine > > given a persistant system is booted with the same parameters. > > cow_persistent=P | cow_persistent=N > > For this extension, should not use the same directory. > > Of course, this "extension" file act as ro-device, will never be > modified during runtime. I create a directory /run/archiso/growspace. cowspace is not touched by my code. Is that ok? > >> We are reaching 3.11 and still no "overlayfs" :( (was promised for 3.10, > >> 3.2, ...) > > > > Well, yes... > > On the other hand I really like the idea to solve this with device > > mapper. ;) > > If at some point linux implements an overlayfs, and I still alive ;), I > want to move back to filesystem solution instead of block-based like now. I am fine with that as long as it is upstream linux. ;) > > Just thinking about how to integrate the userspace tools and handling for > > this. I would like to add a new hook that adds the binary files > > (resize2fs, xfs_growfs, btrfs), so anybody can decide whether or not to > > include. But where do we call this? > > I did not see any real usage of this "extension" inside official image. > If you are building a custom image build, why not just create a big > filesystem from the start? Because bigger filesystem has more metadata. Image size would increase for a use case I need only very seldom. I perfer to add some lines of code, having the possibility to grow device and filesystem when needed. Here is my new patch, this also has some code to actually grow the filesystem. From 7609b2ba1b3b876efc4199d9653b389303cd980a Mon Sep 17 00:00:00 2001 From: Christian Hesse <mail@eworm.de> Date: Fri, 30 Aug 2013 21:10:44 +0200 Subject: [PATCH 1/1] support growing devices and filesystems --- archiso/initcpio/hooks/archiso | 59 ++++++++++++++++++++++++++++++++- archiso/initcpio/install/archiso_growfs | 9 +++++ 2 files changed, 67 insertions(+), 1 deletion(-) create mode 100644 archiso/initcpio/install/archiso_growfs diff --git a/archiso/initcpio/hooks/archiso b/archiso/initcpio/hooks/archiso index 66a6273..76dcb6e 100644 --- a/archiso/initcpio/hooks/archiso +++ b/archiso/initcpio/hooks/archiso @@ -6,11 +6,27 @@ _mnt_fs() { local img_fullname="${img##*/}"; local img_name="${img_fullname%%.*}" local dm_snap_name="${dm_snap_prefix}_${img_name}" - local ro_dev ro_dev_size rw_dev + local ro_dev ro_dev_size rw_dev rw_dev_size pad_dev pad_dev_size ro_dev=$(losetup --find --show --read-only "${img}") echo ${ro_dev} >> /run/archiso/used_block_devices ro_dev_size=$(blockdev --getsz ${ro_dev}) + + if [[ "${grow_ro_dev_size}" -gt 100 ]]; then + pad_dev_size=$((ro_dev_size/100*grow_ro_dev_size - ro_dev_size)) + dd of="/run/archiso/growspace/${img_name}.pad" count=0 seek=${pad_dev_size} &> /dev/null + pad_dev=$(losetup --find --show --read-only "/run/archiso/growspace/${img_name}.pad") + echo ${pad_dev} >> /run/archiso/used_block_devices + echo -e "0 ${ro_dev_size} linear ${ro_dev} 0\n${ro_dev_size} ${pad_dev_size} linear ${pad_dev} 0" | dmsetup create "ro_${dm_snap_name}" + echo $(readlink -f /dev/mapper/${dm_snap_name}) >> /run/archiso/used_block_devices + + ro_dev="/dev/mapper/ro_${dm_snap_name}" + ro_dev_size="$((ro_dev_size + pad_dev_size))" + rw_dev_size="${ro_dev_size}" + + msg ":: Device ${dm_snap_name} has been grown, remember to grow your filesystem!" + fi + if [[ "${cowfile_size}" == "100" ]]; then rw_dev_size=${ro_dev_size} else @@ -43,6 +59,40 @@ _mnt_fs() { echo $(readlink -f /dev/mapper/${dm_snap_name}) >> /run/archiso/used_block_devices } +# args: name, fs_type +_grow_fs() { + local name="${1}" + local fs_type="${2}" + local device="/dev/mapper/${dm_snap_prefix}_${name}" + + case ${aitab_fs_type} in + ext*) + if [[ -x /usr/bin/resize2fs ]]; then + msg ":: Growing ${fs_type} filesystem for ${name}" + resize2fs "${device}" >/dev/null + else + msg ":: Not growing ${name}, userspace tools not available." + fi + ;; + xfs) + if [[ -x /usr/bin/xfs_growfs ]]; then + msg ":: Growing ${fs_type} filesystem for ${name}" + xfs_growfs "${device}" >/dev/null + else + msg ":: Not growing ${name}, userspace tools not available." + fi + ;; + btrfs) + if [[ -x /usr/bin/btrfs ]]; then + msg ":: Growing ${fs_type} filesystem for ${name}" + btrfs resize max "${device}" >/dev/null + else + msg ":: Not growing ${name}, userspace tools not available." + fi + ;; + esac +} + # args: /path/to/image_file, mountpoint _mnt_sfs() { local img="${1}" @@ -112,6 +163,7 @@ run_hook() { else cowfile_size=${cowfile_size/%} fi + [[ -z "${grow_ro_dev_size}" ]] && grow_ro_dev_size="100" if [[ -z "${aitab}" ]]; then aitab="/run/archiso/bootmnt/${archisobasedir}/aitab" @@ -188,6 +240,10 @@ archiso_mount_handler() { fi mkdir -p "/run/archiso/cowspace/${cow_directory}" + if [[ "${grow_ro_dev_size}" -gt 100 ]]; then + mkdir -p /run/archiso/growspace + fi + local aitab_img aitab_mnt aitab_arch aitab_sfs_comp aitab_fs_type aitab_fs_size while read aitab_img aitab_mnt aitab_arch aitab_sfs_comp aitab_fs_type aitab_fs_size; do [[ "${aitab_img#\#}" != "${aitab_img}" ]] && continue @@ -195,6 +251,7 @@ archiso_mount_handler() { if [[ "${aitab_fs_type}" != "none" ]]; then _mnt_sfs "/run/archiso/bootmnt/${archisobasedir}/${aitab_arch}/${aitab_img}.fs.sfs" "/run/archiso/sfs/${aitab_img}" _mnt_fs "/run/archiso/sfs/${aitab_img}/${aitab_img}.fs" "${newroot}" "${aitab_mnt}" + [[ "${grow_ro_dev_size}" -gt 100 ]] && _grow_fs "${aitab_img}" "${aitab_fs_type}" else _mnt_sfs "/run/archiso/bootmnt/${archisobasedir}/${aitab_arch}/${aitab_img}.sfs" "${newroot}${aitab_mnt}" fi diff --git a/archiso/initcpio/install/archiso_growfs b/archiso/initcpio/install/archiso_growfs new file mode 100644 index 0000000..6334a3c --- /dev/null +++ b/archiso/initcpio/install/archiso_growfs @@ -0,0 +1,9 @@ +#!/bin/bash + +build() { + add_binary btrfs + add_binary resize2fs + add_binary xfs_growfs +} + +# vim: set ft=sh ts=4 sw=4 et: -- main(a){char*c=/* Schoene Gruesse */"B?IJj;MEH" "CX:;",b;for(a/* Chris get my mail address: */=0;b=c[a++];) putchar(b-1/(/* gcc -o sig sig.c && ./sig */b/42*2-3)*42);}
On 08/30/2013 06:36 PM, Christian Hesse wrote: > Gerardo Exequiel Pozzi <vmlinuz386@yahoo.com.ar> on Fri, 2013/08/30 13:01: >> On 08/30/2013 06:12 AM, Christian Hesse wrote: >>> Gerardo Exequiel Pozzi <vmlinuz386@yahoo.com.ar> on Thu, 2013/08/29 20:01: >>>> Ignore me, was a bad smart answer, since you want to increse the size of >>>> the device! >>>> >>>> In any case, the semantics of cowfile_size should not be changed. You >>>> can add another parameter like grow_ro_dev_size=, >>> >>> I add a use case for what was completely useless before. (In fact the cow >>> files got bigger, but the resulting device size stayed the same.) Do you >>> really want an new parameter for that? >>> I do not mind, I can change that if desired. >> >> Yes using a new parameter should be more clear. > > Done. > This parameter only takes affect with values greater than 100. > For my first impression (of what name says), not much clear. What is grow_ro_dev_size=? a percent? In that case a default 100% says: grow ro_dev in 100% -> double ro_dev_size and 0% -> no changes. >>>> also should be ensure >>>> the proper device shutdown and persistence. >>> >>> systemd should disassemble the devices, no? What is needed more? >> >> No, systemd can not stop this complex setup, indeed we can not use the >> provided shutdown script by mkinitcpio, we are using a own shutdown script. > > Ah, looks like I missed to add a file to used_block_devices. Is it ok now? > Hope so... Otherwise I have to take a deeper look at the shutdown script. > Maybe, you need to check if all things stopped properly when switch back to initramfs on shutdown. >>> And what do you refer with persistance? Everything should work just fine >>> given a persistant system is booted with the same parameters. >> >> cow_persistent=P | cow_persistent=N >> >> For this extension, should not use the same directory. >> >> Of course, this "extension" file act as ro-device, will never be >> modified during runtime. > > I create a directory /run/archiso/growspace. cowspace is not touched by my > code. Is that ok? Looking at the code, growspace files is created on rootfs then deleted on switch-root. Should be on another tmpfs. But doing in a different way, you can use a dm-zero target instead. Since you never never write to this block device, is just to say "i have a much bigger read-only device". > >>>> We are reaching 3.11 and still no "overlayfs" :( (was promised for 3.10, >>>> 3.2, ...) >>> >>> Well, yes... >>> On the other hand I really like the idea to solve this with device >>> mapper. ;) >> >> If at some point linux implements an overlayfs, and I still alive ;), I >> want to move back to filesystem solution instead of block-based like now. > > I am fine with that as long as it is upstream linux. ;) Sure :) > >>> Just thinking about how to integrate the userspace tools and handling for >>> this. I would like to add a new hook that adds the binary files >>> (resize2fs, xfs_growfs, btrfs), so anybody can decide whether or not to >>> include. But where do we call this? >> >> I did not see any real usage of this "extension" inside official image. >> If you are building a custom image build, why not just create a big >> filesystem from the start? > > Because bigger filesystem has more metadata. Image size would increase for a > use case I need only very seldom. > I perfer to add some lines of code, having the possibility to grow device and > filesystem when needed. > True, but this can be done from another perspective, just create a big file for filesystem, but use few blocks, since squashfs supports sparse files. To do this in the most trivial way is just adding one line to mkarchiso. Let says: truncate -s 16G ${_fs_img} after "mkfs". (or even better! truncate -s 2T ${_fs_img} :D) There is only one downside if cowspace should be stored on filesystems that does not support sparse files like FAT cowspace_size= is much more important to be specified now. This avoid at all this complex logic, what do you think? > Here is my new patch, this also has some code to actually grow the filesystem. > > From 7609b2ba1b3b876efc4199d9653b389303cd980a Mon Sep 17 00:00:00 2001 > From: Christian Hesse <mail@eworm.de> > Date: Fri, 30 Aug 2013 21:10:44 +0200 > Subject: [PATCH 1/1] support growing devices and filesystems > > --- > archiso/initcpio/hooks/archiso | 59 > ++++++++++++++++++++++++++++++++- archiso/initcpio/install/archiso_growfs | > 9 +++++ 2 files changed, 67 insertions(+), 1 deletion(-) > create mode 100644 archiso/initcpio/install/archiso_growfs > > diff --git a/archiso/initcpio/hooks/archiso b/archiso/initcpio/hooks/archiso > index 66a6273..76dcb6e 100644 > --- a/archiso/initcpio/hooks/archiso > +++ b/archiso/initcpio/hooks/archiso > @@ -6,11 +6,27 @@ _mnt_fs() { > local img_fullname="${img##*/}"; > local img_name="${img_fullname%%.*}" > local dm_snap_name="${dm_snap_prefix}_${img_name}" > - local ro_dev ro_dev_size rw_dev > + local ro_dev ro_dev_size rw_dev rw_dev_size pad_dev pad_dev_size > > ro_dev=$(losetup --find --show --read-only "${img}") > echo ${ro_dev} >> /run/archiso/used_block_devices > ro_dev_size=$(blockdev --getsz ${ro_dev}) > + > + if [[ "${grow_ro_dev_size}" -gt 100 ]]; then > + pad_dev_size=$((ro_dev_size/100*grow_ro_dev_size - ro_dev_size)) > + dd of="/run/archiso/growspace/${img_name}.pad" count=0 > seek=${pad_dev_size} &> /dev/null > + pad_dev=$(losetup --find --show --read-only > "/run/archiso/growspace/${img_name}.pad") > + echo ${pad_dev} >> /run/archiso/used_block_devices > + echo -e "0 ${ro_dev_size} linear ${ro_dev} 0\n${ro_dev_size} > ${pad_dev_size} linear ${pad_dev} 0" | dmsetup create "ro_${dm_snap_name}" > + echo $(readlink -f /dev/mapper/${dm_snap_name}) >>> /run/archiso/used_block_devices + > + ro_dev="/dev/mapper/ro_${dm_snap_name}" > + ro_dev_size="$((ro_dev_size + pad_dev_size))" > + rw_dev_size="${ro_dev_size}" > + > + msg ":: Device ${dm_snap_name} has been grown, remember to grow your > filesystem!" > + fi > + > if [[ "${cowfile_size}" == "100" ]]; then > rw_dev_size=${ro_dev_size} > else > @@ -43,6 +59,40 @@ _mnt_fs() { > echo $(readlink -f /dev/mapper/${dm_snap_name}) >>> /run/archiso/used_block_devices } > > +# args: name, fs_type > +_grow_fs() { > + local name="${1}" > + local fs_type="${2}" > + local device="/dev/mapper/${dm_snap_prefix}_${name}" > + > + case ${aitab_fs_type} in > + ext*) > + if [[ -x /usr/bin/resize2fs ]]; then > + msg ":: Growing ${fs_type} filesystem for ${name}" > + resize2fs "${device}" >/dev/null > + else > + msg ":: Not growing ${name}, userspace tools not available." > + fi > + ;; > + xfs) > + if [[ -x /usr/bin/xfs_growfs ]]; then > + msg ":: Growing ${fs_type} filesystem for ${name}" > + xfs_growfs "${device}" >/dev/null > + else > + msg ":: Not growing ${name}, userspace tools not available." > + fi > + ;; > + btrfs) > + if [[ -x /usr/bin/btrfs ]]; then > + msg ":: Growing ${fs_type} filesystem for ${name}" > + btrfs resize max "${device}" >/dev/null > + else > + msg ":: Not growing ${name}, userspace tools not available." > + fi > + ;; > + esac > +} > + > # args: /path/to/image_file, mountpoint > _mnt_sfs() { > local img="${1}" > @@ -112,6 +163,7 @@ run_hook() { > else > cowfile_size=${cowfile_size/%} > fi > + [[ -z "${grow_ro_dev_size}" ]] && grow_ro_dev_size="100" > > if [[ -z "${aitab}" ]]; then > aitab="/run/archiso/bootmnt/${archisobasedir}/aitab" > @@ -188,6 +240,10 @@ archiso_mount_handler() { > fi > mkdir -p "/run/archiso/cowspace/${cow_directory}" > > + if [[ "${grow_ro_dev_size}" -gt 100 ]]; then > + mkdir -p /run/archiso/growspace > + fi > + > local aitab_img aitab_mnt aitab_arch aitab_sfs_comp aitab_fs_type > aitab_fs_size while read aitab_img aitab_mnt aitab_arch aitab_sfs_comp > aitab_fs_type aitab_fs_size; do [[ "${aitab_img#\#}" != "${aitab_img}" ]] && > continue @@ -195,6 +251,7 @@ archiso_mount_handler() { > if [[ "${aitab_fs_type}" != "none" ]]; then > _mnt_sfs > "/run/archiso/bootmnt/${archisobasedir}/${aitab_arch}/${aitab_img}.fs.sfs" > "/run/archiso/sfs/${aitab_img}" _mnt_fs > "/run/archiso/sfs/${aitab_img}/${aitab_img}.fs" "${newroot}" "${aitab_mnt}" > + [[ "${grow_ro_dev_size}" -gt 100 ]] && _grow_fs "${aitab_img}" > "${aitab_fs_type}" else > _mnt_sfs > "/run/archiso/bootmnt/${archisobasedir}/${aitab_arch}/${aitab_img}.sfs" > "${newroot}${aitab_mnt}" fi diff --git > a/archiso/initcpio/install/archiso_growfs > b/archiso/initcpio/install/archiso_growfs new file mode 100644 index > 0000000..6334a3c --- /dev/null > +++ b/archiso/initcpio/install/archiso_growfs > @@ -0,0 +1,9 @@ > +#!/bin/bash > + > +build() { > + add_binary btrfs > + add_binary resize2fs > + add_binary xfs_growfs > +} > + > +# vim: set ft=sh ts=4 sw=4 et: > -- Gerardo Exequiel Pozzi \cos^2\alpha + \sin^2\alpha = 1
Gerardo Exequiel Pozzi <vmlinuz386@yahoo.com.ar> on Fri, 2013/08/30 19:17:
On 08/30/2013 06:36 PM, Christian Hesse wrote:
Gerardo Exequiel Pozzi <vmlinuz386@yahoo.com.ar> on Fri, 2013/08/30 13:01:
On 08/30/2013 06:12 AM, Christian Hesse wrote:
Just thinking about how to integrate the userspace tools and handling for this. I would like to add a new hook that adds the binary files (resize2fs, xfs_growfs, btrfs), so anybody can decide whether or not to include. But where do we call this?
I did not see any real usage of this "extension" inside official image. If you are building a custom image build, why not just create a big filesystem from the start?
Because bigger filesystem has more metadata. Image size would increase for a use case I need only very seldom. I perfer to add some lines of code, having the possibility to grow device and filesystem when needed.
True, but this can be done from another perspective, just create a big file for filesystem, but use few blocks, since squashfs supports sparse files.
To do this in the most trivial way is just adding one line to mkarchiso. Let says: truncate -s 16G ${_fs_img} after "mkfs". (or even better! truncate -s 2T ${_fs_img} :D)
I should have thought of that before... Great idea! Any chance to get that upstream? I do not want it enabled by default, but with a parameter.
There is only one downside if cowspace should be stored on filesystems that does not support sparse files like FAT cowspace_size= is much more important to be specified now.
I do not have fat32 for that here, so no problem.
This avoid at all this complex logic, what do you think?
As said before... Great idea. ;) Will give it a test now. -- main(a){char*c=/* Schoene Gruesse */"B?IJj;MEH" "CX:;",b;for(a/* Chris get my mail address: */=0;b=c[a++];) putchar(b-1/(/* gcc -o sig sig.c && ./sig */b/42*2-3)*42);}
On 09/01/2013 01:34 PM, Christian Hesse wrote:
Gerardo Exequiel Pozzi <vmlinuz386@yahoo.com.ar> on Fri, 2013/08/30 19:17:
On 08/30/2013 06:36 PM, Christian Hesse wrote:
Gerardo Exequiel Pozzi <vmlinuz386@yahoo.com.ar> on Fri, 2013/08/30 13:01:
On 08/30/2013 06:12 AM, Christian Hesse wrote:
Just thinking about how to integrate the userspace tools and handling for this. I would like to add a new hook that adds the binary files (resize2fs, xfs_growfs, btrfs), so anybody can decide whether or not to include. But where do we call this?
I did not see any real usage of this "extension" inside official image. If you are building a custom image build, why not just create a big filesystem from the start?
Because bigger filesystem has more metadata. Image size would increase for a use case I need only very seldom. I perfer to add some lines of code, having the possibility to grow device and filesystem when needed.
True, but this can be done from another perspective, just create a big file for filesystem, but use few blocks, since squashfs supports sparse files.
To do this in the most trivial way is just adding one line to mkarchiso. Let says: truncate -s 16G ${_fs_img} after "mkfs". (or even better! truncate -s 2T ${_fs_img} :D)
I should have thought of that before... Great idea!
Good :)
Any chance to get that upstream? I do not want it enabled by default, but with a parameter.
Its safe to be enabled by default, without adding additional parameter, but not in fixed size, I guess at most as double of filesystem size should be sufficient. If another parameter should be adeed can be in aitab or via an argument in mkarchiso.
There is only one downside if cowspace should be stored on filesystems that does not support sparse files like FAT cowspace_size= is much more important to be specified now.
I do not have fat32 for that here, so no problem.
This avoid at all this complex logic, what do you think?
As said before... Great idea. ;) Will give it a test now.
Nice :) -- Gerardo Exequiel Pozzi \cos^2\alpha + \sin^2\alpha = 1
Gerardo Exequiel Pozzi <vmlinuz386@yahoo.com.ar> on Sun, 2013/09/01 19:36:
On 09/01/2013 01:34 PM, Christian Hesse wrote:
Gerardo Exequiel Pozzi <vmlinuz386@yahoo.com.ar> on Fri, 2013/08/30 19:17:
On 08/30/2013 06:36 PM, Christian Hesse wrote:
Gerardo Exequiel Pozzi <vmlinuz386@yahoo.com.ar> on Fri, 2013/08/30 13:01:
On 08/30/2013 06:12 AM, Christian Hesse wrote:
Just thinking about how to integrate the userspace tools and handling for this. I would like to add a new hook that adds the binary files (resize2fs, xfs_growfs, btrfs), so anybody can decide whether or not to include. But where do we call this?
I did not see any real usage of this "extension" inside official image. If you are building a custom image build, why not just create a big filesystem from the start?
Because bigger filesystem has more metadata. Image size would increase for a use case I need only very seldom. I perfer to add some lines of code, having the possibility to grow device and filesystem when needed.
True, but this can be done from another perspective, just create a big file for filesystem, but use few blocks, since squashfs supports sparse files.
To do this in the most trivial way is just adding one line to mkarchiso. Let says: truncate -s 16G ${_fs_img} after "mkfs". (or even better! truncate -s 2T ${_fs_img} :D)
I should have thought of that before... Great idea!
Good :)
Any chance to get that upstream? I do not want it enabled by default, but with a parameter.
Its safe to be enabled by default, without adding additional parameter, but not in fixed size, I guess at most as double of filesystem size should be sufficient.
I will reply with a patch.
If another parameter should be adeed can be in aitab or via an argument in mkarchiso.
Will think about that. Do we want code to handle the fs growing? This could be done after chroot as fine.
There is only one downside if cowspace should be stored on filesystems that does not support sparse files like FAT cowspace_size= is much more important to be specified now.
I do not have fat32 for that here, so no problem.
This avoid at all this complex logic, what do you think?
As said before... Great idea. ;) Will give it a test now.
Nice :)
Works just perfectly. ;) -- main(a){char*c=/* Schoene Gruesse */"B?IJj;MEH" "CX:;",b;for(a/* Chris get my mail address: */=0;b=c[a++];) putchar(b-1/(/* gcc -o sig sig.c && ./sig */b/42*2-3)*42);}
From: Christian Hesse <mail@eworm.de> This allows to grow the filesystem after system boot up. We have no additional cost as squashfs handles sparse files. --- archiso/mkarchiso | 2 ++ 1 file changed, 2 insertions(+) diff --git a/archiso/mkarchiso b/archiso/mkarchiso index 8f9ed42..563f624 100755 --- a/archiso/mkarchiso +++ b/archiso/mkarchiso @@ -364,6 +364,8 @@ _mkfs () { cp -aT "${_fs_src}/" "${work_dir}/mnt/${_src}/" _msg_info "Done!" _umount_fs "${work_dir}/mnt/${_src}" + # double size with sparse blocks, will allow to grow the filesystem + truncate -s$((_fs_size*2))M "${_fs_img}" } command_checksum () { -- 1.8.4
Christian Hesse <list@eworm.de> on Mon, 2013/09/02 10:03:
From: Christian Hesse <mail@eworm.de>
This allows to grow the filesystem after system boot up. We have no additional cost as squashfs handles sparse files. --- archiso/mkarchiso | 2 ++ 1 file changed, 2 insertions(+)
diff --git a/archiso/mkarchiso b/archiso/mkarchiso index 8f9ed42..563f624 100755 --- a/archiso/mkarchiso +++ b/archiso/mkarchiso @@ -364,6 +364,8 @@ _mkfs () { cp -aT "${_fs_src}/" "${work_dir}/mnt/${_src}/" _msg_info "Done!" _umount_fs "${work_dir}/mnt/${_src}" + # double size with sparse blocks, will allow to grow the filesystem + truncate -s$((_fs_size*2))M "${_fs_img}" }
command_checksum () {
Currently I am maintaining this in a local package for myself. Any chance to get this merged upstream? -- main(a){char*c=/* Schoene Gruesse */"B?IJj;MEH" "CX:;",b;for(a/* Chris get my mail address: */=0;b=c[a++];) putchar(b-1/(/* gcc -o sig sig.c && ./sig */b/42*2-3)*42);}
On 01/09/2014 05:49 AM, Christian Hesse wrote:
Christian Hesse <list@eworm.de> on Mon, 2013/09/02 10:03:
From: Christian Hesse <mail@eworm.de>
This allows to grow the filesystem after system boot up. We have no additional cost as squashfs handles sparse files. --- archiso/mkarchiso | 2 ++ 1 file changed, 2 insertions(+)
diff --git a/archiso/mkarchiso b/archiso/mkarchiso index 8f9ed42..563f624 100755 --- a/archiso/mkarchiso +++ b/archiso/mkarchiso @@ -364,6 +364,8 @@ _mkfs () { cp -aT "${_fs_src}/" "${work_dir}/mnt/${_src}/" _msg_info "Done!" _umount_fs "${work_dir}/mnt/${_src}" + # double size with sparse blocks, will allow to grow the filesystem + truncate -s$((_fs_size*2))M "${_fs_img}" }
command_checksum () {
Currently I am maintaining this in a local package for myself. Any chance to get this merged upstream?
No. This is a special case. If root-image.fs.sfs is copied to a filesystems that does not support sparse files (FAT commonly for USB-keys), this is waste of free space. -- Gerardo Exequiel Pozzi \cos^2\alpha + \sin^2\alpha = 1
On 01/10/2014 11:47 PM, Gerardo Exequiel Pozzi wrote:
On 01/09/2014 05:49 AM, Christian Hesse wrote:
Christian Hesse <list@eworm.de> on Mon, 2013/09/02 10:03:
From: Christian Hesse <mail@eworm.de>
This allows to grow the filesystem after system boot up. We have no additional cost as squashfs handles sparse files. --- archiso/mkarchiso | 2 ++ 1 file changed, 2 insertions(+)
diff --git a/archiso/mkarchiso b/archiso/mkarchiso index 8f9ed42..563f624 100755 --- a/archiso/mkarchiso +++ b/archiso/mkarchiso @@ -364,6 +364,8 @@ _mkfs () { cp -aT "${_fs_src}/" "${work_dir}/mnt/${_src}/" _msg_info "Done!" _umount_fs "${work_dir}/mnt/${_src}" + # double size with sparse blocks, will allow to grow the filesystem + truncate -s$((_fs_size*2))M "${_fs_img}" }
command_checksum () {
Currently I am maintaining this in a local package for myself. Any chance to get this merged upstream?
No. This is a special case. If root-image.fs.sfs is copied to a filesystems that does not support sparse files (FAT commonly for USB-keys), this is waste of free space.
brb, ignore me!! truncate is on root-image.fs not root-images.fs.sfs :P Yes, this can be safe by default and can be included for next release :) -- Gerardo Exequiel Pozzi \cos^2\alpha + \sin^2\alpha = 1
Gerardo Exequiel Pozzi <vmlinuz386@yahoo.com.ar> on Fri, 2014/01/10 23:53:
On 01/10/2014 11:47 PM, Gerardo Exequiel Pozzi wrote:
On 01/09/2014 05:49 AM, Christian Hesse wrote:
Christian Hesse <list@eworm.de> on Mon, 2013/09/02 10:03:
From: Christian Hesse <mail@eworm.de>
This allows to grow the filesystem after system boot up. We have no additional cost as squashfs handles sparse files. --- archiso/mkarchiso | 2 ++ 1 file changed, 2 insertions(+)
diff --git a/archiso/mkarchiso b/archiso/mkarchiso index 8f9ed42..563f624 100755 --- a/archiso/mkarchiso +++ b/archiso/mkarchiso @@ -364,6 +364,8 @@ _mkfs () { cp -aT "${_fs_src}/" "${work_dir}/mnt/${_src}/" _msg_info "Done!" _umount_fs "${work_dir}/mnt/${_src}" + # double size with sparse blocks, will allow to grow the filesystem + truncate -s$((_fs_size*2))M "${_fs_img}" }
command_checksum () {
Currently I am maintaining this in a local package for myself. Any chance to get this merged upstream?
No. This is a special case. If root-image.fs.sfs is copied to a filesystems that does not support sparse files (FAT commonly for USB-keys), this is waste of free space.
brb, ignore me!! truncate is on root-image.fs not root-images.fs.sfs :P
Yes, this can be safe by default and can be included for next release :)
Great, thanks! -- main(a){char*c=/* Schoene Gruesse */"B?IJj;MEH" "CX:;",b;for(a/* Chris get my mail address: */=0;b=c[a++];) putchar(b-1/(/* gcc -o sig sig.c && ./sig */b/42*2-3)*42);}
On 01/11/2014 08:35 AM, Christian Hesse wrote:
Gerardo Exequiel Pozzi <vmlinuz386@yahoo.com.ar> on Fri, 2014/01/10 23:53:
On 01/10/2014 11:47 PM, Gerardo Exequiel Pozzi wrote:
On 01/09/2014 05:49 AM, Christian Hesse wrote:
Christian Hesse <list@eworm.de> on Mon, 2013/09/02 10:03:
From: Christian Hesse <mail@eworm.de>
This allows to grow the filesystem after system boot up. We have no additional cost as squashfs handles sparse files. --- archiso/mkarchiso | 2 ++ 1 file changed, 2 insertions(+)
diff --git a/archiso/mkarchiso b/archiso/mkarchiso index 8f9ed42..563f624 100755 --- a/archiso/mkarchiso +++ b/archiso/mkarchiso @@ -364,6 +364,8 @@ _mkfs () { cp -aT "${_fs_src}/" "${work_dir}/mnt/${_src}/" _msg_info "Done!" _umount_fs "${work_dir}/mnt/${_src}" + # double size with sparse blocks, will allow to grow the filesystem + truncate -s$((_fs_size*2))M "${_fs_img}" }
command_checksum () {
Currently I am maintaining this in a local package for myself. Any chance to get this merged upstream?
No. This is a special case. If root-image.fs.sfs is copied to a filesystems that does not support sparse files (FAT commonly for USB-keys), this is waste of free space.
brb, ignore me!! truncate is on root-image.fs not root-images.fs.sfs :P
Yes, this can be safe by default and can be included for next release :)
Great, thanks!
I am still not changing this because, I am thinking in a different way to do this, in a more "systemd-friendly-way" (because currently I need to mount the squashfs, then fetch the size in blocks of the file inside it...). This is, for all cases, create a fixed size of the filesystem, (i.e 4G or 8G), the overhead is minimal, really :) $ truncate -s 1G coco.fs $ mkfs.ext4 -q -O ^has_journal -E lazy_itable_init=0 -m 0 -F coco.fs $ du -h coco.fs 408K coco.fs $ truncate -s 16G pepe.fs $ mkfs.ext4 -q -O ^has_journal -E lazy_itable_init=0 -m 0 -F pepe.fs $ du -h pepe.fs 4.3M pepe.fs -- Gerardo Exequiel Pozzi \cos^2\alpha + \sin^2\alpha = 1
Gerardo Exequiel Pozzi <vmlinuz386@yahoo.com.ar> on Wed, 2014/03/19 22:21:
On 01/11/2014 08:35 AM, Christian Hesse wrote:
Gerardo Exequiel Pozzi <vmlinuz386@yahoo.com.ar> on Fri, 2014/01/10 23:53:
On 01/10/2014 11:47 PM, Gerardo Exequiel Pozzi wrote:
On 01/09/2014 05:49 AM, Christian Hesse wrote:
Christian Hesse <list@eworm.de> on Mon, 2013/09/02 10:03:
From: Christian Hesse <mail@eworm.de>
This allows to grow the filesystem after system boot up. We have no additional cost as squashfs handles sparse files. --- archiso/mkarchiso | 2 ++ 1 file changed, 2 insertions(+)
diff --git a/archiso/mkarchiso b/archiso/mkarchiso index 8f9ed42..563f624 100755 --- a/archiso/mkarchiso +++ b/archiso/mkarchiso @@ -364,6 +364,8 @@ _mkfs () { cp -aT "${_fs_src}/" "${work_dir}/mnt/${_src}/" _msg_info "Done!" _umount_fs "${work_dir}/mnt/${_src}" + # double size with sparse blocks, will allow to grow the filesystem + truncate -s$((_fs_size*2))M "${_fs_img}" }
command_checksum () {
Currently I am maintaining this in a local package for myself. Any chance to get this merged upstream?
No. This is a special case. If root-image.fs.sfs is copied to a filesystems that does not support sparse files (FAT commonly for USB-keys), this is waste of free space.
brb, ignore me!! truncate is on root-image.fs not root-images.fs.sfs :P
Yes, this can be safe by default and can be included for next release :)
Great, thanks!
I am still not changing this because, I am thinking in a different way to do this, in a more "systemd-friendly-way" (because currently I need to mount the squashfs, then fetch the size in blocks of the file inside it...). This is, for all cases, create a fixed size of the filesystem, (i.e 4G or 8G), the overhead is minimal, really :)
$ truncate -s 1G coco.fs $ mkfs.ext4 -q -O ^has_journal -E lazy_itable_init=0 -m 0 -F coco.fs $ du -h coco.fs 408K coco.fs $ truncate -s 16G pepe.fs $ mkfs.ext4 -q -O ^has_journal -E lazy_itable_init=0 -m 0 -F pepe.fs $ du -h pepe.fs 4.3M pepe.fs
Not sure if I got you right... With this solution you want the filesystem to always fill the whole file? Where is the size specified? I wonder why you need the filesystem size in a systemd setup. What is it good for? You could create a filesystem that fits the needed size, truncate the file and have a fixed size nevertheless. $ _fs_size=1024 $ truncate -s ${_fs_size}M coco.fs $ mkfs.ext4 -q -O ^has_journal -E lazy_itable_init=0 -m 0 -F coco.fs $ du -h coco.fs 408K coco.fs $ truncate -s 16G coco.fs $ du -h coco.fs 408K coco.fs And optionally later in live environment: $ resize2fs coco.fs $ du -h coco.fs 4.3M coco.fs -- main(a){char*c=/* Schoene Gruesse */"B?IJj;MEH" "CX:;",b;for(a/* Chris get my mail address: */=0;b=c[a++];) putchar(b-1/(/* gcc -o sig sig.c && ./sig */b/42*2-3)*42);}
On 03/20/2014 05:24 AM, Christian Hesse wrote:
Gerardo Exequiel Pozzi <vmlinuz386@yahoo.com.ar> on Wed, 2014/03/19 22:21:
On 01/11/2014 08:35 AM, Christian Hesse wrote:
Gerardo Exequiel Pozzi <vmlinuz386@yahoo.com.ar> on Fri, 2014/01/10 23:53:
On 01/10/2014 11:47 PM, Gerardo Exequiel Pozzi wrote:
On 01/09/2014 05:49 AM, Christian Hesse wrote:
Christian Hesse <list@eworm.de> on Mon, 2013/09/02 10:03: > From: Christian Hesse <mail@eworm.de> > > This allows to grow the filesystem after system boot up. We have no > additional cost as squashfs handles sparse files. > --- > archiso/mkarchiso | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/archiso/mkarchiso b/archiso/mkarchiso > index 8f9ed42..563f624 100755 > --- a/archiso/mkarchiso > +++ b/archiso/mkarchiso > @@ -364,6 +364,8 @@ _mkfs () { > cp -aT "${_fs_src}/" "${work_dir}/mnt/${_src}/" > _msg_info "Done!" > _umount_fs "${work_dir}/mnt/${_src}" > + # double size with sparse blocks, will allow to grow the > filesystem > + truncate -s$((_fs_size*2))M "${_fs_img}" > } > > command_checksum () {
Currently I am maintaining this in a local package for myself. Any chance to get this merged upstream?
No. This is a special case. If root-image.fs.sfs is copied to a filesystems that does not support sparse files (FAT commonly for USB-keys), this is waste of free space.
brb, ignore me!! truncate is on root-image.fs not root-images.fs.sfs :P
Yes, this can be safe by default and can be included for next release :)
Great, thanks!
I am still not changing this because, I am thinking in a different way to do this, in a more "systemd-friendly-way" (because currently I need to mount the squashfs, then fetch the size in blocks of the file inside it...). This is, for all cases, create a fixed size of the filesystem, (i.e 4G or 8G), the overhead is minimal, really :)
$ truncate -s 1G coco.fs $ mkfs.ext4 -q -O ^has_journal -E lazy_itable_init=0 -m 0 -F coco.fs $ du -h coco.fs 408K coco.fs $ truncate -s 16G pepe.fs $ mkfs.ext4 -q -O ^has_journal -E lazy_itable_init=0 -m 0 -F pepe.fs $ du -h pepe.fs 4.3M pepe.fs
Not sure if I got you right... With this solution you want the filesystem to always fill the whole file? Where is the size specified?
Yes, just like now. The diference is, while now the size is specified at aitab file using a percent of free space you want, I am now thinking in always using a know big size for total space, since the basic structure of a filesystem is small. In this way I can write systemd.units with fixed values.
I wonder why you need the filesystem size in a systemd setup. What is it good for?
Because this is what is done in archiso hook in the most basic boot: (if you just copy and paste this in a new "archiso" hook, system will boot fine) mkdir -p /run/archiso/bootmnt mount -r /dev/disk/by-label/ARCH_201403 /run/archiso/bootmnt mkdir -p /run/archiso/cowspace mount -t tmpfs -o size=75%,mode=0755 cowspace /run/archiso/cowspace mkdir -p /run/archiso/cowspace/persistent_ARCH_201403/x86_64 losetup --find --read-only /run/archiso/bootmnt/arch/x86_64/root-image.fs.sfs mkdir -p /run/archiso/sfs/root-image mount -r /dev/loop0 /run/archiso/sfs/root-image losetup --find --read-only /run/archiso/sfs/root-image/root-image.fs #blockdev --getsz /dev/loop1 # -> in my case says: 2080768 and is used below dd of=/run/archiso/cowspace/persistent_ARCH_201403/x86_64/root-image.cow count=0 seek=2080768 losetup --find /run/archiso/cowspace/persistent_ARCH_201403/x86_64/root-image.cow dmsetup create arch_root-image --table "0 2080768 snapshot /dev/loop1 /dev/loop2 N 8" mkdir -p /new_root mount -w /dev/mapper/arch_root-image /new_root
You could create a filesystem that fits the needed size, truncate the file and have a fixed size nevertheless.
$ _fs_size=1024 $ truncate -s ${_fs_size}M coco.fs $ mkfs.ext4 -q -O ^has_journal -E lazy_itable_init=0 -m 0 -F coco.fs $ du -h coco.fs 408K coco.fs $ truncate -s 16G coco.fs $ du -h coco.fs 408K coco.fs
And optionally later in live environment:
$ resize2fs coco.fs $ du -h coco.fs 4.3M coco.fs
As I said before, we can just create big FS without significant overhead. -- Gerardo Exequiel Pozzi \cos^2\alpha + \sin^2\alpha = 1
participants (3)
-
Christian Hesse
-
Florian Pritz
-
Gerardo Exequiel Pozzi