[arch-general] broken pipe

[Pascal]

it depends a little bit on the context in which the function will be played
:
- with an SSD disk, the playback of a large file is very fast,
- with a lot of available memory, the file is cached at its first reading
and subsequent readings are almost instantaneous.
if these two conditions are met, indeed, I am not sure that the function
brings a real benefit.
otherwise, there is a real interest in using the function.

the tests below were performed on a large file placed on a USB 3.0 key.

ll -hgG elementaryos-5.1-stable.20191202.iso
-rw-r--r-- 1 1.4G Dec 17 14:22 elementaryos-5.1-stable.20191202.iso

here the large file is kept in cache :

pv elementaryos-5.1-stable.20191202.iso > /dev/null
1.37GiB 0:00:35 [39.4MiB/s] [================================>] 100%
pv elementaryos-5.1-stable.20191202.iso > /dev/null
1.37GiB 0:00:00 [4.51GiB/s] [================================>] 100%

time ( file --mime-type -b -e compress -e tar -e elf
elementaryos-5.1-stable.20191202.iso; md5sum
elementaryos-5.1-stable.20191202.iso; sha1sum
elementaryos-5.1-stable.20191202.iso )
application/octet-stream
d1addd17377aa73700680ff937d7a0f4  elementaryos-5.1-stable.20191202.iso
cbe37d55c44db5bb0ab0ae6f5bf0eb96209bb16f
 elementaryos-5.1-stable.20191202.iso
real 0m4.680s
user 0m4.298s
sys 0m0.318s

file_info(){ echo -n ${1:=/dev/stdin}$'\t'; ( tee < "${1}" >( file
--mime-type -b -e compress -e tar -e elf - >&3; cat >/dev/null ) >( md5sum
>&3 ) >( sha1sum >&3 ) >/dev/null; ) 3>&1 | tr '\n' '\t'; echo; }

time ( file_info elementaryos-5.1-stable.20191202.iso )
elementaryos-5.1-stable.20191202.iso application/octet-stream
cbe37d55c44db5bb0ab0ae6f5bf0eb96209bb16f  -
d1addd17377aa73700680ff937d7a0f4  -
real 0m3.435s
user 0m0.113s
sys 0m1.176s

now the cache is cleared between each access (eg. limited available memory)
:

function flush { sync && sysctl -q vm.drop_caches=3; }

flush; pv elementaryos-5.1-stable.20191202.iso > /dev/null
1.37GiB 0:00:35 [39.4MiB/s] [================================>] 100%
flush; pv elementaryos-5.1-stable.20191202.iso > /dev/null
1.37GiB 0:00:35 [39.4MiB/s] [================================>] 100%

flush; time ( file --mime-type -b -e compress -e tar -e elf
elementaryos-5.1-stable.20191202.iso; flush; md5sum
elementaryos-5.1-stable.20191202.iso; flush; sha1sum
elementaryos-5.1-stable.20191202.iso )
application/octet-stream
d1addd17377aa73700680ff937d7a0f4  elementaryos-5.1-stable.20191202.iso
cbe37d55c44db5bb0ab0ae6f5bf0eb96209bb16f
 elementaryos-5.1-stable.20191202.iso

real 1m11.054s
user 0m8.092s
sys 0m1.441s

flush; time ( file_info elementaryos-5.1-stable.20191202.iso )
elementaryos-5.1-stable.20191202.iso application/octet-stream
cbe37d55c44db5bb0ab0ae6f5bf0eb96209bb16f  -
d1addd17377aa73700680ff937d7a0f4  -

real 0m35.217s
user 0m0.316s
sys 0m2.146s

Le mer. 18 déc. 2019 à 16:42, Andy Pieters <arch-general at andypieters.me.uk>
a écrit :

> On Wed, 18 Dec 2019 at 15:32, Pascal via arch-general
> <arch-general at archlinux.org> wrote:
> >
> > that's awesome, it works !
> > it was so simple with cat taking over and consuming the data until the
> end !
> > (I added a redirect to /dev/null to cat)
> > big thank you.
>
> I'm interested in the amount of effort you put into this. Isn't the
> overhead of the pipes etc going to negate any memory/speed
> improvements you may get from only opening the file once or is there
> another consideration at play here?
>