On Wed, 18 Dec 2019 at 15:32, Pascal via arch-general wrote:

that's awesome, it works ! it was so simple with cat taking over and consuming the data until the end ! (I added a redirect to /dev/null to cat) big thank you.

I'm interested in the amount of effort you put into this. Isn't the overhead of the pipes etc going to negate any memory/speed improvements you may get from only opening the file once or is there another consideration at play here?

it depends a little bit on the context in which the function will be played : - with an SSD disk, the playback of a large file is very fast, - with a lot of available memory, the file is cached at its first reading and subsequent readings are almost instantaneous. if these two conditions are met, indeed, I am not sure that the function brings a real benefit. otherwise, there is a real interest in using the function. the tests below were performed on a large file placed on a USB 3.0 key. ll -hgG elementaryos-5.1-stable.20191202.iso -rw-r--r-- 1 1.4G Dec 17 14:22 elementaryos-5.1-stable.20191202.iso here the large file is kept in cache : pv elementaryos-5.1-stable.20191202.iso > /dev/null 1.37GiB 0:00:35 [39.4MiB/s] [================================>] 100% pv elementaryos-5.1-stable.20191202.iso > /dev/null 1.37GiB 0:00:00 [4.51GiB/s] [================================>] 100% time ( file --mime-type -b -e compress -e tar -e elf elementaryos-5.1-stable.20191202.iso; md5sum elementaryos-5.1-stable.20191202.iso; sha1sum elementaryos-5.1-stable.20191202.iso ) application/octet-stream d1addd17377aa73700680ff937d7a0f4 elementaryos-5.1-stable.20191202.iso cbe37d55c44db5bb0ab0ae6f5bf0eb96209bb16f elementaryos-5.1-stable.20191202.iso real 0m4.680s user 0m4.298s sys 0m0.318s file_info(){ echo -n ${1:=/dev/stdin}$'\t'; ( tee < "${1}" >( file --mime-type -b -e compress -e tar -e elf - >&3; cat >/dev/null ) >( md5sum

&3 ) >( sha1sum >&3 ) >/dev/null; ) 3>&1 | tr '\n' '\t'; echo; }

time ( file_info elementaryos-5.1-stable.20191202.iso ) elementaryos-5.1-stable.20191202.iso application/octet-stream cbe37d55c44db5bb0ab0ae6f5bf0eb96209bb16f - d1addd17377aa73700680ff937d7a0f4 - real 0m3.435s user 0m0.113s sys 0m1.176s now the cache is cleared between each access (eg. limited available memory) : function flush { sync && sysctl -q vm.drop_caches=3; } flush; pv elementaryos-5.1-stable.20191202.iso > /dev/null 1.37GiB 0:00:35 [39.4MiB/s] [================================>] 100% flush; pv elementaryos-5.1-stable.20191202.iso > /dev/null 1.37GiB 0:00:35 [39.4MiB/s] [================================>] 100% flush; time ( file --mime-type -b -e compress -e tar -e elf elementaryos-5.1-stable.20191202.iso; flush; md5sum elementaryos-5.1-stable.20191202.iso; flush; sha1sum elementaryos-5.1-stable.20191202.iso ) application/octet-stream d1addd17377aa73700680ff937d7a0f4 elementaryos-5.1-stable.20191202.iso cbe37d55c44db5bb0ab0ae6f5bf0eb96209bb16f elementaryos-5.1-stable.20191202.iso real 1m11.054s user 0m8.092s sys 0m1.441s flush; time ( file_info elementaryos-5.1-stable.20191202.iso ) elementaryos-5.1-stable.20191202.iso application/octet-stream cbe37d55c44db5bb0ab0ae6f5bf0eb96209bb16f - d1addd17377aa73700680ff937d7a0f4 - real 0m35.217s user 0m0.316s sys 0m2.146s Le mer. 18 déc. 2019 à 16:42, Andy Pieters a écrit :

On Wed, 18 Dec 2019 at 15:32, Pascal via arch-general wrote:

...

that's awesome, it works ! it was so simple with cat taking over and consuming the data until the

end !

...

(I added a redirect to /dev/null to cat) big thank you.

I'm interested in the amount of effort you put into this. Isn't the overhead of the pipes etc going to negate any memory/speed improvements you may get from only opening the file once or is there another consideration at play here?

hi Ralph, thank you for that clarification. the function works a little faster with them. file_info(){ echo -n ${1:=/dev/stdin}$'\t'; ( tee < "${1}" >( file --mime-type -b -e compress -e tar -e elf - >&3; cat >/dev/null ) >( md5sum

&3 ) >( sha1sum >&3 ) >/dev/null; ) 3>&1 | tr '\n' '\t'; echo; }

pv big.tar > /dev/null 1,71GiO 0:00:00 [5,32GiB/s] [==============================================================================================================>] 100% time ( for i in $( seq 24 ); do file_info big.tar ; done ) big.tar application/x-gtar 53f0d0240e5ddc94266087ec96ebb802236fa0bc - 6989542fabd98b04086524d1106b7907 - ... big.tar application/x-gtar 53f0d0240e5ddc94266087ec96ebb802236fa0bc - 6989542fabd98b04086524d1106b7907 - real 3m2,712s user 0m14,988s sys 1m13,303s file_info(){ echo -n ${1:=/dev/stdin}$'\t'; ( trap "" pipe; tee < "${1}" >( md5sum >&3 ) >( sha1sum >&3 ) | file --mime-type -b -e compress -e tar -e elf - ) 3>&1 | tr '\n' '\t'; echo; } pv big.tar > /dev/null 1,71GiO 0:00:00 [5,37GiB/s] [==============================================================================================================>] 100% time ( for i in $( seq 24 ); do file_info big.tar ; done ) big.tar application/x-gtar 53f0d0240e5ddc94266087ec96ebb802236fa0bc - 6989542fabd98b04086524d1106b7907 - ... big.tar application/x-gtar 53f0d0240e5ddc94266087ec96ebb802236fa0bc - 6989542fabd98b04086524d1106b7907 - real 2m36,013s user 0m9,349s sys 0m50,257s Le jeu. 19 déc. 2019 à 11:59, Ralph Corderoy a écrit :

Hi Pascal,

...

file_info(){ echo -n ${1:=/dev/stdin}$'\t' ( tee < "${1}" \ >( file --mime-type -b -e compress -e tar -e elf - >&3 ) \ >( md5sum >&3 ) \ >( sha1sum >&3 ) \ >/dev/null ) 3>&1 | tr '\n' '\t' echo }

it no longer works because the data flow is quickly interrupted by tee which does not consume all the data.

You're missing the reason why. tee(1) receives a SIGPIPE because it writes to a pipe that's closed. Adding a cat(1) is a waste of CPU, as is discarding tee's stdout instead of using it for one of the workers.

Examine these differences.

$ seq 31415 | wc 31415 31415 177384 $ seq 31415 | tee >(sed q) >(wc) > >(tr -d 42 | wc); sleep 1 1 14139 14109 62130 12773 12774 65536 $ seq 31415 | (trap '' pipe; tee >(sed q) >(wc) > >(tr -d 42 | wc)); sleep 1 1 31415 31369 142504 31415 31415 177384 $

Note the output of the commands can be in any order, and intermingle if they're long enough.

tee(1) has -p and --output-error but they're not as specific as stating SIGPIPE is expected for just one worker.

-- Cheers, Ralph.