[arch-general] broken pipe

[Pascal]

hi Ralph,

thank you for that clarification.
the function works a little faster with them.

file_info(){ echo -n ${1:=/dev/stdin}$'\t'; ( tee < "${1}" >( file
--mime-type -b -e compress -e tar -e elf - >&3; cat >/dev/null ) >( md5sum
>&3 ) >( sha1sum >&3 ) >/dev/null; ) 3>&1 | tr '\n' '\t'; echo; }

pv big.tar > /dev/null
1,71GiO 0:00:00 [5,32GiB/s]
[==============================================================================================================>]
100%

time ( for i in $( seq 24 ); do file_info big.tar ; done )
big.tar    application/x-gtar    53f0d0240e5ddc94266087ec96ebb802236fa0bc
-    6989542fabd98b04086524d1106b7907  -
...
big.tar    application/x-gtar    53f0d0240e5ddc94266087ec96ebb802236fa0bc
-    6989542fabd98b04086524d1106b7907  -

real    3m2,712s
user    0m14,988s
sys    1m13,303s

file_info(){ echo -n ${1:=/dev/stdin}$'\t'; ( trap "" pipe; tee < "${1}" >(
md5sum >&3 ) >( sha1sum >&3 ) | file --mime-type -b -e compress -e tar -e
elf - ) 3>&1 | tr '\n' '\t'; echo; }

pv big.tar > /dev/null
1,71GiO 0:00:00 [5,37GiB/s]
[==============================================================================================================>]
100%

time ( for i in $( seq 24 ); do file_info big.tar ; done )
big.tar    application/x-gtar    53f0d0240e5ddc94266087ec96ebb802236fa0bc
-    6989542fabd98b04086524d1106b7907  -
...
big.tar    application/x-gtar    53f0d0240e5ddc94266087ec96ebb802236fa0bc
-    6989542fabd98b04086524d1106b7907  -

real    2m36,013s
user    0m9,349s
sys    0m50,257s

Le jeu. 19 déc. 2019 à 11:59, Ralph Corderoy <ralph at inputplus.co.uk> a
écrit :

> Hi Pascal,
>
> > file_info(){
> >     echo -n ${1:=/dev/stdin}$'\t'
> >     (
> >         tee < "${1}" \
> >             >( file --mime-type -b -e compress -e tar -e elf - >&3 ) \
> >             >( md5sum >&3 ) \
> >             >( sha1sum >&3 ) \
> >             >/dev/null
> >     ) 3>&1 |
> >     tr '\n' '\t'
> >     echo
> > }
> >
> > it no longer works because the data flow is quickly interrupted by tee
> > which does not consume all the data.
>
> You're missing the reason why.  tee(1) receives a SIGPIPE because it
> writes to a pipe that's closed.  Adding a cat(1) is a waste of CPU, as
> is discarding tee's stdout instead of using it for one of the workers.
>
> Examine these differences.
>
>     $ seq 31415 | wc
>       31415   31415  177384
>     $ seq 31415 | tee >(sed q) >(wc) > >(tr -d 42 | wc); sleep 1
>     1
>       14139   14109   62130
>       12773   12774   65536
>     $ seq 31415 | (trap '' pipe; tee >(sed q) >(wc) > >(tr -d 42 | wc));
> sleep 1
>     1
>       31415   31369  142504
>       31415   31415  177384
>     $
>
> Note the output of the commands can be in any order, and intermingle if
> they're long enough.
>
> tee(1) has -p and --output-error but they're not as specific as stating
> SIGPIPE is expected for just one worker.
>
> --
> Cheers, Ralph.
>