[arch-general] broken pipe

[Pascal]

hi,
https://github.com/patatetom/hashs
and merry Christmas.

Le jeu. 19 déc. 2019 à 23:16, Pascal <patatetom at gmail.com> a écrit :

> hi Ralph,
>
> thank you for that clarification.
> the function works a little faster with them.
>
> file_info(){ echo -n ${1:=/dev/stdin}$'\t'; ( tee < "${1}" >( file
> --mime-type -b -e compress -e tar -e elf - >&3; cat >/dev/null ) >( md5sum
> >&3 ) >( sha1sum >&3 ) >/dev/null; ) 3>&1 | tr '\n' '\t'; echo; }
>
> pv big.tar > /dev/null
> 1,71GiO 0:00:00 [5,32GiB/s]
> [==============================================================================================================>]
> 100%
>
> time ( for i in $( seq 24 ); do file_info big.tar ; done )
> big.tar    application/x-gtar    53f0d0240e5ddc94266087ec96ebb802236fa0bc
> -    6989542fabd98b04086524d1106b7907  -
> ...
> big.tar    application/x-gtar    53f0d0240e5ddc94266087ec96ebb802236fa0bc
> -    6989542fabd98b04086524d1106b7907  -
>
> real    3m2,712s
> user    0m14,988s
> sys    1m13,303s
>
> file_info(){ echo -n ${1:=/dev/stdin}$'\t'; ( trap "" pipe; tee < "${1}"
> >( md5sum >&3 ) >( sha1sum >&3 ) | file --mime-type -b -e compress -e tar
> -e elf - ) 3>&1 | tr '\n' '\t'; echo; }
>
> pv big.tar > /dev/null
> 1,71GiO 0:00:00 [5,37GiB/s]
> [==============================================================================================================>]
> 100%
>
> time ( for i in $( seq 24 ); do file_info big.tar ; done )
> big.tar    application/x-gtar    53f0d0240e5ddc94266087ec96ebb802236fa0bc
> -    6989542fabd98b04086524d1106b7907  -
> ...
> big.tar    application/x-gtar    53f0d0240e5ddc94266087ec96ebb802236fa0bc
> -    6989542fabd98b04086524d1106b7907  -
>
> real    2m36,013s
> user    0m9,349s
> sys    0m50,257s
>
> Le jeu. 19 déc. 2019 à 11:59, Ralph Corderoy <ralph at inputplus.co.uk> a
> écrit :
>
>> Hi Pascal,
>>
>> > file_info(){
>> >     echo -n ${1:=/dev/stdin}$'\t'
>> >     (
>> >         tee < "${1}" \
>> >             >( file --mime-type -b -e compress -e tar -e elf - >&3 ) \
>> >             >( md5sum >&3 ) \
>> >             >( sha1sum >&3 ) \
>> >             >/dev/null
>> >     ) 3>&1 |
>> >     tr '\n' '\t'
>> >     echo
>> > }
>> >
>> > it no longer works because the data flow is quickly interrupted by tee
>> > which does not consume all the data.
>>
>> You're missing the reason why.  tee(1) receives a SIGPIPE because it
>> writes to a pipe that's closed.  Adding a cat(1) is a waste of CPU, as
>> is discarding tee's stdout instead of using it for one of the workers.
>>
>> Examine these differences.
>>
>>     $ seq 31415 | wc
>>       31415   31415  177384
>>     $ seq 31415 | tee >(sed q) >(wc) > >(tr -d 42 | wc); sleep 1
>>     1
>>       14139   14109   62130
>>       12773   12774   65536
>>     $ seq 31415 | (trap '' pipe; tee >(sed q) >(wc) > >(tr -d 42 | wc));
>> sleep 1
>>     1
>>       31415   31369  142504
>>       31415   31415  177384
>>     $
>>
>> Note the output of the commands can be in any order, and intermingle if
>> they're long enough.
>>
>> tee(1) has -p and --output-error but they're not as specific as stating
>> SIGPIPE is expected for just one worker.
>>
>> --
>> Cheers, Ralph.
>>
>