[pacman-dev] [PATCH 2/3] Use OpenSSL MD5 crypto functions if available

[Sebastian Nowicki]

On Tue, Sep 7, 2010 at 12:01 AM, Dan McGee <dpmcgee at gmail.com> wrote:
> On Mon, Sep 6, 2010 at 10:45 AM, Allan McRae <allan at archlinux.org> wrote:
>> On 07/09/10 01:15, Dan McGee wrote:
>>>
>>> On Mon, Sep 6, 2010 at 10:04 AM, Allan McRae<allan at archlinux.org>  wrote:
>>>>
>>>> On 07/09/10 00:25, Bryce Gibson wrote:
>>>>>
>>>>> ----- Original message -----
>>>>>>
>>>>>> On 04/09/10 19:16, Jürgen Hötzel wrote:
>>>>>>>
>>>>>>> Hi Dan,
>>>>>>>
>>>>>>> 2010/9/2 Dan McGee<dan at archlinux.org>:
>>>>>>>>
>>>>>>>> This does not remove the MD5 code from our codebase, but it does
>>>>>>>> enable linking against OpenSSL to get their much faster
>>>>>>>> implementation if it is available on whatever platform you are
>>>>>>>> using. At configure-time, we will default to using it if it is
>>>>>>>> available, but this can be easily changed by using the
>>>>>>>> `--with-openssl` or `--without-openssl` arguments to configure.
>>>>>>>
>>>>>>> What about just replacing the current MD5 implementation with the
>>>>>>> OpenSSL implementation?
>>>>>>>
>>>>>>> This would prevent conditional compilation and a direct OpenSSL
>>>>>>> dependency in libalpm.
>>>>>>
>>>>>> Can we do that?   Openssl is BSD code.
>>>>>>
>>>>>>
>>>>>> Anyway, I have concerns...     Think of an openssl upgrade.   pacman is
>>>>>> in
>>>>>> SyncFirst and it pulls in all its deps.   If that pulls in openssl with
>>>>>> a
>>>>>> soname bump, things may get interesting.   I have not check, but I do
>>>>>> not
>>>>>> think --as-needed saves us there.
>>>>>>
>>>>>> Allan
>>>>>>
>>>>> The licenses that openssl is released under aren't compatible with the
>>>>> gpl
>>>>> (according to the fsf) because of the advertising clause... So openssl
>>>>> code
>>>>> can't really be used like that... (Unfortunately)
>>>>
>>>> How does the coreutils md5sum speed compare?  That is GPL code so could
>>>> be
>>>> included directly.  Though I guess it is now GPL3 being a GNU project...
>>>
>>> Quite a bit slower. Test results (for one platform, please test on
>>> whatever you are running on, it is very machine-dependent) and the
>>> test harness included below, run using ./md5-speed.sh<bigfile>
>>
>>
>>> . md5-speed.sh /home/arch/pkgcache/i686/nexuiz-data-2.5.2-1-any.pkg.tar.gz
>> path: /home/arch/pkgcache/i686/nexuiz-data-2.5.2-1-any.pkg.tar.gz
>> md5(/home/arch/pkgcache/i686/nexuiz-data-2.5.2-1-any.pkg.tar.gz) =
>> 40a30098649adf29ba79fcd6699d5f67
>> Version: i686
>> real    0m28.441s
>> user    0m25.692s
>> sys     0m2.643s
>>
>> path: /home/arch/pkgcache/i686/nexuiz-data-2.5.2-1-any.pkg.tar.gz
>> md5(/home/arch/pkgcache/i686/nexuiz-data-2.5.2-1-any.pkg.tar.gz) =
>> 40a30098649adf29ba79fcd6699d5f67
>> Version: native
>> real    0m28.527s
>> user    0m25.455s
>> sys     0m2.970s
>>
>> 40a30098649adf29ba79fcd6699d5f67
>> /home/arch/pkgcache/i686/nexuiz-data-2.5.2-1-any.pkg.tar.gz
>> Program: md5sum
>> real    0m22.553s
>> user    0m20.552s
>> sys     0m1.923s
>>
>> MD5(/home/arch/pkgcache/i686/nexuiz-data-2.5.2-1-any.pkg.tar.gz)=
>> 40a30098649adf29ba79fcd6699d5f67
>> Program: openssl dgst -md5
>> real    0m22.436s
>> user    0m19.745s
>> sys     0m2.650s
>>
>> So not much difference here.
>
> Interesting, good to have someone else try it at least. I tried it on
> three machines and two of the three machines (P4 and Atom, both i686)
> were definitely slower with the coreutils stuff than the OpenSSL
> version.

Thought it might be interesting to compare a 64bit atom (Atom 330,
dual-core 1.6 GHz). Quite a big improvement with both md5sum and
openssl.

path: ../foo.tar
md5(../foo.tar) = 49c19f21d5719a2508066780a41eee4a
Version: x86-64
real	0m12.196s
user	0m11.449s
sys	0m0.733s

path: ../foo.tar
md5(../foo.tar) = 49c19f21d5719a2508066780a41eee4a
Version: atom
real	0m9.938s
user	0m9.216s
sys	0m0.703s

path: ../foo.tar
md5(../foo.tar) = 49c19f21d5719a2508066780a41eee4a
Version: native
real	0m10.128s
user	0m9.279s
sys	0m0.827s

49c19f21d5719a2508066780a41eee4a  ../foo.tar
Program: md5sum
real	0m5.934s
user	0m4.883s
sys	0m0.640s

MD5(../foo.tar)= 49c19f21d5719a2508066780a41eee4a
Program: openssl dgst -md5
real	0m5.134s
user	0m4.360s
sys	0m0.753s