[pacman-dev] [PATCH 2/3] Use OpenSSL MD5 crypto functions if available

[Dan McGee]

On Mon, Sep 6, 2010 at 10:45 AM, Allan McRae <allan at archlinux.org> wrote:
> On 07/09/10 01:15, Dan McGee wrote:
>>
>> On Mon, Sep 6, 2010 at 10:04 AM, Allan McRae<allan at archlinux.org>  wrote:
>>>
>>> On 07/09/10 00:25, Bryce Gibson wrote:
>>>>
>>>> ----- Original message -----
>>>>>
>>>>> On 04/09/10 19:16, Jürgen Hötzel wrote:
>>>>>>
>>>>>> Hi Dan,
>>>>>>
>>>>>> 2010/9/2 Dan McGee<dan at archlinux.org>:
>>>>>>>
>>>>>>> This does not remove the MD5 code from our codebase, but it does
>>>>>>> enable linking against OpenSSL to get their much faster
>>>>>>> implementation if it is available on whatever platform you are
>>>>>>> using. At configure-time, we will default to using it if it is
>>>>>>> available, but this can be easily changed by using the
>>>>>>> `--with-openssl` or `--without-openssl` arguments to configure.
>>>>>>
>>>>>> What about just replacing the current MD5 implementation with the
>>>>>> OpenSSL implementation?
>>>>>>
>>>>>> This would prevent conditional compilation and a direct OpenSSL
>>>>>> dependency in libalpm.
>>>>>
>>>>> Can we do that?   Openssl is BSD code.
>>>>>
>>>>>
>>>>> Anyway, I have concerns...     Think of an openssl upgrade.   pacman is
>>>>> in
>>>>> SyncFirst and it pulls in all its deps.   If that pulls in openssl with
>>>>> a
>>>>> soname bump, things may get interesting.   I have not check, but I do
>>>>> not
>>>>> think --as-needed saves us there.
>>>>>
>>>>> Allan
>>>>>
>>>> The licenses that openssl is released under aren't compatible with the
>>>> gpl
>>>> (according to the fsf) because of the advertising clause... So openssl
>>>> code
>>>> can't really be used like that... (Unfortunately)
>>>
>>> How does the coreutils md5sum speed compare?  That is GPL code so could
>>> be
>>> included directly.  Though I guess it is now GPL3 being a GNU project...
>>
>> Quite a bit slower. Test results (for one platform, please test on
>> whatever you are running on, it is very machine-dependent) and the
>> test harness included below, run using ./md5-speed.sh<bigfile>
>
>
>> . md5-speed.sh /home/arch/pkgcache/i686/nexuiz-data-2.5.2-1-any.pkg.tar.gz
> path: /home/arch/pkgcache/i686/nexuiz-data-2.5.2-1-any.pkg.tar.gz
> md5(/home/arch/pkgcache/i686/nexuiz-data-2.5.2-1-any.pkg.tar.gz) =
> 40a30098649adf29ba79fcd6699d5f67
> Version: i686
> real    0m28.441s
> user    0m25.692s
> sys     0m2.643s
>
> path: /home/arch/pkgcache/i686/nexuiz-data-2.5.2-1-any.pkg.tar.gz
> md5(/home/arch/pkgcache/i686/nexuiz-data-2.5.2-1-any.pkg.tar.gz) =
> 40a30098649adf29ba79fcd6699d5f67
> Version: native
> real    0m28.527s
> user    0m25.455s
> sys     0m2.970s
>
> 40a30098649adf29ba79fcd6699d5f67
> /home/arch/pkgcache/i686/nexuiz-data-2.5.2-1-any.pkg.tar.gz
> Program: md5sum
> real    0m22.553s
> user    0m20.552s
> sys     0m1.923s
>
> MD5(/home/arch/pkgcache/i686/nexuiz-data-2.5.2-1-any.pkg.tar.gz)=
> 40a30098649adf29ba79fcd6699d5f67
> Program: openssl dgst -md5
> real    0m22.436s
> user    0m19.745s
> sys     0m2.650s
>
> So not much difference here.

Interesting, good to have someone else try it at least. I tried it on
three machines and two of the three machines (P4 and Atom, both i686)
were definitely slower with the coreutils stuff than the OpenSSL
version.

>> Keep in mind that "including directly" more than likely is a drawback.
>> I wouldn't be surprised if there are assembly routines involved in
>> both of these codebases which just makes the build process that much
>> more involved and we have to work it into our code. Not that I think
>> these routines change much upstream, but it is one more thing to keep
>> up to date.
>
> Well,  how often do we keep the current md5 code up to date?  Has anyone
> checked the upstream source lately?   It may have gotten faster (and it is
> still GPL2) making the whole thing moot, which it really is anyway...  :P

I checked it before I this work, so every once in a while. :) The
PolarSSL source is also meant to be standalone whereas I'm not sure
with the other MD5 routines, which was convenient.

Our current code really lags when it comes to older processors it
seems, or things that use certain architectures- P4 or Atom processors
notably.

-Dan