[pacman-dev] Paralellising integrity checks

[Tavian Barnes]

On 19 February 2011 18:26, Allan McRae <allan at archlinux.org> wrote:
> On 20/02/11 09:22, Tavian Barnes wrote:
>>
>> On 19 February 2011 18:08, Dan McGee<dpmcgee at gmail.com>  wrote:
>>>
>>> On Sat, Feb 19, 2011 at 3:24 PM, Tavian Barnes
>>> <tavianator at tavianator.com>  wrote:
>>>>
>>>> On 19 February 2011 06:28, Nezmer<git at nezmer.info>  wrote:
>>>>>>
>>>>>> You can look at how x264 guys implemented this. Check out
>>>>>> x264_cpu_num_processors() in common/cpu.c in their source tree.
>>>>
>>>> Seems to be basically a more platform-complete version of what I'm
>>>> doing here [1].  But yeah, we'd need something like that if pacman is
>>>> supposed to run on everything unix-based.  I don't imagine we care
>>>> much about Windows compatibility :)
>>>
>>> Take a look at git; it has plenty of code that has been tested for
>>> doing this and it works just fine on Windows.
>>>
>>> -Dan
>>>
>>>
>>
>> You mean git x264?  That's what I was looking at, I just meant that we
>> wouldn't have to include that part in pacman, considering the rest of
>> the code doesn't look like it would build on Windows anyway.
>>
>
> I'm fairly sure he meant git itself.
>
> Allan
>
>

Oh I get it.  Well, the code in both is pretty similar, but git seems
to support HPUX while x264 doesn't.  Also, git just uses the number of
online processors, so "taskset 0x1 git gc" runs N threads at once on 1
core, while x264 uses the process's CPU affinity on Linux and Windows
to behave better in that case.

Anyway, that's not the main point.  Are you guys interested in this
change?  I'm almost done a better version of the patch that adds an
_alpm_for_each_cpu() function (to util.h) which takes a callback
function to call in N threads.

On a related note, I just tried running the test suite after entirely
patching out integrity checks, and there weren't any regressions.
Maybe the test suite should test the handling of corrupt packages?  I
can add a test case myself if you want, once I've figured out how the
test suite works.

-- 
Tavian Barnes