[pacman-dev] Repository management

[Allan McRae]

On 11/05/17 02:54, Dave Reisner wrote:
> On Tue, May 09, 2017 at 10:54:44PM +1000, Allan McRae wrote:
>> Hi all,
>>
>> Every time I attempt to work on repo-add, I find it to be a very
>> difficult endeavour.  Even though it is half the size of makepkg
>> (without even including any of libmakepkg), it is much more convoluted
>> to work on.
>>
>> We also have a weird repository database system.  We have:
>> - .db dbs with package information, signatures and delta information
>> - .files dbs that are the same as .db dbs but additionally include filelists
>>
>> There are two reasons the .files dbs replicate all information in the
>> .db dbs
>>  - .db and .files dbs getting out of sync could cause issues
>>  - a complete database is useful for things like archweb, mostly to
>> avoid the above
>>
>> I would also like to include information on source packages to these
>> databases.  The files information is separate due to wanting our primary
>> database to be small.  Likewise, source package information needs to be
>> separate (the signatures take most of the size in the .db dbs, so adding
>> source package signatures effectively doubles the size).
>>
>> So two points up for discussion:
>>
>>
>> 1) Sync repository layout?  I don't see any point in leaving the tar
>> based format, as reading of sync databases is not a bottleneck.  (The
>> local db format can be a bottleneck, but that is a separate discussion...)
> 
> Isn't this a historical reversal? IIRC, the sync DBs used to be expanded
> onto disk, and we decided to leave them as tarballs to address
> performance/fragmentation concerns.

To be clear, I was saying to stay tar based and not to move to something
else.

>> Do we split the information in .db out of .files and add a .full db with
>> complete information?  Then any .src db could follow suit and just have
>> source package information.  How do we get around the out of sync issue
>> (e.g., a package is removed from .db, but we have an old .files database
>> with it).  Do we add timestamps, and print a warning on -F operations
>> when the two are out of sync?
>>
>>
>> 2) Do we need a better (read "more easily maintainable") tool for
>> handling database generation and updates?  libalpm already can read in
>> information package files, so we could add libalpm/db_write.c with the
>> database creation functions.   Should we unify our repo format with our
>> local database format which we already write?
>>
> 
> I'd urge you not to make this a part of pacman. It's too far off the
> beaten path for most users to make it a part of an already complicated
> tool.
> 

Definitely not part of pacman.  I was suggesting another program with a
libalpm backend.

>>
>> I am looking for ideas here.  Please brainstorm to your hearts content.
> 
> WRT replacing repo-add, I'd suggest we come up with a the use cases we
> want to support, design an interface to meet them, and then come up with
> the implementation. Might be nice to start with the Arch Linux
> repository layout as an example that we'd want to support (pooled
> packages with symlinks into repo dirs).
> 
>> Cheers,
>> Allan
> .
>