On 11/05/17 02:54, Dave Reisner wrote:
On Tue, May 09, 2017 at 10:54:44PM +1000, Allan McRae wrote:
Hi all,
Every time I attempt to work on repo-add, I find it to be a very difficult endeavour. Even though it is half the size of makepkg (without even including any of libmakepkg), it is much more convoluted to work on.
We also have a weird repository database system. We have: - .db dbs with package information, signatures and delta information - .files dbs that are the same as .db dbs but additionally include filelists
There are two reasons the .files dbs replicate all information in the .db dbs - .db and .files dbs getting out of sync could cause issues - a complete database is useful for things like archweb, mostly to avoid the above
I would also like to include information on source packages to these databases. The files information is separate due to wanting our primary database to be small. Likewise, source package information needs to be separate (the signatures take most of the size in the .db dbs, so adding source package signatures effectively doubles the size).
So two points up for discussion:
1) Sync repository layout? I don't see any point in leaving the tar based format, as reading of sync databases is not a bottleneck. (The local db format can be a bottleneck, but that is a separate discussion...)
Isn't this a historical reversal? IIRC, the sync DBs used to be expanded onto disk, and we decided to leave them as tarballs to address performance/fragmentation concerns.
To be clear, I was saying to stay tar based and not to move to something else.
Do we split the information in .db out of .files and add a .full db with complete information? Then any .src db could follow suit and just have source package information. How do we get around the out of sync issue (e.g., a package is removed from .db, but we have an old .files database with it). Do we add timestamps, and print a warning on -F operations when the two are out of sync?
2) Do we need a better (read "more easily maintainable") tool for handling database generation and updates? libalpm already can read in information package files, so we could add libalpm/db_write.c with the database creation functions. Should we unify our repo format with our local database format which we already write?
I'd urge you not to make this a part of pacman. It's too far off the beaten path for most users to make it a part of an already complicated tool.
Definitely not part of pacman. I was suggesting another program with a libalpm backend.
I am looking for ideas here. Please brainstorm to your hearts content.
WRT replacing repo-add, I'd suggest we come up with a the use cases we want to support, design an interface to meet them, and then come up with the implementation. Might be nice to start with the Arch Linux repository layout as an example that we'd want to support (pooled packages with symlinks into repo dirs).
Cheers, Allan .