[arch-dev-public] noarch packages vs FTP & DB

2007/11/28, Aaron Griffin :

On Nov 27, 2007 4:56 PM, Jason Chu wrote:

...

...

...

* Architecture Independent Repos

More correct: Architecture Independent Packages They were intended to be placed in the same repos as i686/x86_64 packages, and in the same db.tar.gz They can be placed in $repo/os/any/ dir on FTP servers though (to save the space and traffic for mirrors) with $repo/os/{i686,x86_64}/$pkgname-$pkgver-$pkgrel-any.pkg.tar.gz being a symlink to ../any/$pkgname-$pkgver-$pkgrel-any.pkg.tar.gz (this is because packages should be in the same dir as .db.tar.gz for pacman to download them).

Are you sure about that? I thought they were going to be separate repos that would be usable in x86_64 and i686. That way we'd save bandwidth (upload and download) and storage space *and* save time building packages.

I could go either way here - I don't have a strong opinion, but jason has a point about size (bandwidth, disk space, etc).

They cannot be separate _repos_, because the only way to get this clearly would be to create [core-any], [extra-any] etc, which is bad. They can/should/will be in _separate_FTP_dir_ though (which is not the same as a separate repo) - in $ftpurl/$repo/os/any, but here are some issues (one of which I didn't notice when I first proposed noarch packages): How pacman downloads packages from a repo: for example, on i686 system this is in pacman.conf: [somerepo] Server=ftp://someurl/somerepo/os/i686/ Pacman searches for somerepo.db.tar.gz there and for every package listed in that db it tries to download a package _from_the_same_dir_. Solution #1: 1) noarch packages are mentioned in both i686 & x86_64 .db.tar.gz 2) just create a symlink from $repo/os/{i686,x86_64}/$pkgname-$pkgver-$pkgrel-any.pkg.tar.gz to ../any/$pkgname-$pkgver-$pkgrel-any.pkg.tar.gz so noarch packages will be stored only in one place on ftp - $repo/os/any, but will be downloaded in the same way as other (arch-specific) packages for user's architecture. Note that there is no $repo.db.tar.gz in $ftpurl/os/any. 3) mirrors should sync with symlinks (I believe most of them already do this after the current->core move). 4) if someone wants to create a mirror for i686 packages only (for example) - he/she can simplyfy it by using rsync's option to store symlinked file in the place of symlink. So user will get *-{any,i686}.pkg.tar.gz files mixed in a single ~/mirror-for-my-i686-box/$repo dir. This doesn't require any change to pacman or pacman.conf, just ensuring that mirror is getting the $repo/os/any dir too (which is not a problem since most mirrors do a full FTP rsync). Solution #2: 1) Pacman 3.1 already has only one /etc/pacman.d/mirrorlist file for all repos with contents like: Server = ftp://ftp.archlinux.org/$repo/os/i686 and in pacman.conf there is: [core] Include=/etc/pacman.d/mirrorlist Pacman replaces '$repo' with 'core' automatically during a sync operation. 2) Add a similar support for $arch: Server = ftp://ftp.archlinux.org/$repo/os/$arch On i686 machine pacman should replace '$arch' with 'i686' and then with 'any'. This way we will have 3 different .db.tar.gz for ftp dir, including $repo/os/any/$repo.db.tar.gz And we there will be no symlinks in $repo/os/{i686,x86_64} dir. 3) For cross-distro usage it's better not to hardcode $arch=($CARCH 'any') but make it cofigurable (at least at build time). 4) Modify pacman's output so it produces a single progress meter for a repo, not 2 for each 'i686' and 'any' db file. 5) extract 'i686' and 'any' dbs to a single /var/lib/pacman/sync/$repo but then it should guess from which ftp dir to download package (taking -prefix into account again? or store arch info in db?) The difference between 2 solutions is: #1 - still one db per repo per arch, still simple repo-syncing and package downloading (no modifications in pacman & pacman.conf needed) FTP structure: $ftpurl/$repo/os/i686/ : $repo.db.tar.gz $pkgname-$pkgver-$pkgrel-i686.pkg.tar.gz $pkgname-$pkgver-$pkgrel-any.pkg.tar.gz -> ../any/$pkgname-$pkgver-$pkgrel-any.pkg.tar.gz $ftpurl/$repo/os/x86_64/ : $repo.db.tar.gz $pkgname-$pkgver-$pkgrel-x86_64.pkg.tar.gz $pkgname-$pkgver-$pkgrel-any.pkg.tar.gz -> ../any/$pkgname-$pkgver-$pkgrel-any.pkg.tar.gz $ftpurl/$repo/os/any/ : $pkgname-$pkgver-$pkgrel-any.pkg.tar.gz #2 - 2 dbs per repo per arch, pacman has to download/unpack/merge 2 dbs and download packages from 2 dirs. FTP structure: $ftpurl/$repo/os/i686/ : $repo.db.tar.gz $pkgname-$pkgver-$pkgrel-i686.pkg.tar.gz $ftpurl/$repo/os/x86_64/ : $repo.db.tar.gz $pkgname-$pkgver-$pkgrel-x86_64.pkg.tar.gz $ftpurl/$repo/os/any/ : $repo.db.tar.gz $pkgname-$pkgver-$pkgrel-any.pkg.tar.gz While FTP structure in #2 seems cleaner - the implementation is complex and brings more pain than goodness. So I strongly prefer #1 and am going implement it in our db scripts and devtools/aurtools as soon as time permits. -- Roman Kyrylych (Роман Кирилич)