[arch-dev-public] dbscripts pkg pools
Hello all, I've had these patches sitting around for a bit: http://code.phraktured.net/cgit.cgi/dbscripts/log/?h=pkgpools I'd like to see if someone else has the time to test them. I have tested them with the following procedure: * created ./testing/{staging,myrepo,packages,package-cleanup}/ with necessary subdirs * modified config to point to these paths * commented out all the SVN malarky in db-add/db-remove/ftpdir-cleanup/etc * changed get_repos_for_host() to return "myrepo" Testing: * Add a package to myrepo - Package file in packages/os/*/ - Relative symlink in myrepo/os/*/ - In myrepo DB * Remove package from repo 'foo' - Removed from myrepo DB * Run cron-jobs/ftpdir-cleanup - Symlink is removed from myrepo/os/*/ - Package file moved from packages/os/*/ to package-cleanup/ Anyone else want to verify this, or should I push it to master? Aside: Would it be a good idea to add a "svn" function which could: a) handle svn checkout failures (we don't, right now) b) be easily switched out for dry-run style testing
On Wed, 28 Apr 2010 11:56:53 -0500, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
Hello all, I've had these patches sitting around for a bit: http://code.phraktured.net/cgit.cgi/dbscripts/log/?h=pkgpools
I'd like to see if someone else has the time to test them.
I have tested them with the following procedure: * created ./testing/{staging,myrepo,packages,package-cleanup}/ with necessary subdirs * modified config to point to these paths * commented out all the SVN malarky in db-add/db-remove/ftpdir-cleanup/etc * changed get_repos_for_host() to return "myrepo"
Testing: * Add a package to myrepo - Package file in packages/os/*/ - Relative symlink in myrepo/os/*/ - In myrepo DB * Remove package from repo 'foo' - Removed from myrepo DB * Run cron-jobs/ftpdir-cleanup - Symlink is removed from myrepo/os/*/ - Package file moved from packages/os/*/ to package-cleanup/
Anyone else want to verify this, or should I push it to master?
Thanks for your heads-up. That was on my todo list but atm I am not really able to catch up. But those patches are a lot more readable than those from your working branch which virtually changed every line of code. I'll have a closer look at those soon. For now: * Does this also work with the old layout/packages? (especially the cleanup and move scripts) * I still don't know how to handle the community and community-testing repo. We cannot really put those packages into the same folder as core,extra,testing. That could break the cleanup-scripts (different svn) and definitely will break the rsync process from sigurd to gerolde. * Do all packages have the $arch within their filename? (community might be a problem here as they are not using db-scripts for that long) -- Pierre Schmitz, https://users.archlinux.de/~pierre
On Wed, Apr 28, 2010 at 12:21 PM, Pierre Schmitz <pierre@archlinux.de> wrote:
On Wed, 28 Apr 2010 11:56:53 -0500, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
Hello all, I've had these patches sitting around for a bit: http://code.phraktured.net/cgit.cgi/dbscripts/log/?h=pkgpools
I'd like to see if someone else has the time to test them.
I have tested them with the following procedure: * created ./testing/{staging,myrepo,packages,package-cleanup}/ with necessary subdirs * modified config to point to these paths * commented out all the SVN malarky in db-add/db-remove/ftpdir-cleanup/etc * changed get_repos_for_host() to return "myrepo"
Testing: * Add a package to myrepo - Package file in packages/os/*/ - Relative symlink in myrepo/os/*/ - In myrepo DB * Remove package from repo 'foo' - Removed from myrepo DB * Run cron-jobs/ftpdir-cleanup - Symlink is removed from myrepo/os/*/ - Package file moved from packages/os/*/ to package-cleanup/
Anyone else want to verify this, or should I push it to master?
Thanks for your heads-up. That was on my todo list but atm I am not really able to catch up. But those patches are a lot more readable than those from your working branch which virtually changed every line of code. I'll have a closer look at those soon.
Yeah, there's two patches in the working branch that are totally unrelated, so I dumped them for the purposes of these changes.
For now: * Does this also work with the old layout/packages? (especially the cleanup and move scripts)
No. What will have to happen is that all packages will need to be moved to /srv/ftp/packages/os/*/ and symlinked to where they actually end up (with relative symlinks, for the purposes of rsync). This can be scripted away fairly quickly.
* I still don't know how to handle the community and community-testing repo. We cannot really put those packages into the same folder as core,extra,testing. That could break the cleanup-scripts (different svn) and definitely will break the rsync process from sigurd to gerolde.
Hmm, if the new DB scripts are placed on there, it will have it's own /srv/ftp/packages dir and relative symlinks. This is going to be very hard to rsync in. One possibility is to move all of community to a subtree in our ftp layout, i.e. /srv/ftp/repos/other/{community, ...}. This would be a fairly drastic change, but would make this easier.
* Do all packages have the $arch within their filename? (community might be a problem here as they are not using db-scripts for that long)
What is this referencing? Is this related to the ARCHES loop I added?
On 29/04/10 03:42, Aaron Griffin wrote:
On Wed, Apr 28, 2010 at 12:21 PM, Pierre Schmitz<pierre@archlinux.de> wrote: <snip>
* Does this also work with the old layout/packages? (especially the cleanup and move scripts)
No. What will have to happen is that all packages will need to be moved to /srv/ftp/packages/os/*/ and symlinked to where they actually end up (with relative symlinks, for the purposes of rsync). This can be scripted away fairly quickly.
That is going to be a painful initial rsync... Is it just the clean-up script that can not handle the change?
* Do all packages have the $arch within their filename? (community might be a problem here as they are not using db-scripts for that long)
All package in [core] and [extra] have $arch in their filename as of a month or two ago. There is no way community is even close. Allan
On Wed, Apr 28, 2010 at 6:39 PM, Allan McRae <allan@archlinux.org> wrote:
On 29/04/10 03:42, Aaron Griffin wrote:
On Wed, Apr 28, 2010 at 12:21 PM, Pierre Schmitz<pierre@archlinux.de> wrote:
<snip>
* Does this also work with the old layout/packages? (especially the cleanup and move scripts)
No. What will have to happen is that all packages will need to be moved to /srv/ftp/packages/os/*/ and symlinked to where they actually end up (with relative symlinks, for the purposes of rsync). This can be scripted away fairly quickly.
That is going to be a painful initial rsync... Is it just the clean-up script that can not handle the change?
I'm sure we could figure something out... the cleanup script does one thing that is incompatible: it removes (rm) files from each repo if not in the DB - this assumes it is a symlink. It should be fairly easy to replace that with a "if symlink, rm, else mv to cleanup" Can anyone else see anything incompatible here?
On Thu, 29 Apr 2010 12:05:19 -0500, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
That is going to be a painful initial rsync... Is it just the clean-up script that can not handle the change?
I'm sure we could figure something out... the cleanup script does one thing that is incompatible: it removes (rm) files from each repo if not in the DB - this assumes it is a symlink. It should be fairly easy to replace that with a "if symlink, rm, else mv to cleanup"
Can anyone else see anything incompatible here?
I think its really only the cleanup script that is directly affected. db-update doesn't need to know about old packages, db-move and db-remove only work on db files and use mv which should also work with symlinks. I think it's definetly worth to modify the cleanup script to handle both layouts...a complete resync will produce a lot of unnessecary traffic. The issue left is the community repo. -- Pierre Schmitz, https://users.archlinux.de/~pierre
On Sun, May 2, 2010 at 9:53 AM, Pierre Schmitz <pierre@archlinux.de> wrote:
On Thu, 29 Apr 2010 12:05:19 -0500, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
That is going to be a painful initial rsync... Is it just the clean-up script that can not handle the change?
I'm sure we could figure something out... the cleanup script does one thing that is incompatible: it removes (rm) files from each repo if not in the DB - this assumes it is a symlink. It should be fairly easy to replace that with a "if symlink, rm, else mv to cleanup"
Can anyone else see anything incompatible here?
I think its really only the cleanup script that is directly affected. db-update doesn't need to know about old packages, db-move and db-remove only work on db files and use mv which should also work with symlinks.
I think it's definetly worth to modify the cleanup script to handle both layouts...a complete resync will produce a lot of unnessecary traffic.
How's this? http://code.phraktured.net/cgit.cgi/dbscripts/commit/?h=pkgpools&id=1e13a032ac0bef748150c60f3b670e57dad2aee5
The issue left is the community repo.
Ideas?
On Sun, 2 May 2010 10:54:30 -0500, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
How's this?
http://code.phraktured.net/cgit.cgi/dbscripts/commit/?h=pkgpools&id=1e13a032ac0bef748150c60f3b670e57dad2aee5 Looks good to me.
The issue left is the community repo.
Ideas?
Well, if we want to use the same scripts for community and the same layout we have to put them into different root dirs (FTP_BASE) as you already suggested. Maybe we should also cleanup the root dir auf our ftp and put repos into a sub dir. What about a directory structure like this? We could also remove the os subdir and backwards compatibility could be achieved by some symlinks. ftp └── repo ├── arch │ ├── core │ ├── extra │ ├── packages │ └── testing └── community ├── community ├── community-testing └── packages The actual names might not be final, but with this structure we separate our repos from everything else on the ftp and we separate the "official" and community repos from each other. -- Pierre Schmitz, https://users.archlinux.de/~pierre
On 03/05/10 02:10, Pierre Schmitz wrote:
On Sun, 2 May 2010 10:54:30 -0500, Aaron Griffin<aaronmgriffin@gmail.com> wrote:
How's this?
Looks good to me.
The issue left is the community repo.
Ideas?
Well, if we want to use the same scripts for community and the same layout we have to put them into different root dirs (FTP_BASE) as you already suggested. Maybe we should also cleanup the root dir auf our ftp and put repos into a sub dir.
What about a directory structure like this? We could also remove the os subdir and backwards compatibility could be achieved by some symlinks.
ftp └── repo ├── arch │ ├── core │ ├── extra │ ├── packages │ └── testing └── community ├── community ├── community-testing └── packages
The actual names might not be final, but with this structure we separate our repos from everything else on the ftp and we separate the "official" and community repos from each other.
That brings back the whole initial rsync hell we are trying to avoid by not moving everything to the packages directory. Allan
On Mon, 03 May 2010 02:16:44 +1000, Allan McRae <allan@archlinux.org> wrote:
That brings back the whole initial rsync hell we are trying to avoid by not moving everything to the packages directory.
Yes, as I wrote in my other mail. This would be a solution to another problem: separating the "arch" and community repo. My proposed layout has its advantages but the transition is really ugly. -- Pierre Schmitz, https://users.archlinux.de/~pierre
On 03/05/10 02:20, Pierre Schmitz wrote:
On Mon, 03 May 2010 02:16:44 +1000, Allan McRae<allan@archlinux.org> wrote:
That brings back the whole initial rsync hell we are trying to avoid by not moving everything to the packages directory.
Yes, as I wrote in my other mail. This would be a solution to another problem: separating the "arch" and community repo. My proposed layout has its advantages but the transition is really ugly.
If we are prepared to accept an ugly transition (and a several day rsync), then I like the layout that you propose.
On Sun, 02 May 2010 18:10:40 +0200, Pierre Schmitz <pierre@archlinux.de> wrote:
What about a directory structure like this?
This change would be quite extreme though and it would be probably easier to really issue a complete resync if we actually do that. The "compatibility symlinks" would just be there to not break "old" mirrorlists. But in this case it might be worth it. -- Pierre Schmitz, https://users.archlinux.de/~pierre
On 05/02/2010 07:10 PM, Pierre Schmitz wrote:
On Sun, 2 May 2010 10:54:30 -0500, Aaron Griffin<aaronmgriffin@gmail.com> wrote:
How's this?
Looks good to me.
The issue left is the community repo.
Ideas?
Well, if we want to use the same scripts for community and the same layout we have to put them into different root dirs (FTP_BASE) as you already suggested. Maybe we should also cleanup the root dir auf our ftp and put repos into a sub dir.
What about a directory structure like this? We could also remove the os subdir and backwards compatibility could be achieved by some symlinks.
ftp └── repo ├── arch │ ├── core │ ├── extra │ ├── packages │ └── testing └── community ├── community ├── community-testing └── packages
The actual names might not be final, but with this structure we separate our repos from everything else on the ftp and we separate the "official" and community repos from each other.
if i understand well using this schema we can't have a generic server link and only one mirrorlist file? -- Ionut
On Sun, 02 May 2010 18:10:40 +0200, Pierre Schmitz <pierre@archlinux.de> wrote:
What about a directory structure like this? We could also remove the os subdir and backwards compatibility could be achieved by some symlinks.
ftp └── repo ├── arch │ ├── core │ ├── extra │ ├── packages │ └── testing └── community ├── community ├── community-testing └── packages
The actual names might not be final, but with this structure we separate our repos from everything else on the ftp and we separate the "official" and community repos from each other.
Here is a proposal which is more backwards-compatible and doesn't need a complete resync; just a few modificatinos to the db-scripts. (e.g. we need to define which repos the cleanup-script should check) ftp ├── core ├── extra ├── testing ├── community ├── community-testing └── packages ├── arch │ ├── i686 │ ├── x86_64 │ └── any └── community ├── i686 ├── x86_64 └── any The top dir repo dirs are kept. They include symlinks to the packages in the packages dir. Packages from community are kept separate. We now just need to modify the db-scripts and define the repository names it should work on. (especialy the cleanup-script should only search in core,extra and testing but not community for example. The community, community-testing and packages/community dirs would still be rsynced from sigurd. What do you think about this layout? It wont need any chagne on the clients (mirrorlists still work) and no resyncs are needed. Greetings, Pierre -- Pierre Schmitz, https://users.archlinux.de/~pierre
On Mon, May 3, 2010 at 5:59 AM, Pierre Schmitz <pierre@archlinux.de> wrote:
On Sun, 02 May 2010 18:10:40 +0200, Pierre Schmitz <pierre@archlinux.de> wrote:
What about a directory structure like this? We could also remove the os subdir and backwards compatibility could be achieved by some symlinks.
ftp └── repo ├── arch │ ├── core │ ├── extra │ ├── packages │ └── testing └── community ├── community ├── community-testing └── packages
The actual names might not be final, but with this structure we separate our repos from everything else on the ftp and we separate the "official" and community repos from each other.
Here is a proposal which is more backwards-compatible and doesn't need a complete resync; just a few modificatinos to the db-scripts. (e.g. we need to define which repos the cleanup-script should check)
ftp ├── core ├── extra ├── testing ├── community ├── community-testing └── packages ├── arch │ ├── i686 │ ├── x86_64 │ └── any └── community ├── i686 ├── x86_64 └── any
The top dir repo dirs are kept. They include symlinks to the packages in the packages dir. Packages from community are kept separate. We now just need to modify the db-scripts and define the repository names it should work on. (especialy the cleanup-script should only search in core,extra and testing but not community for example.
The community, community-testing and packages/community dirs would still be rsynced from sigurd.
What do you think about this layout? It wont need any chagne on the clients (mirrorlists still work) and no resyncs are needed.
In the spirit of this, I broke out the "packages" dir to a variable so that we can change it: http://code.phraktured.net/cgit.cgi/dbscripts/commit/?h=pkgpools&id=ee4074de910df5c637691ace06ec5a9d475fb4a3 On gerolde, it'd be packages/arch and on sigurd, packages/community (in theory)
On Mon, May 3, 2010 at 11:19 AM, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
On Mon, May 3, 2010 at 5:59 AM, Pierre Schmitz <pierre@archlinux.de> wrote:
On Sun, 02 May 2010 18:10:40 +0200, Pierre Schmitz <pierre@archlinux.de> wrote:
What about a directory structure like this? We could also remove the os subdir and backwards compatibility could be achieved by some symlinks.
ftp └── repo ├── arch │ ├── core │ ├── extra │ ├── packages │ └── testing └── community ├── community ├── community-testing └── packages
The actual names might not be final, but with this structure we separate our repos from everything else on the ftp and we separate the "official" and community repos from each other.
Here is a proposal which is more backwards-compatible and doesn't need a complete resync; just a few modificatinos to the db-scripts. (e.g. we need to define which repos the cleanup-script should check)
ftp ├── core ├── extra ├── testing ├── community ├── community-testing └── packages ├── arch │ ├── i686 │ ├── x86_64 │ └── any └── community ├── i686 ├── x86_64 └── any
The top dir repo dirs are kept. They include symlinks to the packages in the packages dir. Packages from community are kept separate. We now just need to modify the db-scripts and define the repository names it should work on. (especialy the cleanup-script should only search in core,extra and testing but not community for example.
The community, community-testing and packages/community dirs would still be rsynced from sigurd.
What do you think about this layout? It wont need any chagne on the clients (mirrorlists still work) and no resyncs are needed.
In the spirit of this, I broke out the "packages" dir to a variable so that we can change it: http://code.phraktured.net/cgit.cgi/dbscripts/commit/?h=pkgpools&id=ee4074de910df5c637691ace06ec5a9d475fb4a3
On gerolde, it'd be packages/arch and on sigurd, packages/community (in theory)
Though... we'd still need to get rid of that silly "os" dir as part of this.
On Mon, May 3, 2010 at 12:19, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
Though... we'd still need to get rid of that silly "os" dir as part of this.
The rankmirrors script will also need to be updated for that I believe.
On Mon, 3 May 2010 11:19:35 -0500, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
In the spirit of this, I broke out the "packages" dir to a variable so that we can change it:
On gerolde, it'd be packages/arch and on sigurd, packages/community (in theory)
Yes, that should be fine this way. Could you merge this into master and pull to gerolde? Two things I wanted to chagne: * add a localconfig file which optionally overrides the config options (this way we can still manage the main configs in git, but adjust settings for community etc.) * we need to define a list of repos the cleanup-script (especially this find call) should look into.
Though... we'd still need to get rid of that silly "os" dir as part of this.
I don't think it's worth to have a complete resync and adjusting client's configs just to remove this dir. It doesn't hurt that much and who knows for what reason it was created. So, I would prefer a way with least breakage. -- Pierre Schmitz, https://users.archlinux.de/~pierre
On Tue, May 4, 2010 at 3:26 AM, Pierre Schmitz <pierre@archlinux.de> wrote:
On Mon, 3 May 2010 11:19:35 -0500, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
In the spirit of this, I broke out the "packages" dir to a variable so that we can change it:
On gerolde, it'd be packages/arch and on sigurd, packages/community (in theory)
Yes, that should be fine this way. Could you merge this into master and pull to gerolde? Two things I wanted to chagne: * add a localconfig file which optionally overrides the config options (this way we can still manage the main configs in git, but adjust settings for community etc.)
What's wrong with just changing the config file for community? That was the intent. If you modify it, and commit it locally, everything will work out fine if you "git pull --rebase". Adding a localconfig just seems like extra complexity for no real reason.
* we need to define a list of repos the cleanup-script (especially this find call) should look into.
Can we use the get_repos_for_host for this?
On Tue, 4 May 2010 06:57:12 -0500, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
* add a localconfig file which optionally overrides the config options (this way we can still manage the main configs in git, but adjust settings for community etc.)
What's wrong with just changing the config file for community? That was the intent. If you modify it, and commit it locally, everything will work out fine if you "git pull --rebase". Adding a localconfig just seems like extra complexity for no real reason.
You are right; let's do it that way.
* we need to define a list of repos the cleanup-script (especially this find call) should look into.
Can we use the get_repos_for_host for this?
Yes, good catch. Forgot about that function. Just loop through the repo dirs and execute the function added by this commit: http://code.phraktured.net/cgit.cgi/dbscripts/commit/?h=pkgpools&id=22c5f235fa25ea3327d95a28c6dc8ae399d6cb9a Btw: I cannot checkout your repo "warning: remote HEAD refers to nonexistent ref, unable to checkout." Could you push it to gerolde/master? I think we are very close to get this stuff ready for production. Greetings, Pierre -- Pierre Schmitz, https://users.archlinux.de/~pierre
On Fri, 07 May 2010 14:35:55 +0200, Pierre Schmitz <pierre@archlinux.de> wrote:
Btw: I cannot checkout your repo "warning: remote HEAD refers to nonexistent ref, unable to checkout." Could you push it to gerolde/master? I think we are very close to get this stuff ready for production.
bump. -- Pierre Schmitz, https://users.archlinux.de/~pierre
On Fri, May 28, 2010 at 5:13 AM, Pierre Schmitz <pierre@archlinux.de> wrote:
On Fri, 07 May 2010 14:35:55 +0200, Pierre Schmitz <pierre@archlinux.de> wrote:
Btw: I cannot checkout your repo "warning: remote HEAD refers to nonexistent ref, unable to checkout." Could you push it to gerolde/master? I think we are very close to get this stuff ready for production.
bump.
Pushed
On Fri, 28 May 2010 13:36:04 -0500, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
On Fri, May 28, 2010 at 5:13 AM, Pierre Schmitz <pierre@archlinux.de> wrote:
On Fri, 07 May 2010 14:35:55 +0200, Pierre Schmitz <pierre@archlinux.de> wrote:
Btw: I cannot checkout your repo "warning: remote HEAD refers to nonexistent ref, unable to checkout." Could you push it to gerolde/master? I think we are very close to get this stuff ready for production.
bump.
Pushed
Thanks. I made the adjustments. Would be nice if someone would have a quick look at my changes: http://projects.archlinux.org/dbscripts.git/log/ What I just noticed: We had the ftp-arch group for those devs who are able to add packages to the core repo. This distinction wont be really doable with a package pool. So we might want to remove that completely. What is missing? 1) Do a little test locally (help is more than welcome here) 2) Someone with root access on gerolde/sigurd: gerolde: install -d -o ftp -g ftp-extra /srv/ftp/packages/arch/{i686,x86_64,any} rsync the packages/community dir from sigurd git pull new db-scripts sigurd: install -d -o root -g tusers /srv/ftp/packages/community/{i686,x86_64,any} git pull new db-scripts 3) Do we need to adjust our rsync modules for our mirrors? -- Pierre Schmitz, https://users.archlinux.de/~pierre
On Sun, Jun 20, 2010 at 11:57 AM, Pierre Schmitz <pierre@archlinux.de> wrote:
1) Do a little test locally (help is more than welcome here)
Testing the db-scripts is always tedious. Any ideas to speed up testing would help here.
2) Someone with root access on gerolde/sigurd: gerolde: install -d -o ftp -g ftp-extra /srv/ftp/packages/arch/{i686,x86_64,any} rsync the packages/community dir from sigurd git pull new db-scripts sigurd: install -d -o root -g tusers /srv/ftp/packages/community/{i686,x86_64,any} git pull new db-scripts
Seems right.
3) Do we need to adjust our rsync modules for our mirrors?
Yeah, I think so - the package pool dirs need to be in there, or else the symlinks will be broken.
On Mon, 21 Jun 2010 13:31:01 -0500, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
On Sun, Jun 20, 2010 at 11:57 AM, Pierre Schmitz <pierre@archlinux.de> wrote:
1) Do a little test locally (help is more than welcome here)
Testing the db-scripts is always tedious. Any ideas to speed up testing would help here.
I did thought about some kind of test suite, but it's probably way to complicated. So I think I'll do some simple tests locally and then at least use the dry-run mode for the cleanup script for a while when we start using this scripts in production.
2) Someone with root access on gerolde/sigurd: gerolde: install -d -o ftp -g ftp-extra /srv/ftp/packages/arch/{i686,x86_64,any} rsync the packages/community dir from sigurd git pull new db-scripts sigurd: install -d -o root -g tusers /srv/ftp/packages/community/{i686,x86_64,any} git pull new db-scripts
Seems right.
3) Do we need to adjust our rsync modules for our mirrors?
Yeah, I think so - the package pool dirs need to be in there, or else the symlinks will be broken.
I think creating the needed dirs and adjusting the rsync module/job can be done right now. (It shouldn't harm if those dirs aren't used atm) -- Pierre Schmitz, https://users.archlinux.de/~pierre
On 2010-05-03, Pierre Schmitz <pierre@archlinux.de> wrote:
On Sun, 02 May 2010 18:10:40 +0200, Pierre Schmitz <pierre@archlinux.de> wrote:
What about a directory structure like this? We could also remove the os subdir and backwards compatibility could be achieved by some symlinks.
ftp └── repo ├── arch │ ├── core │ ├── extra │ ├── packages │ └── testing └── community ├── community ├── community-testing └── packages
The actual names might not be final, but with this structure we separate our repos from everything else on the ftp and we separate the "official" and community repos from each other.
I like this.
Here is a proposal which is more backwards-compatible and doesn't need a complete resync; just a few modificatinos to the db-scripts. (e.g. we need to define which repos the cleanup-script should check)
ftp ├── core ├── extra ├── testing ├── community ├── community-testing └── packages ├── arch │ ├── i686 │ ├── x86_64 │ └── any └── community ├── i686 ├── x86_64 └── any
The top dir repo dirs are kept. They include symlinks to the packages in the packages dir. Packages from community are kept separate. We now just need to modify the db-scripts and define the repository names it should work on. (especialy the cleanup-script should only search in core,extra and testing but not community for example.
The community, community-testing and packages/community dirs would still be rsynced from sigurd.
What do you think about this layout? It wont need any chagne on the clients (mirrorlists still work) and no resyncs are needed.
I fail to see why resyncs are not needed in this situation. Let's suppose we have /foo/bar on a downstream mirror, and upstream has /foo/bar -> /baz/bar will rsync guess that it should not download bar and instead just copy it from foo to baz locally? -- Roman Kyrylych (Роман Кирилич)
On Tue, 4 May 2010 12:39:33 +0300, Roman Kyrylych <roman.kyrylych@gmail.com> wrote:
I fail to see why resyncs are not needed in this situation. Let's suppose we have /foo/bar on a downstream mirror, and upstream has /foo/bar -> /baz/bar will rsync guess that it should not download bar and instead just copy it from foo to baz locally?
It's the other way round we keep the old dirs; so packages will be replaced by symlinks to the packages dir when they got updated. -- Pierre Schmitz, https://users.archlinux.de/~pierre
On Sun, May 2, 2010 at 19:10, Pierre Schmitz <pierre@archlinux.de> wrote:
What about a directory structure like this? We could also remove the os subdir and backwards compatibility could be achieved by some symlinks.
ftp └── repo ├── arch │ ├── core │ ├── extra │ ├── packages │ └── testing └── community ├── community ├── community-testing └── packages
The actual names might not be final, but with this structure we separate our repos from everything else on the ftp and we separate the "official" and community repos from each other.
Regardless of the internal structure of arch and community trees, do we _want_ to separate them? If it's just because of cleanup scripts, access rights etc., then it can be solved this way: arch - packages from gerolde go there pkg - packages are written/removed by db-scripts here pkg1.i686.tar.xz pkg2.any.tar.xz pkg3.x86_64.tar.xz repo - db files and symlinks are written/removed by db-scripts here testing i686 testing.db.tar.gz pkg1.i686.tar.xz -> ../../../pkg/pkg1.i686.tar.xz pkg2.any.tar.xz -> ../../../pkg/pkg2.any.tar.xz x86_64 core extra community - packages from sigurd go there pkg repo community-testing community ftp - this is what users will see iso pkg - union mount of arch/pkg and community/pkg repo - union mount of arch/repo and community/repo The magic: mount -t aufs -o br=/arch/pkg=ro+wh:/community/pkg=ro+wh none /ftp/pkg mount -t aufs -o br=/arch/repo=ro+wh:/community/repo=ro+wh none /ftp/repo Once union mounts hit the upstream it will be even easier: mount /arch/pkg /ftp/pkg mount --union /community/pkg /ftp/pkg mount /arch/repo /ftp/pkg mount --union /community/repo /ftp/pkg Pros/cons of this particular scheme: * no arch/community separation from the user point of view * no os prefix * one huge pool of packages * packages can be moved even between official and community repos without the need for mirrors/users to redownload them * all packages must have arch suffix * a big initial resync is needed Opinions? P.S.: this is just an idea, I'm not going to argue about it, so if you don't like it - just ignore it. -- Roman Kyrylych (Роман Кирилич)
On Tue, 4 May 2010 18:16:25 +0300, Roman Kyrylych <roman.kyrylych@gmail.com> wrote:
Opinions?
P.S.: this is just an idea, I'm not going to argue about it, so if you don't like it - just ignore it.
But I'll add some points why I don't like it that much though: ;-) * if I got it right this will also require a complete resync * it's way to complicated * aufs is slow and crashes the kernel the hard way on heavy load. (that's why I removed its usage from makechrootpkg) The overlay magic might look nice in theory but the risk of a major mess is too high imho. -- Pierre Schmitz, https://users.archlinux.de/~pierre
On Thu, 29 Apr 2010 09:39:52 +1000, Allan McRae <allan@archlinux.org> wrote:
* Do all packages have the $arch within their filename? (community might be a problem here as they are not using db-scripts for that long)
All package in [core] and [extra] have $arch in their filename as of a month or two ago. There is no way community is even close.
Forget about this. I thought al packages were in the /packages dir, but Aaron created subdirs for each arch. So no problem here. -- Pierre Schmitz, https://users.archlinux.de/~pierre
On 04/29/2010 02:39 AM, Allan McRae wrote:
On 29/04/10 03:42, Aaron Griffin wrote:
On Wed, Apr 28, 2010 at 12:21 PM, Pierre Schmitz<pierre@archlinux.de> wrote: <snip>
* Does this also work with the old layout/packages? (especially the cleanup and move scripts)
No. What will have to happen is that all packages will need to be moved to /srv/ftp/packages/os/*/ and symlinked to where they actually end up (with relative symlinks, for the purposes of rsync). This can be scripted away fairly quickly.
That is going to be a painful initial rsync... Is it just the clean-up script that can not handle the change?
* Do all packages have the $arch within their filename? (community might be a problem here as they are not using db-scripts for that long)
All package in [core] and [extra] have $arch in their filename as of a month or two ago. There is no way community is even close.
Allan
that is not a problem. we can always rebuild all of them. the one that can't be rebuilt we move it into aur. -- Ionut
On 03/05/10 00:46, Ionut Biru wrote:
On 04/29/2010 02:39 AM, Allan McRae wrote:
On 29/04/10 03:42, Aaron Griffin wrote:
On Wed, Apr 28, 2010 at 12:21 PM, Pierre Schmitz<pierre@archlinux.de> wrote: <snip>
* Does this also work with the old layout/packages? (especially the cleanup and move scripts)
No. What will have to happen is that all packages will need to be moved to /srv/ftp/packages/os/*/ and symlinked to where they actually end up (with relative symlinks, for the purposes of rsync). This can be scripted away fairly quickly.
That is going to be a painful initial rsync... Is it just the clean-up script that can not handle the change?
* Do all packages have the $arch within their filename? (community might be a problem here as they are not using db-scripts for that long)
All package in [core] and [extra] have $arch in their filename as of a month or two ago. There is no way community is even close.
Allan
that is not a problem. we can always rebuild all of them. the one that can't be rebuilt we move it into aur.
You are a bit keen.... it is over half the community repo (957/1874) that would need rebuilt. If you really want to do some mindless rebuilding, see http://wiki.archlinux.org/index.php/DeveloperWiki:Unimportant_Rebuild_List . Allan
participants (6)
-
Aaron Griffin
-
Allan McRae
-
Daenyth Blank
-
Ionut Biru
-
Pierre Schmitz
-
Roman Kyrylych