[pacman-dev] PATHNAME in sync/*/desc

older
[pacman-dev] [PATCH] Don't ask for...

Mark Constable

11 Mar 2008 11 Mar '08

6:52 a.m.

I'm just checking if this could be a reasonable idea and if so I'd look into providing a patch. Would it be feasible to add a %PATHNAME% variable to the sync desc file ? For instance in this case, fetching "any" arch packages would not need to symlinked into the same directory to be found... %FILENAME% hotstuff-0.9.2-1-any.pkg.tar.bz2 %PATHNAME% ../any/ It may provide repo maintainers with some flexibility (hard to test without it being possible) and perhaps %PATHNAME% could also be a URL and that might also be interesting, but the simple relative path prefix would be useful. Any thoughts? --markc

Show replies by date

Roman Kyrylych

11 Mar 11 Mar

1:25 p.m.

2008/3/11, Mark Constable :

...

I'm just checking if this could be a reasonable idea and if so I'd look into providing a patch. Would it be feasible to add a %PATHNAME% variable to the sync desc file ?

For instance in this case, fetching "any" arch packages would not need to symlinked into the same directory to be found...

%FILENAME% hotstuff-0.9.2-1-any.pkg.tar.bz2

%PATHNAME% ../any/

It may provide repo maintainers with some flexibility (hard to test without it being possible) and perhaps %PATHNAME% could also be a URL and that might also be interesting, but the simple relative path prefix would be useful.

Any thoughts?

Hm, so basically, this adds a possibility to have packages in different directory than .db.tar.gz, or even multiple directories. Seems useful for me, and as you've said - repo layout for 'any' arch will become cleaner. -- Roman Kyrylych (Роман Кирилич)

Aaron Griffin

4:17 p.m.

On Tue, Mar 11, 2008 at 1:52 AM, Mark Constable wrote:

...

I'm just checking if this could be a reasonable idea and if so I'd look into providing a patch. Would it be feasible to add a %PATHNAME% variable to the sync desc file ?

For instance in this case, fetching "any" arch packages would not need to symlinked into the same directory to be found...

%FILENAME% hotstuff-0.9.2-1-any.pkg.tar.bz2

%PATHNAME% ../any/

It may provide repo maintainers with some flexibility (hard to test without it being possible) and perhaps %PATHNAME% could also be a URL and that might also be interesting, but the simple relative path prefix would be useful.

What's wrong with: %FILENAME% ../any/hotstuff-0.9.2-1-any.pkg.tar.bz2 As far as I know, that'd work just fine (assuming the http server respects '..', at least)

eliott

4:24 p.m.

On 3/11/08, Aaron Griffin wrote:

...

On Tue, Mar 11, 2008 at 1:52 AM, Mark Constable wrote:

...
I'm just checking if this could be a reasonable idea and if so I'd look into providing a patch. Would it be feasible to add a %PATHNAME% variable to the sync desc file ?

For instance in this case, fetching "any" arch packages would not need to symlinked into the same directory to be found...

%FILENAME% hotstuff-0.9.2-1-any.pkg.tar.bz2

%PATHNAME% ../any/

It may provide repo maintainers with some flexibility (hard to test without it being possible) and perhaps %PATHNAME% could also be a URL and that might also be interesting, but the simple relative path prefix would be useful.

What's wrong with:

%FILENAME% ../any/hotstuff-0.9.2-1-any.pkg.tar.bz2

As far as I know, that'd work just fine (assuming the http server respects '..', at least)

Relative pathnames in the repodb... for some reason this seems pretty horrible to me.

Aaron Griffin

4:36 p.m.

On Tue, Mar 11, 2008 at 11:24 AM, eliott wrote:

...

On 3/11/08, Aaron Griffin wrote:

...
On Tue, Mar 11, 2008 at 1:52 AM, Mark Constable wrote:

...
I'm just checking if this could be a reasonable idea and if so I'd look into providing a patch. Would it be feasible to add a %PATHNAME% variable to the sync desc file ?

For instance in this case, fetching "any" arch packages would not need to symlinked into the same directory to be found...

%FILENAME% hotstuff-0.9.2-1-any.pkg.tar.bz2

%PATHNAME% ../any/

It may provide repo maintainers with some flexibility (hard to test without it being possible) and perhaps %PATHNAME% could also be a URL and that might also be interesting, but the simple relative path prefix would be useful.

What's wrong with:

%FILENAME% ../any/hotstuff-0.9.2-1-any.pkg.tar.bz2

As far as I know, that'd work just fine (assuming the http server respects '..', at least)

Relative pathnames in the repodb... for some reason this seems pretty horrible to me.

That's not really my point. I was just trying to say that this does actually work right now.

eliott

4:46 p.m.

On 3/11/08, Aaron Griffin wrote:

...

On Tue, Mar 11, 2008 at 11:24 AM, eliott wrote:

...
On 3/11/08, Aaron Griffin wrote:

...
On Tue, Mar 11, 2008 at 1:52 AM, Mark Constable wrote:

...
I'm just checking if this could be a reasonable idea and if so I'd look into providing a patch. Would it be feasible to add a %PATHNAME% variable to the sync desc file ?

For instance in this case, fetching "any" arch packages would not need to symlinked into the same directory to be found...

%FILENAME% hotstuff-0.9.2-1-any.pkg.tar.bz2

%PATHNAME% ../any/

It may provide repo maintainers with some flexibility (hard to test without it being possible) and perhaps %PATHNAME% could also be a URL and that might also be interesting, but the simple relative path prefix would be useful.

What's wrong with:

%FILENAME% ../any/hotstuff-0.9.2-1-any.pkg.tar.bz2

As far as I know, that'd work just fine (assuming the http server respects '..', at least)

Relative pathnames in the repodb... for some reason this seems pretty horrible to me.

That's not really my point. I was just trying to say that this does actually work right now.

you were just the last in the thread, so i hit reply. also .. clarification.. I meant 'upwardly' relative. It is 'downwardly' relative right now.

Roman Kyrylych

9:24 p.m.

2008/3/11, eliott :

...

On 3/11/08, Aaron Griffin wrote:

...
On Tue, Mar 11, 2008 at 1:52 AM, Mark Constable wrote:

...
I'm just checking if this could be a reasonable idea and if so I'd look into providing a patch. Would it be feasible to add a %PATHNAME% variable to the sync desc file ?

For instance in this case, fetching "any" arch packages would not need to symlinked into the same directory to be found...

%FILENAME% hotstuff-0.9.2-1-any.pkg.tar.bz2

%PATHNAME% ../any/

It may provide repo maintainers with some flexibility (hard to test without it being possible) and perhaps %PATHNAME% could also be a URL and that might also be interesting, but the simple relative path prefix would be useful.

What's wrong with:

%FILENAME% ../any/hotstuff-0.9.2-1-any.pkg.tar.bz2

As far as I know, that'd work just fine (assuming the http server respects '..', at least)

Relative pathnames in the repodb... for some reason this seems pretty horrible to me.

Wow! I didn't know that works. Thanks, Aaron! :-) -- Roman Kyrylych (Роман Кирилич)

Dan McGee

10:36 p.m.

On Tue, Mar 11, 2008 at 1:52 AM, Mark Constable wrote:

...

I'm just checking if this could be a reasonable idea and if so I'd look into providing a patch. Would it be feasible to add a %PATHNAME% variable to the sync desc file ?

For instance in this case, fetching "any" arch packages would not need to symlinked into the same directory to be found...

%FILENAME% hotstuff-0.9.2-1-any.pkg.tar.bz2

%PATHNAME% ../any/

It may provide repo maintainers with some flexibility (hard to test without it being possible) and perhaps %PATHNAME% could also be a URL and that might also be interesting, but the simple relative path prefix would be useful.

Any thoughts?

Yeah, I have some thoughts. :) Not a huge fan of this behavior, as I really like the current scenario where the DB always lives with the packages. It greatly simplifies the need for repo-add to do anything funny with the paths, as we just know to always add the basename of the file. That would need to be addressed if we added this functionality- how would you add relative paths using repo-add? Would it generate the relative path on its own (possibly traversing out of webserver accessible directories which will break your repository), or would you have to specify an additional path? What if I build my DB in one place, but serve it in another (something I actually do)? We haven't even begun to talk about deltas and where they should live. This came up when Xavier and I were researching the delta creation being done in repo-add. Of course, I see the advantages this could offer with the -any packages, I just wanted to get my two cents in from the other side. -Dan

Mark Constable

12 Mar 12 Mar

4:52 a.m.

On Wednesday 12 March 2008 08:36:16 Dan McGee wrote:

...

...
Would it be feasible to add a %PATHNAME% variable to the sync desc file ? For instance in this case, fetching "any" arch packages would not need to symlinked into the same directory to be found...

%FILENAME% hotstuff-0.9.2-1-any.pkg.tar.bz2

%PATHNAME% ../any/

...

Not a huge fan of this behavior, as I really like the current scenario where the DB always lives with the packages.

One added complexity, that is currently available, is having symlinks to packages outside of the current directory so, in a way, the DB does not always live with the packages. The symlinks remain in the same directory but not the actual packages. To me, this *may* be clarified better by having the path component explicitly noted in the DB itself rather than implied by an external filesystem symlink. Maybe.

...

It greatly simplifies the need for repo-add to do anything funny with the paths, as we just know to always add the basename of the file. That would need to be addressed if we added this functionality- how would you add relative paths using repo-add?

Well I just tried this below but the resulting package could not be found and the local sync DB showed no sign of including the path... cd my/i686 repo-add my.db.tar.gz *.pkg.* ../any/*.pkg.* so I thought/hoped it might do as Aaron suggested and end up with a full relative path but it didn't when I checked the local sync DB entry for the packages in ../any/. %FILENAME% ../any/hotstuff-0.9.2-1-any.pkg.tar.bz2

...

Would it generate the relative path on its own (possibly traversing out of webserver accessible directories which will break your repository),

I would think this relative path is matched to the remote DB and would not get added if it could't be found at the time repo-add is invoked. It would be assumed the same layout, as stated in the DB, would exist in reality regardless of how the whole repo is copied or mirrored.

...

or would you have to specify an additional path? What if I build my DB in one place, but serve it in another (something I actually do)?

I rsync my local repo to a remote live server and in this case there would be no problem that I can forsee.

...

We haven't even begun to talk about deltas and where they should live.

They're just more file components that could follow the same relative patch agenda.

...

This came up when Xavier and I were researching the delta creation being done in repo-add.

Of course, I see the advantages this could offer with the -any packages, I just wanted to get my two cents in from the other side.

Sure, as is any change worth the tradeoff between added benefit or possible breakage. *** On the PATHNAME = URL idea, I could imagine this could also be useful, if at all possible, because I could offload some packages to a variety of other backend servers according to demand. I could have the DBs hosted on a server with limited bandwidth but high availability and offload some or all of the packages to another server with a higher and/or cheaper bandwidth quota. I could perhaps use Amazon S3 cloud for important popular packages and a Dreamhost cheapie account for less popular packages, all from the same repo sync DB. I'm not suggesting this idea has any high yield benefits but if it were available then I could imagine taking advantage of it. However, I suspect bittorenting packages would probably have the best impact on distribution load. Better still, if it were possible to "capture" what would be otherwise transferred via an rsync update to a local binary blob diff (for the whole archive, or per repo section) and use bittorent technology to transfer that blob and patch the other end in one go. --markc

Xavier

2:04 p.m.

On Wed, Mar 12, 2008 at 02:52:43PM +1000, Mark Constable wrote:

...

***

On the PATHNAME = URL idea, I could imagine this could also be useful, if at all possible, because I could offload some packages to a variety of other backend servers according to demand. I could have the DBs hosted on a server with limited bandwidth but high availability and offload some or all of the packages to another server with a higher and/or cheaper bandwidth quota. I could perhaps use Amazon S3 cloud for important popular packages and a Dreamhost cheapie account for less popular packages, all from the same repo sync DB.

That sounds ugly. Don't you have more transparent ways to achieve the same?

Mark Constable

11:36 p.m.

On 13 Mar 2008 00:04, Xavier wrote:

...

...
On the PATHNAME = URL idea, I could imagine this could also be useful, if at all possible, because I could offload some packages to a variety of other backend servers according to demand. I could have the DBs hosted on a server with limited bandwidth but high availability and offload some or all of the packages to another server with a higher and/or cheaper bandwidth quota. I could perhaps use Amazon S3 cloud for important popular packages and a Dreamhost cheapie account for less popular packages, all from the same repo sync DB.

That sounds ugly.

I would have thought it to be "amazingly flexible" and not availble with any other packaging system.

...

Don't you have more transparent ways to achieve the same?

I could perhaps set up apache rewrite rules for individual packages to point to other destinations but that, to me, would indeed be truly fugly. On this issue of mirroring, I guess I'll do the regular thing of contacting major mirroring services and see if I can hop on board. --markc

5899

Age (days ago)

5900

Last active (days ago)

List overview

Download

10 comments

6 participants

participants (6)

Aaron Griffin
Dan McGee
eliott
Mark Constable
Roman Kyrylych
Xavier