[arch-general] An evil idea --- use Git to manage the repositories
Is it possible? advantage: 1 The mirrors do not need download the total new pkgs if it just updated several files. 2 old versions of package can be retrived. 3 GIT is fast. 4 It seems cool! dis-advantage: 1 It is a torture of GIT 2 Huge of works to be done.
On Wed, 2 Sep 2009 16:35:12 +0800 goodmenzy@gmail.com wrote:
Is it possible?
advantage: 1 The mirrors do not need download the total new pkgs if it just updated several files. rsync does this too 2 old versions of package can be retrived. YAGNI 3 GIT is fast. rsync is too 4 It seems cool! rsync is more suited for the job
dis-advantage: 1 It is a torture of GIT 2 Huge of works to be done.
I think a big improvemt will be when we fix it so that if we move packages from testing to extra/core, mirrors will not see as a "file remove and a new file", which causes a lot of unneeded traffic. if we make this system use symlinks or whatever this can be handled much better. but other then that, I think the current syncing system is fine. (the "getting" part is something different. see tiered setup,mirrorbrain etc. discussions) Dieter
goodmenzy@gmail.com schrieb:
Is it possible?
advantage: 1 The mirrors do not need download the total new pkgs if it just updated several files. 2 old versions of package can be retrived. 3 GIT is fast. 4 It seems cool!
dis-advantage: 1 It is a torture of GIT 2 Huge of works to be done.
I don't think git is suitable for maintaining large amounts of binary files. The diffs probably won't be very efficient, so storage size will explode. And - all mirrors use rsync (they also mirror other stuff besides Arch), we can't just tell them all to do it differently.
On Wed, Sep 2, 2009 at 10:35 AM, <goodmenzy@gmail.com> wrote:
Is it possible?
advantage: 1 The mirrors do not need download the total new pkgs if it just updated several files. 2 old versions of package can be retrived. 3 GIT is fast. 4 It seems cool!
dis-advantage: 1 It is a torture of GIT 2 Huge of works to be done.
Without mentioning that any scm would probably explode handling such big amount of binary files, did this idea only concern the mirror synchronization part ? who would choose to retrieve an older version of a package ? and how would you do that ? (anyway the first point makes any discussion worthless).
you are forgetting how quickly and ginormassly huge the git repo's would get. -- Caleb Cushing http://xenoterracide.blogspot.com
Forsight linux does something like this, but their server side was closed source the last time I took a look(back in February). http://en.wikipedia.org/wiki/Conary_(package_manager) It is a very interesting idea, and could speed things up a lot. It would make downloading packages to install elsewhere a pain in some cases though. Using GIT as-is would be a bad idea(perhaps hack mercurial to do the work instead?). It would be rather non-KISS, but it does not sound too hard so long as you do not allow for rollbacks(Just store a list of what files are different and thus would need to be updated). goodmenz if you want to try to hack something together, I should have time to assist, although not much(still in school, busy semester). Anyone else interested? Forking mercurial and pacman into MerMan would be the new Linux package management system which finally bring about the year of the Linux desktop! ~Nekody 林克迪 On 9/9/09, Caleb Cushing <xenoterracide@gmail.com> wrote:
you are forgetting how quickly and ginormassly huge the git repo's would get.
-- Caleb Cushing
Hi, It is just an idea, nothing more than it. These days, I had think more than had done anything( a bad hehaver....) As an clumzy idea, git will help to manage the config/build script files. And binary pkgs and source pkgs should be maintained by something else. As an embeded engineer, my insteresting is to maintain a repository which contains a software of several version(Stupid idea? but my customer liks old versions). It may support many versions of kernels( maybe include BSD kernel? ) And I need some tools to help to install a new OS from scretch VERY quickly. Build from souce code should be as easy as ABS. Maybe Gentoo is a good reference..... Now, I do not had any clear thought, needless to say coding. It will takes some time to develop a prototype system. So many linux distros exists, maybe this will be another one(if Arch community refuse it to be called Arch )? On 2009-09-17 23:10:10, nekomancer davion wrote:
Date: Thu, 17 Sep 2009 23:10:10 -0400 From: nekomancer davion <ladislaio@gmail.com> To: General Discusson about Arch Linux <arch-general@archlinux.org> Subject: Re: [arch-general] An evil idea --- use Git to manage the repositories Reply-To: General Discusson about Arch Linux <arch-general@archlinux.org> Message-ID: <bd5ae8c50909172010q110ceb4aif005c02f8063c413@mail.gmail.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable List-Id: General Discusson about Arch Linux <arch-general.archlinux.org>
Forsight linux does something like this, but their server side was closed source the last time I took a look(back in February). http://en.wikipedia.org/wiki/Conary_(package_manager)
It is a very interesting idea, and could speed things up a lot. It would make downloading packages to install elsewhere a pain in some cases though. Using GIT as-is would be a bad idea(perhaps hack mercurial to do the work instead?).
It would be rather non-KISS, but it does not sound too hard so long as you do not allow for rollbacks(Just store a list of what files are different and thus would need to be updated).
goodmenz if you want to try to hack something together, I should have time to assist, although not much(still in school, busy semester). Anyone else interested?
Forking mercurial and pacman into MerMan would be the new Linux package management system which finally bring about the year of the Linux desktop!
~Nekody 林克迪
On 9/9/09, Caleb Cushing <xenoterracide@gmail.com> wrote:
you are forgetting how quickly and ginormassly huge the git repo's would get.
-- Caleb Cushing
A new distro? evil idea!! By the way: reinvent the wheel may be an interesting progress............................................................................. On 2009-09-17 23:10:10, nekomancer davion wrote:
Date: Thu, 17 Sep 2009 23:10:10 -0400 From: nekomancer davion <ladislaio@gmail.com> To: General Discusson about Arch Linux <arch-general@archlinux.org> Subject: Re: [arch-general] An evil idea --- use Git to manage the repositories Reply-To: General Discusson about Arch Linux <arch-general@archlinux.org> Message-ID: <bd5ae8c50909172010q110ceb4aif005c02f8063c413@mail.gmail.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable List-Id: General Discusson about Arch Linux <arch-general.archlinux.org>
Forsight linux does something like this, but their server side was closed source the last time I took a look(back in February). http://en.wikipedia.org/wiki/Conary_(package_manager)
It is a very interesting idea, and could speed things up a lot. It would make downloading packages to install elsewhere a pain in some cases though. Using GIT as-is would be a bad idea(perhaps hack mercurial to do the work instead?).
It would be rather non-KISS, but it does not sound too hard so long as you do not allow for rollbacks(Just store a list of what files are different and thus would need to be updated).
goodmenz if you want to try to hack something together, I should have time to assist, although not much(still in school, busy semester). Anyone else interested?
Forking mercurial and pacman into MerMan would be the new Linux package management system which finally bring about the year of the Linux desktop!
~Nekody 林克迪
On 9/9/09, Caleb Cushing <xenoterracide@gmail.com> wrote:
you are forgetting how quickly and ginormassly huge the git repo's would get.
-- Caleb Cushing
Not a new distro, goodness no. Just an alternate package management system, which would work something like this: It would be transparent with pacman. On could easily use pacman or it interchangeably, but this would have additional package management 'tools'. Many functions, such as installing a new package, re-installing a package, and package removal would be offloaded onto pacman anyway. The only real difference with this would be when upgrading(and possibly downgrading) a package, and possibly it would have some more robust information about package file information. All scripts, config files, pacbuilds, and such would be maintained by a distributed scm (Mercurial is what I like, but Git could work too). This may have to be customized to more closely fit our purposes, or maybe even re-implemented with as the tasks we would demand of it would not be as great(perhaps). This may also allow for a person to simply reset all config files to their defaults in a very lazy fashion. Binaries would be individually hashed (SHA1?), and then this information would be stored. After being hashed the binaries would be tightly compressed(Bzip2?) as individual files, if the binary does not already have a hash value which matches up with a previous hash of that file(in this case the new file is simply discarded - it did not change). A hash for checking the integrity of the compressed file would also be stored. When a person decides to sync their package information, the data on file differences is what syncs(in addition to the package version info). When the client decides to upgrade a file, the clients computer would check what the different files between the currently installed package and the package on the mirror were. These files would then be what they would request to download. Following downloading these files, they would be extracted to the proper locations (using .pacnew for new config files that exist in the system). A low priority would be for there to be both an old config file backup system and a merge of new config files with old config files function(most likely taken directly from the scm being used). This would not be enforced on the user, but some may like it as an option (opposed to manually hunting down config files that need to be updated). Verbose logging of all operations would be a very high priority. The hashes of the uncompressed binaries would be important, as it would allow for the package management software to determine if binaries have be modified since they were installed. By default I do not think it would care when upgrading, but some people(esp. those with RPM roots) may enjoy this for being able to check for rootkits and file corruption. Downgrading of the system would not be a high priority at all, but it may not be excluded. This system would have a greater dependence on a local internet connection, and thus I would not like to force it upon people. I like pacman a lot, but this is an interesting idea on management(and I have been jealous of Forsight linux's capabilities similar to this. If only it was a lightweight build your own like Arch...) And that is the summery of my current thoughts on this. Any thoughts or willing helpers out there? ~Nekody 林科迪 On 9/18/09, goodmenzy@gmail.com <goodmenzy@gmail.com> wrote:
A new distro? evil idea!! By the way: reinvent the wheel may be an interesting progress.............................................................................
On 2009-09-17 23:10:10, nekomancer davion wrote:
Date: Thu, 17 Sep 2009 23:10:10 -0400 From: nekomancer davion <ladislaio@gmail.com> To: General Discusson about Arch Linux <arch-general@archlinux.org> Subject: Re: [arch-general] An evil idea --- use Git to manage the repositories Reply-To: General Discusson about Arch Linux <arch-general@archlinux.org> Message-ID: <bd5ae8c50909172010q110ceb4aif005c02f8063c413@mail.gmail.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable List-Id: General Discusson about Arch Linux <arch-general.archlinux.org>
Forsight linux does something like this, but their server side was closed source the last time I took a look(back in February). http://en.wikipedia.org/wiki/Conary_(package_manager)
It is a very interesting idea, and could speed things up a lot. It would make downloading packages to install elsewhere a pain in some cases though. Using GIT as-is would be a bad idea(perhaps hack mercurial to do the work instead?).
It would be rather non-KISS, but it does not sound too hard so long as you do not allow for rollbacks(Just store a list of what files are different and thus would need to be updated).
goodmenz if you want to try to hack something together, I should have time to assist, although not much(still in school, busy semester). Anyone else interested?
Forking mercurial and pacman into MerMan would be the new Linux package management system which finally bring about the year of the Linux desktop!
~Nekody 林克迪
On 9/9/09, Caleb Cushing <xenoterracide@gmail.com> wrote:
you are forgetting how quickly and ginormassly huge the git repo's would get.
-- Caleb Cushing
you should all go look at funtoo... since it already uses git with portage for package management.
participants (6)
-
Caleb Cushing
-
Dieter Plaetinck
-
goodmenzy＠gmail.com
-
nekomancer davion
-
Thomas Bächler
-
Xavier