[aur-general] AUR 4 => Migrating 1 git to git-per-package - Aur-general - lists.archlinux.org

newer
[aur-general] Disowning some...

[aur-general] AUR 4 => Migrating 1 git to git-per-package

older
[aur-general] package review

Phillip Smith

30 Dec 2014 30 Dec '14

12:53 a.m.

For those of us already storing our packages in git[0], has anyone got some documentation/pointers for how to handle migrating from having 1 big git repository to dozens of individual git repositories? Do we just blat away our current repository and re-initialize each package as it's own repository or is there a smarter way? [0] https://github.com/fukawi2/aur-packages

Show replies by date

Ido Rosen

30 Dec 30 Dec

12:54 a.m.

git help subtree. On Mon, Dec 29, 2014 at 7:53 PM, Phillip Smith wrote:

For those of us already storing our packages in git[0], has anyone got some documentation/pointers for how to handle migrating from having 1 big git repository to dozens of individual git repositories?

Do we just blat away our current repository and re-initialize each package as it's own repository or is there a smarter way?

[0] https://github.com/fukawi2/aur-packages

Ido Rosen

12:58 a.m.

On Mon, Dec 29, 2014 at 7:54 PM, Ido Rosen wrote:

git help subtree.

On Mon, Dec 29, 2014 at 7:53 PM, Phillip Smith wrote:

...
For those of us already storing our packages in git[0], has anyone got some documentation/pointers for how to handle migrating from having 1 big git repository to dozens of individual git repositories?

Do we just blat away our current repository and re-initialize each package as it's own repository or is there a smarter way?

"git help subtree", but to clarify further: I do this right now in https://github.com/ido/packages-archlinux ...I have a script prepared using git subtree and a filter-branch command to split out the /aur directory in that repo into multiple packages. I'm testing it locally. The best part is that I can still maintain the individual package repos separately from the main one, and keep a separate git repository with all my individual repos as submodules, then just use "git submodule foreach git subtree pull --squash" (or similar) to update all the packages at once and only if those subfolders were changed in the one-big-repository (ido/packages-archlinux.git)

...
[0] https://github.com/fukawi2/aur-packages

Ido Rosen

12:59 a.m.

On Mon, Dec 29, 2014 at 7:58 PM, Ido Rosen wrote:

On Mon, Dec 29, 2014 at 7:54 PM, Ido Rosen wrote:

...
git help subtree.

On Mon, Dec 29, 2014 at 7:53 PM, Phillip Smith wrote:

...
For those of us already storing our packages in git[0], has anyone got some documentation/pointers for how to handle migrating from having 1 big git repository to dozens of individual git repositories?

Do we just blat away our current repository and re-initialize each package as it's own repository or is there a smarter way?

"git help subtree", but to clarify further: I do this right now in https://github.com/ido/packages-archlinux

...I have a script prepared using git subtree and a filter-branch command to split out the /aur directory in that repo into multiple packages. I'm testing it locally. The best part is that I can still maintain the individual package repos separately from the main one, and keep a separate git repository with all my individual repos as submodules, then just use "git submodule foreach git subtree pull --squash" (or similar) to update all the packages at once and only if those subfolders were changed in the one-big-repository (ido/packages-archlinux.git)

(...which can be done as a post-commit or post-receive hook in the master repo.)

...
...
[0] https://github.com/fukawi2/aur-packages

Phillip Smith

1:04 a.m.

On 30 December 2014 at 11:54, Ido Rosen wrote:

git help subtree.

My git-fu is weak beyond add/commit/push/pull. Looks a little complex: http://makingsoftware.wordpress.com/2013/02/16/using-git-subtrees-for-reposi... Perhaps we should develop some kind of migration script to assist?

Ido Rosen

1:12 a.m.

If you follow the pattern I do, of having a directory within your git repository (I call mine "aur") that contains directories named after each package's $pkgbase (or $pkgname if not a split package), then I should be able to post my script here (after I've tested it a little more on aur-dev) later tonight. On Mon, Dec 29, 2014 at 8:04 PM, Phillip Smith wrote:

On 30 December 2014 at 11:54, Ido Rosen wrote:

...
git help subtree.

My git-fu is weak beyond add/commit/push/pull. Looks a little complex: http://makingsoftware.wordpress.com/2013/02/16/using-git-subtrees-for-reposi...

Perhaps we should develop some kind of migration script to assist?

Ido Rosen

4:38 a.m.

Hi all, TL;DR: https://github.com/ido/packages-archlinux/blob/master/bin/import-to-aur4.sh I maintain my AUR packages in https://github.com/ido/packages-archlinux, under a subdirectory called "aur", with each PKGBUILD in its own folder under that "aur" folder. So, for example, ceph-git is in $WORKDIR/aur/ceph-git/PKGBUILD... To migrate this to AUR 4.0 per-package git repositories, I used git subtree and git filter-branch (to get around that pesky server-side update hook that looks for a valid .SRCINFO). 0) It does all of its work in a temp dir (/tmp/importaur4.XXXXX by default; export TMPDIR=$HOME/ to do it in your $HOME for example). It clones a fresh copy of the upstream one-big-git-repository (use file:/// if you want to do this from a local repo). It accepts two arguments: $1 = URL to repo to git clone, $2 = subdirectory relative path from workdir root. Warning: Use ssh-add/ssh-agent to add your AUR ssh key otherwise you will have to do a lot of typing! 1) It uses git subtree split to split out packages into their own branches (this works much like git filter-branch --subdirectory-filter). 2) It uses the code from pkgbuild-introspection to create a SRCINFO file. 3) It uses git filter-branch --tree-filter to copy in the .SRCINFO it generated for any commits on the per-package branches that do not already have a .SRCINFO file in them. 4) It uses git push to push the per-package branches (aur4/$pkgbase) to the master branch on the aur-dev server. 5) It creates a "superproject" git repository for you in /tmp/importaur4.XXXX/ to track all of your per-package repos in one place, which has git submodules for each per-package repository. This is for convenience and is located in /tmp/importaur4.XXXX/aur4-superproject. Please send pull requests for any improvements you make; I wrote this quickly with the intent of using it for my own packages. It fails on some repositories, mostly due to the server-side update hook in AUR4 right now. Example failures shown below. The script is located at: https://github.com/ido/packages-archlinux/blob/master/bin/import-to-aur4.sh Enjoy! Ido Problem 1: blobs larger than 100kB are rejected by aur-dev...so they don't get imported by this script. This is problematic for kernel packages with inline patches. + git push ssh+git://aur@aur-dev.archlinux.org:2222/linux-lts-tresor.git/ aur4/linux-lts-tresor:master Counting objects: 33, done. Delta compression using up to 8 threads. Compressing objects: 100% (32/32), done. Writing objects: 100% (33/33), 53.36 KiB | 0 bytes/s, done. Total 33 (delta 16), reused 6 (delta 1) remote: error: The following error occurred when parsing commit remote: error: 22828bff8f2c9890b386e8584a927e64507fe6c1: remote: error: maximum blob size (100kB) exceeded remote: error: hook declined to update refs/heads/master To ssh+git://aur@aur-dev.archlinux.org:2222/linux-lts-tresor.git/ ! [remote rejected] aur4/linux-lts-tresor -> master (hook declined) --- Problem 2: The AUR4 update hook is poorly written, it compares a string to an int when looking at the pkginfo['epoch'], so packages with an epoch set fail to pass the update hook. Failed to setup-repo ceph-git ... maybe it already exists? + git push ssh+git://aur@aur-dev.archlinux.org:2222/ceph-git.git/ aur4/ceph-git:master Counting objects: 35, done. Delta compression using up to 8 threads. Compressing objects: 100% (25/25), done. Writing objects: 100% (35/35), 7.48 KiB | 0 bytes/s, done. Total 35 (delta 17), reused 14 (delta 10) remote: Traceback (most recent call last): remote: File "hooks/update", line 212, in <module> remote: save_srcinfo(srcinfo, db, cur, user) remote: File "hooks/update", line 45, in save_srcinfo remote: if 'epoch' in pkginfo and pkginfo['epoch'] > 0: remote: TypeError: unorderable types: str() > int() remote: error: hook declined to update refs/heads/master To ssh+git://aur@aur-dev.archlinux.org:2222/ceph-git.git/ ! [remote rejected] aur4/ceph-git -> master (hook declined) On Mon, Dec 29, 2014 at 8:12 PM, Ido Rosen wrote:

If you follow the pattern I do, of having a directory within your git repository (I call mine "aur") that contains directories named after each package's $pkgbase (or $pkgname if not a split package), then I should be able to post my script here (after I've tested it a little more on aur-dev) later tonight.

On Mon, Dec 29, 2014 at 8:04 PM, Phillip Smith wrote:

...
On 30 December 2014 at 11:54, Ido Rosen wrote:

...
git help subtree.

My git-fu is weak beyond add/commit/push/pull. Looks a little complex: http://makingsoftware.wordpress.com/2013/02/16/using-git-subtrees-for-reposi...

Perhaps we should develop some kind of migration script to assist?

Ido Rosen

4:41 a.m.

On Mon, Dec 29, 2014 at 11:38 PM, Ido Rosen wrote:

Hi all,

TL;DR: https://github.com/ido/packages-archlinux/blob/master/bin/import-to-aur4.sh

This is how I invoke it: bin/import-to-aur4.sh https://github.com/ido/packages-archlinux aur (replace ido/packages-archlinux and aur with what's appropriate for you, if you replace aur with . then it will work in the root of the git repo. It checks for existence of a PKGBUILD file, so it should do the right thing most of the time.)

Hugo Osvaldo Barrera

18 Jan 18 Jan

10:26 p.m.

On 2014-12-29 23:41, Ido Rosen wrote:

On Mon, Dec 29, 2014 at 11:38 PM, Ido Rosen wrote:

...
Hi all,

TL;DR: https://github.com/ido/packages-archlinux/blob/master/bin/import-to-aur4.sh

This is how I invoke it: bin/import-to-aur4.sh https://github.com/ido/packages-archlinux aur

(replace ido/packages-archlinux and aur with what's appropriate for you, if you replace aur with . then it will work in the root of the git repo. It checks for existence of a PKGBUILD file, so it should do the right thing most of the time.)

I would have gone with `git branch --orphan` and `git cherry-pick`, but this looks like an interesting alternative I had not thought of. Thanks for the script! -- Hugo Osvaldo Barrera A: Because we read from top to bottom, left to right. Q: Why should I start my reply below the quoted text?

Ido Rosen

10:39 p.m.

On Sun, Jan 18, 2015 at 5:26 PM, Hugo Osvaldo Barrera wrote:

On 2014-12-29 23:41, Ido Rosen wrote:

...
On Mon, Dec 29, 2014 at 11:38 PM, Ido Rosen wrote:

...
Hi all,

TL;DR: https://github.com/ido/packages-archlinux/blob/master/bin/import-to-aur4.sh

This is how I invoke it: bin/import-to-aur4.sh https://github.com/ido/packages-archlinux aur

(replace ido/packages-archlinux and aur with what's appropriate for you, if you replace aur with . then it will work in the root of the git repo. It checks for existence of a PKGBUILD file, so it should do the right thing most of the time.)

I would have gone with `git branch --orphan` and `git cherry-pick`, but this looks like an interesting alternative I had not thought of.

Thanks for the script!

You're welcome, and I'm totally open to improvements/changes to the script if you want to send a pull request! :-)

-- Hugo Osvaldo Barrera A: Because we read from top to bottom, left to right. Q: Why should I start my reply below the quoted text?

3379

Age (days ago)

3398

Last active (days ago)

Download

9 comments

3 participants

tags

participants (3)

Hugo Osvaldo Barrera
Ido Rosen
Phillip Smith