[aur-general] AUR 4 => Migrating 1 git to git-per-package

[Ido Rosen]

Hi all,

TL;DR: https://github.com/ido/packages-archlinux/blob/master/bin/import-to-aur4.sh

  I maintain my AUR packages in
https://github.com/ido/packages-archlinux, under a subdirectory called
"aur", with each PKGBUILD in its own folder under that "aur" folder.
So, for example, ceph-git is in $WORKDIR/aur/ceph-git/PKGBUILD...

To migrate this to AUR 4.0 per-package git repositories, I used git
subtree and git filter-branch (to get around that pesky server-side
update hook that looks for a valid .SRCINFO).

0) It does all of its work in a temp dir (/tmp/importaur4.XXXXX by
default; export TMPDIR=$HOME/ to do it in your $HOME for example).  It
clones a fresh copy of the upstream one-big-git-repository (use
file:/// if you want to do this from a local repo).  It accepts two
arguments: $1 = URL to repo to git clone, $2 = subdirectory relative
path from workdir root.  Warning: Use ssh-add/ssh-agent to add your
AUR ssh key otherwise you will have to do a lot of typing!
1) It uses git subtree split to split out packages into their own
branches (this works much like git filter-branch
--subdirectory-filter).
2) It uses the code from pkgbuild-introspection to create a SRCINFO file.
3) It uses git filter-branch --tree-filter to copy in the .SRCINFO it
generated for any commits on the per-package branches that do not
already have a .SRCINFO file in them.
4) It uses git push to push the per-package branches (aur4/$pkgbase)
to the master branch on the aur-dev server.
5) It creates a "superproject" git repository for you in
/tmp/importaur4.XXXX/ to track all of your per-package repos in one
place, which has git submodules for each per-package repository.  This
is for convenience and is located in
/tmp/importaur4.XXXX/aur4-superproject.

Please send pull requests for any improvements you make; I wrote this
quickly with the intent of using it for my own packages.  It fails on
some repositories, mostly due to the server-side update hook in AUR4
right now.  Example failures shown below.

The script is located at:
https://github.com/ido/packages-archlinux/blob/master/bin/import-to-aur4.sh

Enjoy!
Ido

Problem 1: blobs larger than 100kB are rejected by aur-dev...so they
don't get imported by this script.  This is problematic for kernel
packages with inline patches.

+ git push ssh+git://aur@aur-dev.archlinux.org:2222/linux-lts-tresor.git/
aur4/linux-lts-tresor:master
Counting objects: 33, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (32/32), done.
Writing objects: 100% (33/33), 53.36 KiB | 0 bytes/s, done.
Total 33 (delta 16), reused 6 (delta 1)
remote: error: The following error occurred when parsing commit
remote: error: 22828bff8f2c9890b386e8584a927e64507fe6c1:
remote: error: maximum blob size (100kB) exceeded
remote: error: hook declined to update refs/heads/master
To ssh+git://aur@aur-dev.archlinux.org:2222/linux-lts-tresor.git/
 ! [remote rejected] aur4/linux-lts-tresor -> master (hook declined)

---
Problem 2: The AUR4 update hook is poorly written, it compares a
string to an int when looking at the pkginfo['epoch'], so packages
with an epoch set fail to pass the update hook.

Failed to setup-repo ceph-git ... maybe it already exists?
+ git push ssh+git://aur@aur-dev.archlinux.org:2222/ceph-git.git/
aur4/ceph-git:master
Counting objects: 35, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (25/25), done.
Writing objects: 100% (35/35), 7.48 KiB | 0 bytes/s, done.
Total 35 (delta 17), reused 14 (delta 10)
remote: Traceback (most recent call last):
remote:   File "hooks/update", line 212, in <module>
remote:     save_srcinfo(srcinfo, db, cur, user)
remote:   File "hooks/update", line 45, in save_srcinfo
remote:     if 'epoch' in pkginfo and pkginfo['epoch'] > 0:
remote: TypeError: unorderable types: str() > int()
remote: error: hook declined to update refs/heads/master
To ssh+git://aur@aur-dev.archlinux.org:2222/ceph-git.git/
 ! [remote rejected] aur4/ceph-git -> master (hook declined)




On Mon, Dec 29, 2014 at 8:12 PM, Ido Rosen <ido at kernel.org> wrote:
> If you follow the pattern I do, of having a directory within your git
> repository (I call mine "aur") that contains directories named after
> each package's $pkgbase (or $pkgname if not a split package), then I
> should be able to post my script here (after I've tested it a little
> more on aur-dev) later tonight.
>
> On Mon, Dec 29, 2014 at 8:04 PM, Phillip Smith <fukawi2 at gmail.com> wrote:
>> On 30 December 2014 at 11:54, Ido Rosen <ido at kernel.org> wrote:
>>>
>>> git help subtree.
>>
>> My git-fu is weak beyond add/commit/push/pull. Looks a little complex:
>> http://makingsoftware.wordpress.com/2013/02/16/using-git-subtrees-for-repository-separation/
>>
>> Perhaps we should develop some kind of migration script to assist?