[arch-general] Any suggestions on frequently rebuilding a git package?
Hi, I build Emacs from git quite frequently, about twice a month or so. However the PKGBUILD from AUR rebuilds everything each time, which takes too many time I think. Perhaps Emacs is small, but I think there should be a way to prevent too many rebuilds --- is that possible? -- Carl Lei (XeCycle) Department of Physics, Shanghai Jiao Tong University OpenPGP public key: 7795E591 Fingerprint: 1FB6 7F1F D45D F681 C845 27F7 8D71 8EC4 7795 E591
On Sun, Sep 18, 2011 at 10:33 AM, XeCycle <xecycle@gmail.com> wrote:
Hi, I build Emacs from git quite frequently, about twice a month or so. However the PKGBUILD from AUR rebuilds everything each time, which takes too many time I think. Perhaps Emacs is small, but I think there should be a way to prevent too many rebuilds --- is that possible?
Don't install from the AUR, keep your local checkout wherever and just run "make" yourself, which will only build changes, with the occasional need to recompile entirely in the case of emacs. I don't see a direct way of writing a PKGBUILD that doesn't recompile everything, at least not without saving the source tree, and I'm pretty sure there isn't a standard way of doing that within arch's build system, but I could be wrong. I think it makes sense for PKGBUILDs to "recompile" everything. There are a lot of assumptions that would need to be made explicit that I doubt anyone would be too comfy with. If I'm wrong, and there's a standard way of doing this within PKGBUILDs, awesome. Assuming I'm not, I think the best you could do would be to not compile at all if the version you'd be compiling is the same as what's installed on your system, which you may already be doing.
On 19 September 2011 01:40, Jeremiah Dodds <jeremiah.dodds@gmail.com> wrote:
If I'm wrong, and there's a standard way of doing this within PKGBUILDs, awesome. Assuming I'm not, I think the best you could do would be to not compile at all if the version you'd be compiling is the same as what's installed on your system, which you may already be doing.
You're not wrong, at least not on one account (the one you mentioned first). 1. Keep buildscripts locally and update the package manually (this is a standard) You will no longer depend on auto-update mechanisms (yaourt et al) based on package updates from package maintainers, but keep the buildscripts on your system for yourself. You will then initiate makepkg as usual, and provided the PKGBUILD conforms to GIT packaging standards, the checkout itself will always be there and will be updated as and when needed. 2. There's no standard "caching" location for vcs packages However, you can simply adopt your own convention across PKGBUILDs that you use. One example is to have a standard location to keep all vcs checkouts, and refer to the respective dir in there and make your checks from within the PKGBUILD. -- GPG/PGP ID: 8AADBB10
2011/9/18 XeCycle <xecycle@gmail.com>:
Hi, I build Emacs from git quite frequently, about twice a month or so. However the PKGBUILD from AUR rebuilds everything each time, which takes too many time I think. Perhaps Emacs is small, but I think there should be a way to prevent too many rebuilds --- is that possible?
I assume you are keeping the same pkgbuild in your home dir, with the src dir too, so that the source won't be downloaded everytime again. Also use ccache, wich will speed up frequent recompiles. -- (\_ /) copy the bunny to your profile (0.o ) to help him achieve world domination. (> <) come join the dark side. /_|_\ (we have cookies.)
Excerpts from XeCycle's message of 2011-09-18 17:33:56 +0200:
Hi, I build Emacs from git quite frequently, about twice a month or so. However the PKGBUILD from AUR rebuilds everything each time, which takes too many time I think. Perhaps Emacs is small, but I think there should be a way to prevent too many rebuilds --- is that possible?
1. Don't use -c (clean) 2. Comment line 47 and 48 (PKGBUILD-git.proto), uncomment if you want a clean build. Regards, Philipp
On Mon, Sep 19, 2011 at 12:35 AM, Philipp Überbacher <hollunder@lavabit.com>wrote:
Excerpts from XeCycle's message of 2011-09-18 17:33:56 +0200:
Hi, I build Emacs from git quite frequently, about twice a month or so. However the PKGBUILD from AUR rebuilds everything each time, which takes too many time I think. Perhaps Emacs is small, but I think there should be a way to prevent too many rebuilds --- is that possible?
1. Don't use -c (clean)
2. Comment line 47 and 48 (PKGBUILD-git.proto), uncomment if you want a clean build.
Regards, Philipp
I used to do this with a couple of packages and it worked for me. Keep a clone of all git repos under some folder like ~/.vcs and then I put http://pastie.org/2558220 into ~/.vcs/update I then ran the update script before each time I wanted to rebuild the packages. Finally change the _gitroot variable to /home/username/.vcs/projectname and it will automatiically clone it from your hard drive. I usually saved this PKGBUILD for later so that I don't have to change it each time. You could also write a script to autodl the pkgbuild and replace the _gitroot. Hope this helps. Cheers
Many thanks to all of you, I'm trying them out. So the point is to never clean the code tree, nor cleaning those compiled objects. However I think re-configuring is not always needed, which results in lots of rebuilding. So It's better to decide myself when it's needed? Philipp Überbacher <hollunder@lavabit.com> writes:
Excerpts from XeCycle's message of 2011-09-18 17:33:56 +0200:
Hi, I build Emacs from git quite frequently, about twice a month or so. However the PKGBUILD from AUR rebuilds everything each time, which takes too many time I think. Perhaps Emacs is small, but I think there should be a way to prevent too many rebuilds --- is that possible?
1. Don't use -c (clean)
I already did it.
2. Comment line 47 and 48 (PKGBUILD-git.proto), uncomment if you want a clean build.
Well, this is my PKGBUILD-git.proto: 43 # 44 # BUILD HERE 45 # 46 ./autogen.sh 47 ./configure --prefix=/usr 48 make Do you mean commenting 46 & 47? -- Carl Lei (XeCycle) Department of Physics, Shanghai Jiao Tong University OpenPGP public key: 7795E591 Fingerprint: 1FB6 7F1F D45D F681 C845 27F7 8D71 8EC4 7795 E591
On Tue, Sep 20, 2011 at 05:01:58PM +0800, XeCycle wrote:
Many thanks to all of you, I'm trying them out.
So the point is to never clean the code tree, nor cleaning those compiled objects.
However I think re-configuring is not always needed, which results in lots of rebuilding. So It's better to decide myself when it's needed?
Philipp Überbacher <hollunder@lavabit.com> writes:
Excerpts from XeCycle's message of 2011-09-18 17:33:56 +0200:
Hi, I build Emacs from git quite frequently, about twice a month or so. However the PKGBUILD from AUR rebuilds everything each time, which takes too many time I think. Perhaps Emacs is small, but I think there should be a way to prevent too many rebuilds --- is that possible?
1. Don't use -c (clean)
I already did it.
2. Comment line 47 and 48 (PKGBUILD-git.proto), uncomment if you want a clean build.
Well, this is my PKGBUILD-git.proto:
43 # 44 # BUILD HERE 45 # 46 ./autogen.sh 47 ./configure --prefix=/usr 48 make
Do you mean commenting 46 & 47?
---- $ sed -n '47,48p' /usr/share/pacman/PKGBUILD-git.proto rm -rf "$srcdir/$_gitname-build" git clone "$srcdir/$_gitname" "$srcdir/$_gitname-build" ---- You should upgrade abs to 2.4.3-1.
Lukas Fleischer <archlinux@cryptocrack.de> writes: [...]
2. Comment line 47 and 48 (PKGBUILD-git.proto), uncomment if you want a clean build.
Well, this is my PKGBUILD-git.proto:
43 # 44 # BUILD HERE 45 # 46 ./autogen.sh 47 ./configure --prefix=/usr 48 make
Do you mean commenting 46 & 47?
---- $ sed -n '47,48p' /usr/share/pacman/PKGBUILD-git.proto rm -rf "$srcdir/$_gitname-build" git clone "$srcdir/$_gitname" "$srcdir/$_gitname-build" ----
You should upgrade abs to 2.4.3-1.
$ LC_ALL=C pacman -Qo /usr/share/pacman/PKGBUILD-git.proto /usr/share/pacman/PKGBUILD-git.proto is owned by abs 2.4.3-1 What's up? -- Carl Lei (XeCycle) Department of Physics, Shanghai Jiao Tong University OpenPGP public key: 7795E591 Fingerprint: 1FB6 7F1F D45D F681 C845 27F7 8D71 8EC4 7795 E591
On Tue, Sep 20, 2011 at 06:02:21PM +0800, XeCycle wrote:
Lukas Fleischer <archlinux@cryptocrack.de> writes:
[...]
2. Comment line 47 and 48 (PKGBUILD-git.proto), uncomment if you want a clean build.
Well, this is my PKGBUILD-git.proto:
43 # 44 # BUILD HERE 45 # 46 ./autogen.sh 47 ./configure --prefix=/usr 48 make
Do you mean commenting 46 & 47?
---- $ sed -n '47,48p' /usr/share/pacman/PKGBUILD-git.proto rm -rf "$srcdir/$_gitname-build" git clone "$srcdir/$_gitname" "$srcdir/$_gitname-build" ----
You should upgrade abs to 2.4.3-1.
$ LC_ALL=C pacman -Qo /usr/share/pacman/PKGBUILD-git.proto /usr/share/pacman/PKGBUILD-git.proto is owned by abs 2.4.3-1
What's up?
You either modified that file (use `pacman -S abs` to reinstall the package in that case) or use a weird editor with weird line numbering.
Lukas Fleischer <archlinux@cryptocrack.de> writes:
On Tue, Sep 20, 2011 at 06:02:21PM +0800, XeCycle wrote:
[...]
Well, this is my PKGBUILD-git.proto:
43 # 44 # BUILD HERE 45 # 46 ./autogen.sh 47 ./configure --prefix=/usr 48 make
Do you mean commenting 46 & 47?
---- $ sed -n '47,48p' /usr/share/pacman/PKGBUILD-git.proto rm -rf "$srcdir/$_gitname-build" git clone "$srcdir/$_gitname" "$srcdir/$_gitname-build" ----
You should upgrade abs to 2.4.3-1.
$ LC_ALL=C pacman -Qo /usr/share/pacman/PKGBUILD-git.proto /usr/share/pacman/PKGBUILD-git.proto is owned by abs 2.4.3-1
What's up?
You either modified that file (use `pacman -S abs` to reinstall the package in that case) or use a weird editor with weird line numbering.
Oops, I got it. I used `nl', which ignored blank lines. -- Carl Lei (XeCycle) Department of Physics, Shanghai Jiao Tong University OpenPGP public key: 7795E591 Fingerprint: 1FB6 7F1F D45D F681 C845 27F7 8D71 8EC4 7795 E591
Excerpts from XeCycle's message of 2011-09-20 11:01:58 +0200:
Many thanks to all of you, I'm trying them out.
So the point is to never clean the code tree, nor cleaning those compiled objects.
However I think re-configuring is not always needed, which results in lots of rebuilding. So It's better to decide myself when it's needed?
Philipp Überbacher <hollunder@lavabit.com> writes:
Excerpts from XeCycle's message of 2011-09-18 17:33:56 +0200:
Hi, I build Emacs from git quite frequently, about twice a month or so. However the PKGBUILD from AUR rebuilds everything each time, which takes too many time I think. Perhaps Emacs is small, but I think there should be a way to prevent too many rebuilds --- is that possible?
1. Don't use -c (clean)
I already did it.
2. Comment line 47 and 48 (PKGBUILD-git.proto), uncomment if you want a clean build.
Well, this is my PKGBUILD-git.proto:
43 # 44 # BUILD HERE 45 # 46 ./autogen.sh 47 ./configure --prefix=/usr 48 make
Do you mean commenting 46 & 47?
Seems like my line numbering in vim is different from yours, anyway, here's what I meant: .. msg "GIT checkout done or server timeout" msg "Starting build..." # rm -rf "$srcdir/$_gitname-build" # git clone "$srcdir/$_gitname" "$srcdir/$_gitname-build" cd "$srcdir/$_gitname-build" .. Regards, Philipp
On 18 September 2011 17:33, XeCycle <xecycle@gmail.com> wrote:
Hi, I build Emacs from git quite frequently, about twice a month or so. However the PKGBUILD from AUR rebuilds everything each time, which takes too many time I think. Perhaps Emacs is small, but I think there should be a way to prevent too many rebuilds --- is that possible?
-- Carl Lei (XeCycle) Department of Physics, Shanghai Jiao Tong University OpenPGP public key: 7795E591 Fingerprint: 1FB6 7F1F D45D F681 C845 27F7 8D71 8EC4 7795 E591
I don't know which package you are using, but in my opinion the best approach is the following: Look in the PKGBUILD and change "cd $srcdir/some_build_directory" (which is most likely created git clone) to a directory where the sources are downloaded (in most cases it's "$srcdir/$_gitname"). Now when you build package using makepkg, only the updated sources are rebuilt. Generally, it's a good idea to read and understand the PKGBUILD before building. Then it's pretty simple to edit it in way which allows incremental builds. If you would have any problems with editing this particular PKGBUILD to suit your need, you can send the it here and I (or maybe someone else) will try to help you with that. Lukas
2011/9/19 Lukáš Jirkovský <l.jirkovsky@gmail.com>:
On 18 September 2011 17:33, XeCycle <xecycle@gmail.com> wrote:
Hi, I build Emacs from git quite frequently, about twice a month or so. However the PKGBUILD from AUR rebuilds everything each time, which takes too many time I think. Perhaps Emacs is small, but I think there should be a way to prevent too many rebuilds --- is that possible?
I don't know which package you are using, but in my opinion the best approach is the following:
Look in the PKGBUILD and change "cd $srcdir/some_build_directory" (which is most likely created git clone) to a directory where the sources are downloaded (in most cases it's "$srcdir/$_gitname"). Now when you build package using makepkg, only the updated sources are rebuilt.
Generally, it's a good idea to read and understand the PKGBUILD before building. Then it's pretty simple to edit it in way which allows incremental builds.
If you would have any problems with editing this particular PKGBUILD to suit your need, you can send the it here and I (or maybe someone else) will try to help you with that.
i posted the solution i use for my AUR/internal packages awhile back, with the intention of suggesting it as a standardized way to handle this use case: http://mailman.archlinux.org/pipermail/arch-general/2011-July/021078.html ... not all that long of a thread ;-), but i'll highlight a few key benefits over locally storing repositories [in build dir], or other ad hoc solutions: ) uses a targeted fetch instead of a naive clone (for kernel IIRC you get ~50% size reduction, something like ~180MB) ) stores repositories in a known list of locations, with predictable ordering (allows for reuse between package variants) ) generates a lightweight proxy in the event a repository is found but it is read-only (allows for safe use concurrently or within a chroot [mkchrootpkg]) ... this doesn't stop emacs/etc from needing a fresh rebuild each time, but the repo will forever be intact until removed (it was unclear if the OP used `rebuild` in the sense of the software [make] or the repo [clone]) you'd need to add the routines to the target package (they are at the end of the link i provided). this package: http://aur.archlinux.org/packages/py/pyjamas-engine-pythonwebkit/PKGBUILD ... makes use of them, and serves as an example. it's not quite as "simple" or "KISS" as some may prefer, but hey, features/flexibility don't always agree with simplicity ... it's not complicated, but it's not a one-liner either. in the end it saves me a whole 'lotta time, and frankly, what else really matters? -- C Anthony
On Mon, Sep 19, 2011 at 11:36 AM, C Anthony Risinger <anthony@xtfx.me> wrote:
2011/9/19 Lukáš Jirkovský <l.jirkovsky@gmail.com>:
On 18 September 2011 17:33, XeCycle <xecycle@gmail.com> wrote:
Hi, I build Emacs from git quite frequently, about twice a month or so. However the PKGBUILD from AUR rebuilds everything each time, which takes too many time I think. Perhaps Emacs is small, but I think there should be a way to prevent too many rebuilds --- is that possible?
I don't know which package you are using, but in my opinion the best approach is the following:
Look in the PKGBUILD and change "cd $srcdir/some_build_directory" (which is most likely created git clone) to a directory where the sources are downloaded (in most cases it's "$srcdir/$_gitname"). Now when you build package using makepkg, only the updated sources are rebuilt.
Generally, it's a good idea to read and understand the PKGBUILD before building. Then it's pretty simple to edit it in way which allows incremental builds.
If you would have any problems with editing this particular PKGBUILD to suit your need, you can send the it here and I (or maybe someone else) will try to help you with that.
i posted the solution i use for my AUR/internal packages awhile back, with the intention of suggesting it as a standardized way to handle this use case:
http://mailman.archlinux.org/pipermail/arch-general/2011-July/021078.html
... not all that long of a thread ;-), but i'll highlight a few key benefits over locally storing repositories [in build dir], or other ad hoc solutions:
) uses a targeted fetch instead of a naive clone (for kernel IIRC you get ~50% size reduction, something like ~180MB) ) stores repositories in a known list of locations, with predictable ordering (allows for reuse between package variants) ) generates a lightweight proxy in the event a repository is found but it is read-only (allows for safe use concurrently or within a chroot [mkchrootpkg])
... this doesn't stop emacs/etc from needing a fresh rebuild each time, but the repo will forever be intact until removed (it was unclear if the OP used `rebuild` in the sense of the software [make] or the repo [clone])
you'd need to add the routines to the target package (they are at the end of the link i provided). this package:
http://aur.archlinux.org/packages/py/pyjamas-engine-pythonwebkit/PKGBUILD
... makes use of them, and serves as an example. it's not quite as "simple" or "KISS" as some may prefer, but hey, features/flexibility don't always agree with simplicity ... it's not complicated, but it's not a one-liner either. in the end it saves me a whole 'lotta time, and frankly, what else really matters?
... ah forgot to mention one other thing i found to be useful; said routines introduce the concept of a `_gitspec` variable -- this var falls back to `_gitname` (the branch to be checked out) if missing, and is used when the working directory is being updated: git --git-dir="${g}" --work-tree="${w}" read-tree --reset -u "${_gitspec:-${_gitname}}" ... setting it allows you build any arbitrary revision (must exist on the branch your targeting -- targeted fetch means you don't have the whole repo, only the specific branch you need). i've used this to quickly rebuild stuff when HEAD turned out to be busted, bisecting for bugs, etc. -- C Anthony
On Sun, Sep 18, 2011 at 11:33 PM, XeCycle <xecycle@gmail.com> wrote:
Hi, I build Emacs from git quite frequently, about twice a month or so. However the PKGBUILD from AUR rebuilds everything each time, which takes too many time I think. Perhaps Emacs is small, but I think there should be a way to prevent too many rebuilds --- is that possible?
-- Carl Lei (XeCycle) Department of Physics, Shanghai Jiao Tong University OpenPGP public key: 7795E591 Fingerprint: 1FB6 7F1F D45D F681 C845 27F7 8D71 8EC4 7795 E591
yaourt has an option to compile vcs packages in a separate directory rather than the default /tmp directory. in this way, the download part in every recompile can be minimized. just fyi.
On Mon, Sep 19, 2011 at 6:20 PM, Auguste Pop <auguste@gmail.com> wrote:
yaourt has an option to compile vcs packages in a separate directory rather than the default /tmp directory. in this way, the download part in every recompile can be minimized. just fyi.
btw I added exactly this functionality to packer. cheers! mar77i
hi Nobody mentionned one quite easy solution: use ccache It's built-in in makepkg so you have only to remove the ! before ccache in BUILDENV variable in /etc/makepkg.conf and install also ccache of course. Then for any C or C++ program that you compile with makepkg, every file that has already been compiled once is saved. If ever you make a mistake in a PKGBUILD and have to recompile a twice a software, ccache is your saver because the second time is way faster. Just try it. Note, as emacs might be written in something else than C or C++ like some sort of lisp that might not work. or may be , it could work; I don't know emacs. Vim rocks \o/ You might want to clean your ccache cache frome time to times too.
On Mon, Sep 19, 2011 at 12:59 PM, solsTiCe d'Hiver <solstice.dhiver@gmail.com> wrote:
Note, as emacs might be written in something else than C or C++ like some sort of lisp that might not work. or may be , it could work; I don't know emacs. Vim rocks \o/
Emacs core, which is a lisp interpreter, is written in C.
participants (12)
-
Auguste Pop
-
C Anthony Risinger
-
Jeremiah Dodds
-
Jesse Jaara
-
Lukas Fleischer
-
Lukáš Jirkovský
-
Martti Kühne
-
Philipp Überbacher
-
Ray Rashif
-
solsTiCe d'Hiver
-
Thomas Dziedzic
-
XeCycle