[aur-general] Augeas request
I have an odd request with regards to an aur package - augeas: https://aur.archlinux.org/packages.php?ID=18533 I have been working toward porting an extremely powerful virtualization application to ArchLinux - Libguestfs: http://libguestfs.org/ The libguestfs package will also enable a number of other virtualization packages to be packaged for Archlinux. The port of Libguestfs has required a number of changes to the upstream project and I have had to submit a fair amount of code to make it possible, and the upstream developers have been more than helpful in this effort. The problem I am facing now is one that I do not yet have the power to repair, as I am not yet a TU and I doubt I have enough clout yet to apply. You see, the build process needs to create a tiny virtual machine image, (the image is quite remarkable, a vm image for Arch in less than 50K) but the process to create the image requires that all packages for the image be available as binary downloads from a main Arch repo, aka core, extra and community. This brings me to my request, the augeas package. Augeas and a number of other packages which are only available in the AUR are required by this image, including hfsutils and zfs-fuse. While hfsutils and zfs-fuse are "nice to have" in the libguestfs image, and can be safely omitted, augeas is a hard requirement. So, since I can not do this myself yet I will request that a TU take up this small responsibility and maintain augeas in community so I can finish the port of libguestfs to Arch. Once I am a TU I will be more than take it off your hands! I have spent the last 3 months working tirelessly in my effort to bring better kvm/libvirt support to Arch, I am the author of varch: https://aur.archlinux.org/packages.php?ID=42428 and I have more software in the wings to bring better virtualization support to Arch that requires libguestfs, please help me out! Thanks Thomas S Hatch - thatch45
2010/12/9 Thomas S Hatch <thatch45@gmail.com>
You see, the build process needs to create a tiny virtual machine image, (the image is quite remarkable, a vm image for Arch in less than 50K) but the process to create the image requires that all packages for the image be available as binary downloads from a main Arch repo, aka core, extra and community.
Could you explain why it requires these pkgs to be in the repositorys? Im quite sure that it isint reallya MUST -- (\_ /) copy the bunny to your profile (0.o ) to help him achieve world domination. (> <) come join the dark side. /_|_\ (we have cookies.)
On Thu, Dec 9, 2010 at 9:01 AM, jesse jaara <jesse.jaara@gmail.com> wrote:
2010/12/9 Thomas S Hatch <thatch45@gmail.com>
You see, the build process needs to create a tiny virtual machine image, (the image is quite remarkable, a vm image for Arch in less than 50K) but the process to create the image requires that all packages for the image
be
available as binary downloads from a main Arch repo, aka core, extra and community.
Could you explain why it requires these pkgs to be in the repositorys? Im quite sure that it isint reallya MUST
-- (\_ /) copy the bunny to your profile (0.o ) to help him achieve world domination. (> <) come join the dark side. /_|_\ (we have cookies.)
Ok, the process of building the libguestfs virtual machine image, called "supermin", must be executed as an unprivileged user during the libguestfs make process. The supermin build process is managed by a program called febootstrap, I had to write an additional module for febootstrap to support Arch - (and the upstream devs rewrote febootstrap to make it possible - because they are awesome). but since the make process needs to run as an unprivileged user febootstrap needs to download the required packages using pacman and then use tar to extract them into place. I could get around it by heavily modifying febootstrap so that I can pass a third party repo into it that I maintain, but the libguestfs guys really don't want those patches upstream and it will make a real mess of things. So what it boils down to is that the make process requires pacman, and only has access to the main repos, and there is only one package missing from said repos that is required - augeas. -Tom
On Thu, Dec 9, 2010 at 9:20 AM, Thomas S Hatch <thatch45@gmail.com> wrote:
On Thu, Dec 9, 2010 at 9:01 AM, jesse jaara <jesse.jaara@gmail.com> wrote:
2010/12/9 Thomas S Hatch <thatch45@gmail.com>
You see, the build process needs to create a tiny virtual machine image, (the image is quite remarkable, a vm image for Arch in less than 50K)
but
the process to create the image requires that all packages for the image be available as binary downloads from a main Arch repo, aka core, extra and community.
Could you explain why it requires these pkgs to be in the repositorys? Im quite sure that it isint reallya MUST
-- (\_ /) copy the bunny to your profile (0.o ) to help him achieve world domination. (> <) come join the dark side. /_|_\ (we have cookies.)
Ok, the process of building the libguestfs virtual machine image, called "supermin", must be executed as an unprivileged user during the libguestfs make process. The supermin build process is managed by a program called febootstrap, I had to write an additional module for febootstrap to support Arch - (and the upstream devs rewrote febootstrap to make it possible - because they are awesome). but since the make process needs to run as an unprivileged user febootstrap needs to download the required packages using pacman and then use tar to extract them into place.
I could get around it by heavily modifying febootstrap so that I can pass a third party repo into it that I maintain, but the libguestfs guys really don't want those patches upstream and it will make a real mess of things.
So what it boils down to is that the make process requires pacman, and only has access to the main repos, and there is only one package missing from said repos that is required - augeas.
-Tom
Upon closer inspection, the backdoor I was going to use to enable a third party repo has been removed from the upstream code, seems my "messy" option is no longer an option. I am still looking for another solution, but as of right now the inclusion of augeas in community is still the olny option I can see. -Tom
Thomas S Hatch wrote:
Ok, the process of building the libguestfs virtual machine image, called "supermin", must be executed as an unprivileged user during the libguestfs make process. The supermin build process is managed by a program called febootstrap, I had to write an additional module for febootstrap to support Arch - (and the upstream devs rewrote febootstrap to make it possible - because they are awesome). but since the make process needs to run as an unprivileged user febootstrap needs to download the required packages using pacman and then use tar to extract them into place.
I could get around it by heavily modifying febootstrap so that I can pass a third party repo into it that I maintain, but the libguestfs guys really don't want those patches upstream and it will make a real mess of things.
So what it boils down to is that the make process requires pacman, and only has access to the main repos, and there is only one package missing from said repos that is required - augeas.
-Tom
Upon closer inspection, the backdoor I was going to use to enable a third party repo has been removed from the upstream code, seems my "messy" option is no longer an option. I am still looking for another solution, but as of right now the inclusion of augeas in community is still the olny option I can see.
-Tom
Hi Tom, I'm curious about the febootstrap process. Is Pacman hard-coded in the module that you wrote? If it is, is there no way to generalize the process in such a way that it can call an external script based on the installation context and let that script handle everything itself? I understand that there is probably a good deal of complexity involved, but from the description it sounds as though the design could be improved (read: made more flexible). I wouldn't want to be the author of a project that requires considerable code changes in order to support different distros. It would require constant maintenance. It may well be unavoidable, but I would take it as a challenge to make it generic. Does the febootstrap process need to handle non-native architectures, i.e. does it need to be able to install binary packages that it can't build locally itself (if it were even possible to build the local packages). As a test case, I wrote a very small script that can retrieve binary packages from pacman or, failing that, build the binary package from the AUR and move it to the current directory, i.e. regardless of where the package is (repos, AUR), you end up with the binary package in the current dir. Obviously it's inherently dangerous as it blindly trusts the packages, but if you only need a few packages that you maintain yourself then it shouldn't be a problem. It's pure bash and very small (wc test.sh -> 28 87 731 test.sh) so I wonder if it could be worked into a general solution. There are, of course, several AUR helpers that could do the job, provided that the febootstrap could handle them and that the architecture limitation is irrelevant, but they would then become makedeps at least. Of course, augeas will probably get moved to [community] thus circumventing such concerns, but having a more flexible system would probably be better in the long run. /Xyne
On Thu, Dec 9, 2010 at 12:14 PM, Xyne <xyne@archlinux.ca> wrote:
Thomas S Hatch wrote:
Ok, the process of building the libguestfs virtual machine image, called "supermin", must be executed as an unprivileged user during the libguestfs make process. The supermin build process is managed by a program called febootstrap, I had to write an additional module for febootstrap to support Arch - (and the upstream devs rewrote febootstrap to make it possible - because they are awesome). but since the make process needs to run as an unprivileged user febootstrap needs to download the required packages using pacman and then use tar to extract them into place.
I could get around it by heavily modifying febootstrap so that I can pass a third party repo into it that I maintain, but the libguestfs guys really don't want those patches upstream and it will make a real mess of things.
So what it boils down to is that the make process requires pacman, and only has access to the main repos, and there is only one package missing from said repos that is required - augeas.
-Tom
Upon closer inspection, the backdoor I was going to use to enable a third party repo has been removed from the upstream code, seems my "messy" option is no longer an option. I am still looking for another solution, but as of right now the inclusion of augeas in community is still the olny option I can see.
-Tom
Hi Tom,
I'm curious about the febootstrap process. Is Pacman hard-coded in the module that you wrote? If it is, is there no way to generalize the process in such a way that it can call an external script based on the installation context and let that script handle everything itself?
I understand that there is probably a good deal of complexity involved, but from the description it sounds as though the design could be improved (read: made more flexible). I wouldn't want to be the author of a project that requires considerable code changes in order to support different distros. It would require constant maintenance. It may well be unavoidable, but I would take it as a challenge to make it generic.
Does the febootstrap process need to handle non-native architectures, i.e. does it need to be able to install binary packages that it can't build locally itself (if it were even possible to build the local packages). As a test case, I wrote a very small script that can retrieve binary packages from pacman or, failing that, build the binary package from the AUR and move it to the current directory, i.e. regardless of where the package is (repos, AUR), you end up with the binary package in the current dir.
Obviously it's inherently dangerous as it blindly trusts the packages, but if you only need a few packages that you maintain yourself then it shouldn't be a problem. It's pure bash and very small (wc test.sh -> 28 87 731 test.sh) so I wonder if it could be worked into a general solution.
There are, of course, several AUR helpers that could do the job, provided that the febootstrap could handle them and that the architecture limitation is irrelevant, but they would then become makedeps at least.
Of course, augeas will probably get moved to [community] thus circumventing such concerns, but having a more flexible system would probably be better in the long run.
/Xyne
I completely agree Xyne, flexibility is critical in theses realms of development. Originally libguestfs was developed for redhat and the problem of distro porting has long been inherently difficult, even getting the supermin appliance to build in the scope of koji (fedora/redhat package build system) was a serious problem. Originally porting it to Arch looked downright impossible, and I presented the same arguments you have presented here to the libguestfs team. They took the supermin build process and developed a way to support other distros, thus making it far more flexible. Right now to port this to another distro you need to write an ocaml module to support the package manager and distro. Honestly I had not considered scripting out that portion, and that would work, either to run the commands entirely in ocaml or to script them out. The more I mull this over the more I think it is viable to use the aur package, although I hope I don't need to as it adds extra complexity and like you say, if augeas is in community the problem is circumvented. I would still be interested in your script, I still lack a great deal of pacman fu! -Tom
It still needs a lot of refinement, but here is the ocaml febootstrap_pacman module if you are interested: http://git.annexia.org/?p=febootstrap.git;a=blob;f=febootstrap_pacman.ml;h=5... The more I look at it the more I see better ways of handling the situations presented in the module, as per usual, I love suggestions :) - oh, and I don't really know ocaml, I am learning it to make the port :) On Thu, Dec 9, 2010 at 12:33 PM, Thomas S Hatch <thatch45@gmail.com> wrote:
On Thu, Dec 9, 2010 at 12:14 PM, Xyne <xyne@archlinux.ca> wrote:
Thomas S Hatch wrote:
because they are awesome). but since the make process needs to run as an unprivileged user febootstrap needs to download the required packages using pacman and then use tar to extract them into place.
I could get around it by heavily modifying febootstrap so that I can pass a third party repo into it that I maintain, but the libguestfs guys really don't want those patches upstream and it will make a real mess of things.
So what it boils down to is that the make process requires pacman, and only has access to the main repos, and there is only one package missing from said repos that is required - augeas.
-Tom
Upon closer inspection, the backdoor I was going to use to enable a third party repo has been removed from the upstream code, seems my "messy" option is no longer an option. I am still looking for another solution, but as of right now the inclusion of augeas in community is still the olny option I can see.
-Tom
Ok, the process of building the libguestfs virtual machine image, called "supermin", must be executed as an unprivileged user during the libguestfs make process. The supermin build process is managed by a program called febootstrap, I had to write an additional module for febootstrap to support Arch - (and the upstream devs rewrote febootstrap to make it possible
Hi Tom,
I'm curious about the febootstrap process. Is Pacman hard-coded in the module that you wrote? If it is, is there no way to generalize the process in such a way that it can call an external script based on the installation context and let that script handle everything itself?
I understand that there is probably a good deal of complexity involved, but from the description it sounds as though the design could be improved (read: made more flexible). I wouldn't want to be the author of a project that requires considerable code changes in order to support different distros. It would require constant maintenance. It may well be unavoidable, but I would take it as a challenge to make it generic.
Does the febootstrap process need to handle non-native architectures, i.e. does it need to be able to install binary packages that it can't build locally itself (if it were even possible to build the local packages). As a test case, I wrote a very small script that can retrieve binary packages from pacman or, failing that, build the binary package from the AUR and move it to the current directory, i.e. regardless of where the package is (repos, AUR), you end up with the binary package in the current dir.
Obviously it's inherently dangerous as it blindly trusts the packages, but if you only need a few packages that you maintain yourself then it shouldn't be a problem. It's pure bash and very small (wc test.sh -> 28 87 731 test.sh) so I wonder if it could be worked into a general solution.
There are, of course, several AUR helpers that could do the job, provided that the febootstrap could handle them and that the architecture limitation is irrelevant, but they would then become makedeps at least.
Of course, augeas will probably get moved to [community] thus circumventing such concerns, but having a more flexible system would probably be better in the long run.
/Xyne
I completely agree Xyne, flexibility is critical in theses realms of development. Originally libguestfs was developed for redhat and the problem of distro porting has long been inherently difficult, even getting the supermin appliance to build in the scope of koji (fedora/redhat package build system) was a serious problem. Originally porting it to Arch looked downright impossible, and I presented the same arguments you have presented here to the libguestfs team. They took the supermin build process and developed a way to support other distros, thus making it far more flexible.
Right now to port this to another distro you need to write an ocaml module to support the package manager and distro. Honestly I had not considered scripting out that portion, and that would work, either to run the commands entirely in ocaml or to script them out.
The more I mull this over the more I think it is viable to use the aur package, although I hope I don't need to as it adds extra complexity and like you say, if augeas is in community the problem is circumvented.
I would still be interested in your script, I still lack a great deal of pacman fu!
-Tom
You know, this is one of the situations where I am kind of embarrassed I even brought this up. I can solve this problem much more cleanly with bacman, and it will clean up a number of aspects of the build process. Ok, do me a favor, and next time I poke my head out, don't remember mas as a fool :) -Thomas Hatch On Thu, Dec 9, 2010 at 12:36 PM, Thomas S Hatch <thatch45@gmail.com> wrote:
It still needs a lot of refinement, but here is the ocaml febootstrap_pacman module if you are interested:
http://git.annexia.org/?p=febootstrap.git;a=blob;f=febootstrap_pacman.ml;h=5...
The more I look at it the more I see better ways of handling the situations presented in the module, as per usual, I love suggestions :) - oh, and I don't really know ocaml, I am learning it to make the port :)
On Thu, Dec 9, 2010 at 12:33 PM, Thomas S Hatch <thatch45@gmail.com>wrote:
On Thu, Dec 9, 2010 at 12:14 PM, Xyne <xyne@archlinux.ca> wrote:
Thomas S Hatch wrote:
because they are awesome). but since the make process needs to run as an unprivileged user febootstrap needs to download the required packages using pacman and then use tar to extract them into place.
I could get around it by heavily modifying febootstrap so that I can pass a third party repo into it that I maintain, but the libguestfs guys really don't want those patches upstream and it will make a real mess of things.
So what it boils down to is that the make process requires pacman, and only has access to the main repos, and there is only one package missing from said repos that is required - augeas.
-Tom
Upon closer inspection, the backdoor I was going to use to enable a third party repo has been removed from the upstream code, seems my "messy" option is no longer an option. I am still looking for another solution, but as of right now the inclusion of augeas in community is still the olny option I can see.
-Tom
Ok, the process of building the libguestfs virtual machine image, called "supermin", must be executed as an unprivileged user during the libguestfs make process. The supermin build process is managed by a program called febootstrap, I had to write an additional module for febootstrap to support Arch - (and the upstream devs rewrote febootstrap to make it possible
Hi Tom,
I'm curious about the febootstrap process. Is Pacman hard-coded in the module that you wrote? If it is, is there no way to generalize the process in such a way that it can call an external script based on the installation context and let that script handle everything itself?
I understand that there is probably a good deal of complexity involved, but from the description it sounds as though the design could be improved (read: made more flexible). I wouldn't want to be the author of a project that requires considerable code changes in order to support different distros. It would require constant maintenance. It may well be unavoidable, but I would take it as a challenge to make it generic.
Does the febootstrap process need to handle non-native architectures, i.e. does it need to be able to install binary packages that it can't build locally itself (if it were even possible to build the local packages). As a test case, I wrote a very small script that can retrieve binary packages from pacman or, failing that, build the binary package from the AUR and move it to the current directory, i.e. regardless of where the package is (repos, AUR), you end up with the binary package in the current dir.
Obviously it's inherently dangerous as it blindly trusts the packages, but if you only need a few packages that you maintain yourself then it shouldn't be a problem. It's pure bash and very small (wc test.sh -> 28 87 731 test.sh) so I wonder if it could be worked into a general solution.
There are, of course, several AUR helpers that could do the job, provided that the febootstrap could handle them and that the architecture limitation is irrelevant, but they would then become makedeps at least.
Of course, augeas will probably get moved to [community] thus circumventing such concerns, but having a more flexible system would probably be better in the long run.
/Xyne
I completely agree Xyne, flexibility is critical in theses realms of development. Originally libguestfs was developed for redhat and the problem of distro porting has long been inherently difficult, even getting the supermin appliance to build in the scope of koji (fedora/redhat package build system) was a serious problem. Originally porting it to Arch looked downright impossible, and I presented the same arguments you have presented here to the libguestfs team. They took the supermin build process and developed a way to support other distros, thus making it far more flexible.
Right now to port this to another distro you need to write an ocaml module to support the package manager and distro. Honestly I had not considered scripting out that portion, and that would work, either to run the commands entirely in ocaml or to script them out.
The more I mull this over the more I think it is viable to use the aur package, although I hope I don't need to as it adds extra complexity and like you say, if augeas is in community the problem is circumvented.
I would still be interested in your script, I still lack a great deal of pacman fu!
-Tom
I don't know if it would help but I have created an ocaml module that you could use to interface with libalpm. The github repo is at https://github.com/juster/ocaml-alpm but it isn't completely finished. It works and is comprehensive but has no docs and no Makefile. Making packages from the AUR isn't very complicated if you already know which packages are from the AUR beforehand. You just download the source package (wget/curl), extract it, run makepkg in the extracted directory, and do what you want with the binary package. You loop/recurse over a list of package names doing this for each one. I suppose this would be a list of one single element: augeas? Having a needed package in the AUR doesn't seem like a deal breaker unless there is some sort of upstream policy preventing you from using the AUR. -- -Justin
On Thu, Dec 9, 2010 at 10:00 PM, Justin Davis <jrcd83@gmail.com> wrote:
I don't know if it would help but I have created an ocaml module that you could use to interface with libalpm. The github repo is at https://github.com/juster/ocaml-alpm but it isn't completely finished. It works and is comprehensive but has no docs and no Makefile.
Making packages from the AUR isn't very complicated if you already know which packages are from the AUR beforehand. You just download the source package (wget/curl), extract it, run makepkg in the extracted directory, and do what you want with the binary package. You loop/recurse over a list of package names doing this for each one. I suppose this would be a list of one single element: augeas?
Having a needed package in the AUR doesn't seem like a deal breaker unless there is some sort of upstream policy preventing you from using the AUR. -- -Justin
Thanks, I will need to look into this, but I have found a solution! I will have libguestfs packages up shortly! I didn't think it would be wise to build a package from aur during the build of a package, so I hardcoded in the use of bacman. As for your ocaml package, I will have to take a look! -Thomas Hatch
Thomas S Hatch wrote:
I would still be interested in your script, I still lack a great deal of pacman fu!
Here it is, just for reference: http://xyne.archlinux.ca/scripts/pacman/#pacget Thomas S Hatch wrote:
It still needs a lot of refinement, but here is the ocaml febootstrap_pacman module if you are interested: http://git.annexia.org/?p=febootstrap.git;a=blob;f=febootstrap_pacman.ml;h=5...
The more I look at it the more I see better ways of handling the situations presented in the module, as per usual, I love suggestions :) - oh, and I don't really know ocaml, I am learning it to make the port :)
I don't know ocaml either but it's similar enough to Haskell that I think I can follow it. Looking at it, I like the modular approach that the devs have taken. It should provide all the flexibility you could need. I'm glad that you've found a solution.
On Fri, Dec 10, 2010 at 7:43 AM, Xyne <xyne@archlinux.ca> wrote:
Thomas S Hatch wrote:
I would still be interested in your script, I still lack a great deal of pacman fu!
Here it is, just for reference:
http://xyne.archlinux.ca/scripts/pacman/#pacget
Thomas S Hatch wrote:
It still needs a lot of refinement, but here is the ocaml febootstrap_pacman module if you are interested:
http://git.annexia.org/?p=febootstrap.git;a=blob;f=febootstrap_pacman.ml;h=5...
The more I look at it the more I see better ways of handling the
situations
presented in the module, as per usual, I love suggestions :) - oh, and I don't really know ocaml, I am learning it to make the port :)
I don't know ocaml either but it's similar enough to Haskell that I think I can follow it. Looking at it, I like the modular approach that the devs have taken. It should provide all the flexibility you could need.
I'm glad that you've found a solution.
Thanks Xyne, some of what your script does could make this run better! I did get it all working last night, so I will have it in the AUR soon. -Tom
participants (4)
-
jesse jaara
-
Justin Davis
-
Thomas S Hatch
-
Xyne