[arch-general] GHC 7.8.1 packaging decisions for Arch Linux
Hello all, With the arrival of ghc 7.8.1 [0], I would like to address the following problems with a restructuring of how we treat haskell packages in archlinux: Problem 1: Updating any haskell package has been delayed until we bump ghc. Explanation: ghc is unable to produce a library that has a stable abi. In other words, if a library gets rebuilt (even if it's the same exact source), we will need to rebuild all package that depend on it, and this would in turn a messy rebuild for any kind of rebuild. Problem 2: Users are confused whether they should install packages from the repos or using cabal-install. This in turn sometimes causes them to install some packages from official repos, some from the aur, and some using cabal-install Explanation: Packages should ideally be installed from one place, either using repos/aur or cabal-install. Said that, it is entirely possible to use both repos/aur and cabal-install but for simplicity, one source is preferred. Problem 3: Our repos are unable to package every haskell package Explanation: hackage has over 5000 haskell packages in it currently. There is no way we will be able to support that many packages in our official repos without some kind of automation. And even if it would be possible, it would not necessarily be desirable. In addition, it is entirely possible to have multiple versions of the same package installed at the same time. Cabal has a very elaborate dependency solver in order to find versions that will work for all package versions installed. In addition, cabal-install 1.18 (the latest released) has added support for sandboxes, meaning that when developing haskell programs, you are now able to setup a sandbox environment that will only contain the haskell packages you want. I would like to propose the following changes to address the mentioned problems and having the ability to use sandboxes to isolate development environments. Change 1: Move every haskell related package out of [extra] into [community] except ghc and cabal-install. This includes the following 8 packages: haskell-http, haskell-mtl, haskell-network, haskell-parsec, haskell-random, haskell-text, haskell-transformers, haskell-zlib Explanation: These packages are only required to build cabal-install. Since we converted the cabal-install package to use the bootstrap script that comes with it, we no longer depend on these packages for anything in [extra]. Change 2: Make a news item stating that cabal-install is now the recommended way to install haskell packages. This wouldn't pollute the filesystem since cabal-install installs packages to the ~/.cabal directory by default. We might need to include a tip sheet about how you would handle ghc updates since it requires extra user steps. Change 3: Support users who are unable to install haskell packages that do not compile under archlinux. This would require working with the user and upstream to open up tickets and write patches for programs. At the very least we can work with the user if they do not to open up upstream bug reports and track them in our own bug tracker. There might be some packages which we would probably consider unsupported like bindings to packages that are not in the supported repos and packages that have no upstream activity and ones that are effectively unmaintained. Hopefully with the proposed changes, we can improve our user experience with using official tools to maintain haskell packages. I realize that some changes might be controversial especially change 3. This is why I posted this here in order to discuss the possibilities. I would also love to hear some opinions from vegai since he has some previous experience package haskell and has had similar thoughts in the past. Also if you're not a developer or trusted user, I would also like to hear your opinion since it would affect you also. Thanks for your time, - Tom [0] - https://github.com/ghc/ghc/releases/tag/ghc-7.8.1-release
Tom, I might come across as very critical below, but I'm really not. As you probably realise I've also thought a bit about related questions and I'm just really interested in your thoughts and answers. On Wed, Apr 9, 2014 at 7:27 AM, Thomas Dziedzic <gostrc@gmail.com> wrote:
Hello all,
With the arrival of ghc 7.8.1 [0], I would like to address the following problems with a restructuring of how we treat haskell packages in archlinux: [...] Change 1: Move every haskell related package out of [extra] into [community] except ghc and cabal-install. This includes the following 8 packages: haskell-http, haskell-mtl, haskell-network, haskell-parsec, haskell-random, haskell-text, haskell-transformers, haskell-zlib Explanation: These packages are only required to build cabal-install. Since we converted the cabal-install package to use the bootstrap script that comes with it, we no longer depend on these packages for anything in [extra].
I'm guessing this means cabal-install now is the only package outside of [community] that uses ghc to build. Is that right? Is the plan then that any future tools (i.e. non-libraries) implemented in Haskell would go into [community]? <devil advocate> There is nothing that say one HAS to wait for a ghc upgrade in order to provide newer versions of Haskell packages. As you point all that's needed is a rebuild of all the packages that depend on the upgraded one. If that's messy it sounds like you are using bad tools to handle upgrading. Are you really suggesting ArchLinux abandon packaging a whole class of software just because the tools are inadequate? </devils advocate>
Change 2: Make a news item stating that cabal-install is now the recommended way to install haskell packages. This wouldn't pollute the filesystem since cabal-install installs packages to the ~/.cabal directory by default. We might need to include a tip sheet about how you would handle ghc updates since it requires extra user steps.
It should be noted that cabal-install isn't a package manager in the true sense[1]. I'm not sure this is an argument against making the change you propose, but it's worth noting. There are quite a few other language/frameworks that have language-specific build/package systems, Python, Ruby, Perl, node.js... Are Python developers on Arch pointed towards using pip to install Python libs? I think sometimes the right thing is to point users to another package manager, e.g. packaging vim scripts for system wide installation is a bit silly, since installing a vim script affects ALL users on the system. So doing that would require providing some sort of vim-script manager to users. Then there's very little difference compared to just telling users to use Vundle/Pathogen/whatever directly instead. However, this isn't the case for Haskell/GHC...
Change 3: Support users who are unable to install haskell packages that do not compile under archlinux. This would require working with the user and upstream to open up tickets and write patches for programs. At the very least we can work with the user if they do not to open up upstream bug reports and track them in our own bug tracker. There might be some packages which we would probably consider unsupported like bindings to packages that are not in the supported repos and packages that have no upstream activity and ones that are effectively unmaintained.
How do you envision this actually working? The set of packages in [extra]/[community] is rather small today, in the order of 3 dozen, so does this mean that users are already turning to the Arch devs when they are having problems compiling Haskell packages? /M [1]: http://is.gd/vzse5G- -- Magnus Therning OpenPGP: 0xAB4DFBA4 email: magnus@therning.org jabber: magnus@therning.org twitter: magthe http://therning.org/magnus
Tom,
I might come across as very critical below, but I'm really not. As you probably realise I've also thought a bit about related questions and I'm just really interested in your thoughts and answers.
On Wed, Apr 9, 2014 at 7:27 AM, Thomas Dziedzic <gostrc@gmail.com> wrote:
Hello all,
With the arrival of ghc 7.8.1 [0], I would like to address the following problems with a restructuring of how we treat haskell packages in archlinux: [...] Change 1: Move every haskell related package out of [extra] into [community] except ghc and cabal-install. This includes the following 8 packages: haskell-http, haskell-mtl, haskell-network, haskell-parsec, haskell-random, haskell-text, haskell-transformers, haskell-zlib Explanation: These packages are only required to build cabal-install. Since we converted the cabal-install package to use the bootstrap script that comes with it, we no longer depend on these packages for anything in [extra].
I'm guessing this means cabal-install now is the only package outside of [community] that uses ghc to build. Is that right?
Is the plan then that any future tools (i.e. non-libraries) implemented in Haskell would go into [community]?
I would like to keep XMonad/XMobar in [community] it does seem to take up a big chunk of the haskell-* packages we have in our repos. But I've never ran into real big issues packaging haskell libraries, one minor issue is
On Wed, Apr 9, 2014 at 9:07 AM, Magnus Therning <magnus@therning.org> wrote: that the developers tend to oversplit packages for example haskell-data-default-* . This really makes packaging haskell libraries annoying. It should be noted that cabal-install isn't a package manager in the
true sense[1]. I'm not sure this is an argument against making the change you propose, but it's worth noting.
There are quite a few other language/frameworks that have language-specific build/package systems, Python, Ruby, Perl, node.js... Are Python developers on Arch pointed towards using pip to install Python libs?
I think sometimes the right thing is to point users to another package manager, e.g. packaging vim scripts for system wide installation is a bit silly, since installing a vim script affects ALL users on the system. So doing that would require providing some sort of vim-script manager to users. Then there's very little difference compared to just telling users to use Vundle/Pathogen/whatever directly instead. However, this isn't the case for Haskell/GHC...
I would prefer that we don't package vim plugins or firefox extensions. Firefox has it's own extension manager and vim has a lot of solutions which work better then pacman.
not compile under archlinux. This would require working with the user and upstream to open up tickets and write patches for programs. At the very least we can work with the user if they do not to open up upstream bug reports and track them in our own bug tracker. There might be some
which we would probably consider unsupported like bindings to packages
Change 3: Support users who are unable to install haskell packages that do packages that
are not in the supported repos and packages that have no upstream activity and ones that are effectively unmaintained.
How do you envision this actually working? The set of packages in [extra]/[community] is rather small today, in the order of 3 dozen, so does this mean that users are already turning to the Arch devs when they are having problems compiling Haskell packages?
How many haskell developers actually use our packages in the repos/aur rather then using caba-install?
-- Jelle van der Waa
On Wed, Apr 9, 2014 at 12:35 AM, Jelle van der Waa <jelle@vdwaa.nl> wrote:
I would like to keep XMonad/XMobar in [community] it does seem to take up a big chunk of the haskell-* packages we have in our repos. But I've never ran into real big issues packaging haskell libraries, one minor issue is that the developers tend to oversplit packages for example haskell-data-default-* . This really makes packaging haskell libraries annoying.
I don't want to prevent people from maintaining haskell packages in supported repos. I just want to make cabal-install a path with which people don't have to think twice about.
I would prefer that we don't package vim plugins or firefox extensions. Firefox has it's own extension manager and vim has a lot of solutions which work better then pacman
I also share this belief about haskell. Both pacman and cabal-install have their own pros and cons, but for my personally, I find that cabal-install has more benefit to me personally for haskell packages.
How many haskell developers actually use our packages in the repos/aur rather then using caba-install?
I can only speak from personal experience, but I have used cabal-install exclusively for a couple of years now not just for development but for also installing tools like xmonad. My guess is that given the limited supply of packages in our repos, I can guestimate 0 use it for development but there might be a lot of users that actually use the haskell tools like xmonad.
On Wed, Apr 9, 2014 at 12:07 AM, Magnus Therning <magnus@therning.org>wrote:
I'm guessing this means cabal-install now is the only package outside of [community] that uses ghc to build. Is that right?
That would be correct.
Is the plan then that any future tools (i.e. non-libraries) implemented in Haskell would go into [community]?
This would also be correct. I believe that most people who use packages in our supported repos don't actually use the haskell libraries themselves, but rather the tools that depend on them. (e.g. xmonad) I am not against keeping these tools around and their dependencies if someone wants to maintain them, but I personally have no interest in maintaining them myself.
<devil advocate> There is nothing that say one HAS to wait for a ghc upgrade in order to provide newer versions of Haskell packages. As you point all that's needed is a rebuild of all the packages that depend on the upgraded one. If that's messy it sounds like you are using bad tools to handle upgrading. Are you really suggesting ArchLinux abandon packaging a whole class of software just because the tools are inadequate? </devils advocate>
I am aware that the way we are packaging haskell packages is non optimal and could be automated away. I already have a tool that automates the rebuild of [extra] packages, but since I don't/want to maintain any packages that aren't dependencies of cabal-install, I haven't looked into automating those builds. I don't want to focus solely on this point though because I just felt like I should make people aware of this problem. This isn't the only reason why I want to move away from haskell packages in pacman. It should be noted that cabal-install isn't a package manager in the
true sense[1]. I'm not sure this is an argument against making the change you propose, but it's worth noting.
I agree. Cabal does lack even basic features that other people would expect from a package manager. For example, upgrades, removal. Although there are ways to have the same effect as upgrading since you can have multiple versions of the same package installed, you could just install the same package after a cabal update. That's why I suggested writing up a tip sheet on basic cabal usage for people who aren't as used to cabal as others.
There are quite a few other language/frameworks that have language-specific build/package systems, Python, Ruby, Perl, node.js... Are Python developers on Arch pointed towards using pip to install Python libs?
I think that the languages you mentioned could be split up into 2 categories. Python and Perl have a large number of interested developers and trusted users maintaining those packages. Python has over 700 packages, and Perl has over 500 packages and so, there is a lot of interest in those languages. On the other hand, you have Haskell, Ruby, and Node.Js where they have a smaller following. Haskell has about 90 packages (including i686/x86_64), Ruby has about 60, and Node.Js has approximately none. The popular languages have a critical mass which I believe lets people get away with using the pacman package manager. So for languages like python and perl, it might make sense to use pacman. But for languages like haskell, ruby, and node.js I believe using the system package manager leaves a lot to be desired.
How do you envision this actually working?
I would like to officially announce that there are 2 paths of maintaining haskell packages on your system. One would be using cabal-install and I would encourage users to try using it. The other would be to use whatever is in the supported repositories. I would also like to officially announce that one or the other is supported, but they should be mutually exclusive.
The set of packages in [extra]/[community] is rather small today, in the order of 3 dozen, so does this mean that users are already turning to the Arch devs when they are having problems compiling Haskell packages?
No. I believe that most users actually know what the correct path is. I just want to make it officially announced rather than having any potential confusion on which path to take and which paths are supported.
On 2014-04-09 09:07, Magnus Therning wrote:
Change 2: Make a news item stating that cabal-install is now the recommended way to install haskell packages. This wouldn't pollute the filesystem since cabal-install installs packages to the ~/.cabal directory by default. We might need to include a tip sheet about how you would handle ghc updates since it requires extra user steps.
It should be noted that cabal-install isn't a package manager in the true sense[1]. I'm not sure this is an argument against making the change you propose, but it's worth noting.
With sandboxing/hsenv I've actually found cabal-install it to work much better than attempting to use distro packages for some libraries. (There error messages for stuff that requires native C libraries aren't always stellar, but that's something you quickly get used to.)
There are quite a few other language/frameworks that have language-specific build/package systems, Python, Ruby, Perl, node.js... Are Python developers on Arch pointed towards using pip to install Python libs?
I can only speak to node.js, but I think the general consensus there is to only use the distro-provided node.js and npm and then have user-specific/project-specific node_modules directories. (This is driven in part by npm's design and the extreme pace at which the ecosystem seems to be moving.) Given the limitations of GHC when it comes to module interoperability between versions, I think this approach makes sense for GHC too. (Btw, there's work afoot to adress this, but it's probably a few years off: http://plv.mpi-sws.org/backpack/ )
I think sometimes the right thing is to point users to another package manager, e.g. packaging vim scripts for system wide installation is a bit silly, since installing a vim script affects ALL users on the system. So doing that would require providing some sort of vim-script manager to users. Then there's very little difference compared to just telling users to use Vundle/Pathogen/whatever directly instead. However, this isn't the case for Haskell/GHC...
AFAIK (and unfortunately) globally installed Haskell/GHC packages can greatly constrain what other packages you can install in a sandbox. That's been a huge problem for me in practice, which is why I personally always install ONLY ghc and cabal-install and use cabal sandboxing/hsenv for the rest. (Did this on Ubuntu as well when I was a user of that.) I think it would be a good idea to strip everything back to just having GHC and cabal-install in the "base" and to take some time to rethink how packaging everything else should work. I don't know if Cabal supports such a thing, but one idea off the top of my head for a new approach would be to have a distro-provided read-only *sandbox* which other read/write sandboxes could be *overlaid* on top of. That would mean that you could opt out of the distro-installed packages if you need to, but could still install distro packages if you, e.g. wanted/needed a Haskell program which depended on having haskell packages installed (git-annex?). Crazy idea? Regards,
On Thu, Apr 10, 2014 at 01:34:00PM +0200, Bardur Arantsson wrote:
On 2014-04-09 09:07, Magnus Therning wrote:
Change 2: Make a news item stating that cabal-install is now the recommended way to install haskell packages. This wouldn't pollute the filesystem since cabal-install installs packages to the ~/.cabal directory by default. We might need to include a tip sheet about how you would handle ghc updates since it requires extra user steps.
It should be noted that cabal-install isn't a package manager in the true sense[1]. I'm not sure this is an argument against making the change you propose, but it's worth noting.
With sandboxing/hsenv I've actually found cabal-install it to work much better than attempting to use distro packages for some libraries. (There error messages for stuff that requires native C libraries aren't always stellar, but that's something you quickly get used to.)
I know some people swear by cabal-install, but I've personally never gotten into using it. The sandbox feature does look neat and I've been meaning to try it out. A very first attempt just now did not impress me though: it wanted to downgrade bytestring for no discernible reason. Probably a user error though.
I think it would be a good idea to strip everything back to just having GHC and cabal-install in the "base" and to take some time to rethink how packaging everything else should work.
Personally I think it depends on what "everything else" is. One approach would be to centre everything around applications. To some extent I think that is the (maybe not explicit) rule for the rest of the packages. That would mean libs aren't packaged for their own sake, only in order to provide an application. This would mean that it'd be entirely possible to /only/ have ghc + cabal-install and still package all applications written in Haskell[^1]. /M [^1]: Ghc still links applications statically it seems so building in a sandbox and then only package the resulting executables. -- Magnus Therning OpenPGP: 0xAB4DFBA4 email: magnus@therning.org jabber: magnus@therning.org twitter: magthe http://therning.org/magnus I invented the term Object-Oriented, and I can tell you I did not have C++ in mind. -- Alan Kay
On 04/09/2014 01:27 AM, Thomas Dziedzic wrote:
Problem 2: Users are confused whether they should install packages from the repos or using cabal-install. This in turn sometimes causes them to install some packages from official repos, some from the aur, and some using cabal-install Explanation: Packages should ideally be installed from one place, either using repos/aur or cabal-install. Said that, it is entirely possible to use both repos/aur and cabal-install but for simplicity, one source is preferred.
Doesn't every language with its own package manager have this problem? For example, Python. Is there a good solution? Users knowing about this issue and making their own decisions is the current solution on every distro I'm familiar with.
Change 2: Make a news item stating that cabal-install is now the recommended way to install haskell packages. This wouldn't pollute the filesystem since cabal-install installs packages to the ~/.cabal directory by default. We might need to include a tip sheet about how you would handle ghc updates since it requires extra user steps.
This is the way I do it on my personal computer, FWIW. If there were any packages in the repo written in Haskell that are used by non-Haskellers, then all those packages' deps would have to be in the repos too. (e.g. darcs and pandoc, which are in AUR currently). Cheers, -Isaac
On 2014-04-08 22:27, Thomas Dziedzic wrote:
Hello all,
With the arrival of ghc 7.8.1 [0], I would like to address the following problems with a restructuring of how we treat haskell packages in archlinux:
Problem 1: Updating any haskell package has been delayed until we bump ghc. Explanation: ghc is unable to produce a library that has a stable abi. In other words, if a library gets rebuilt (even if it's the same exact source), we will need to rebuild all package that depend on it, and this would in turn a messy rebuild for any kind of rebuild.
Changing depends on packages to exact versions will make these incompatibilities rise quickly. Eg: package A should depend on packageB=x.y.z-n That would avoid mixing up different versions. This would avoid a library being updated in a system until all packages that depend on it have been rebuilt. -- Hugo Osvaldo Barrera
participants (6)
-
Bardur Arantsson
-
Hugo Osvaldo Barrera
-
Isaac Dupree
-
Jelle van der Waa
-
Magnus Therning
-
Thomas Dziedzic