[arch-projects] Pluggable wrapper
There are two conflicting schools of thought with respect to adding features to pacman: - everything should be written into pacman, because there's less to worry about that way - everything should be a wrapper, because it complicates pacman less
For the most part, I believe the latter. There are some things that would benefit everyone if they were added to pacman, but most things would just add unnecessary complexity that's only used by a subset of all the users.
There is a related issue to wrappers though: if you have more than one wrapper that do complimentary things, shouldn't you be able to run both of them at the same time?
Enter the pluggable wrapper script. It would be event based, interfacing (almost) seamlessly with pacman, and allow you to enabled and disable wrapper functionality, such as srcpac.
My two example cases were srcpac and rollbacks. Rollbacks could be implementable through such a system, but I'm already told it'll be going right into pacman itself. Another example is regular expression searching.
What does everyone think of this idea? Is there enough demand for different functionalities that would be nice for some people but annoying, problematic or just stupid to add to pacman?
Jason
On Mon, 2004-08-30 at 14:55, Jason Chu wrote:
There are two conflicting schools of thought with respect to adding features to pacman:
- everything should be written into pacman, because there's less to worry
about that way
- everything should be a wrapper, because it complicates pacman less
Actually, three schools of thought. This would be mine: - some things should be in a wrapper, some things should be in pacman
Almost everything I've seen brought up so far I thought should go in Pacman. Now you know me, I like things light. But most of the features are important enough to be in pacman proper. For example, I think rollbacks and orphans should be in pacman.
Basically, if a feature is important enough to write a wrapper for, it's important enough to be in Pacman, most of the time. Prove me wrong, but so far no one has.
SRCPAC is a bit different. All the source management tools in Arch are actually BASH scripts. So writing Srcpac in BASH was a logical step. Unless you plan on integrating ABS/Makepkg into Pacman (don't), keep Srcpac as BASH too (or Python :P).
There is a related issue to wrappers though: if you have more than one wrapper that do complimentary things, shouldn't you be able to run both of them at the same time?
If you have to run more than one wrapper at once, things have gotten out of hand. That's my opinion. This is where the conflict exists. People like things light, and fear featuritis, but we've seen programs that need a whole mess of things on top just to be useful. This is messy, and it pisses people off. One example is Slackware + all the 3rd party package management tools. Look how many there are. They're not all compatible, and almost all slackware users download them right after installing. If so many people use the features, they should be included by default (though Pat has them on ftp.slackware.com). This "unofficial" requirement has made large, compatible Slackware repositories non-existent. Compare that to Arch, Debian, et al. In fact, many people switch to Arch just for a Slackware + Swaret-like system (I know I and a bunch of people on the IRC channel did).
This will cause issues, because each wrapper is essentially a hack, and it will always be that way until Pacman is made to a library, and the wrappers are written in a Language that can interface directly with C.
Wrappers are also unofficial, by definition. You should never have to rely on two wrappers working together, because they are (in theory) written by completely independent people.
Enter the pluggable wrapper script. It would be event based, interfacing (almost) seamlessly with pacman, and allow you to enabled and disable wrapper functionality, such as srcpac.
Going with my "wrappers are a hack" theory, you're just legitimizing a hack as the official "plugin" system to Pacman. Pacman should eventually allow plugins natively if you really want this. This would probably be the solution to all of our problems, as you choose the features you want.
Of course Pacman couldn't be a plugin/library interface for a very long time, but I'm saying this is a bad road. I guarantee you that if you make a wrapper framework and begin writing lots of wrappers, this "hack" will become permanent, and cause innumerable issues (the slackware example again).
My two example cases were srcpac and rollbacks. Rollbacks could be implementable through such a system, but I'm already told it'll be going right into pacman itself. Another example is regular expression searching.
I've already stated other places that I think RegExps should be in the pacman proper (if they're deemed worthy enough). Why? Because they conflict with an existent Pacman feature (Search). Now instead of just adding features, you're going to modify existing ones? This could become like Perl, with Wrapper dependency hell. Wrappers will start to implement "essential" features, so wrappers will depend on one another. Then you'll need 10 wrappers working together just to get Pacman up to suff.
What does everyone think of this idea? Is there enough demand for different functionalities that would be nice for some people but annoying, problematic or just stupid to add to pacman?
Plugins would be nice. Do it right or don't do it at all (:P). But like I said, most of the requested features I like, and think should go into pacman. The others I think shouldn't happen in any form. I do think Srcpac should eventually become part of the default Pacman distribution, just like Makepkg/ABS.
Ben
On Mon, Aug 30, 2004 at 11:54:46PM -0500, Ben Mazer wrote:
On Mon, 2004-08-30 at 14:55, Jason Chu wrote:
There are two conflicting schools of thought with respect to adding features to pacman:
- everything should be written into pacman, because there's less to worry
about that way
- everything should be a wrapper, because it complicates pacman less
Actually, three schools of thought. This would be mine:
- some things should be in a wrapper, some things should be in pacman
Almost everything I've seen brought up so far I thought should go in Pacman. Now you know me, I like things light. But most of the features are important enough to be in pacman proper. For example, I think rollbacks and orphans should be in pacman.
Basically, if a feature is important enough to write a wrapper for, it's important enough to be in Pacman, most of the time. Prove me wrong, but so far no one has.
SRCPAC is a bit different. All the source management tools in Arch are actually BASH scripts. So writing Srcpac in BASH was a logical step. Unless you plan on integrating ABS/Makepkg into Pacman (don't), keep Srcpac as BASH too (or Python :P).
I think that's contradictory. "Anything important enough to write a wrapper for should be integrated in pacman." "Srcpac is a wrapper but should stay a wrapper."
srcpac was written in bash because of the simplicity of the functionality not because it called other bash scripts.
There is a related issue to wrappers though: if you have more than one wrapper that do complimentary things, shouldn't you be able to run both of them at the same time?
If you have to run more than one wrapper at once, things have gotten out of hand. That's my opinion. This is where the conflict exists. People like things light, and fear featuritis, but we've seen programs that need a whole mess of things on top just to be useful. This is messy, and it pisses people off. One example is Slackware + all the 3rd party package management tools. Look how many there are. They're not all compatible, and almost all slackware users download them right after installing. If so many people use the features, they should be included by default (though Pat has them on ftp.slackware.com). This "unofficial" requirement has made large, compatible Slackware repositories non-existent. Compare that to Arch, Debian, et al. In fact, many people switch to Arch just for a Slackware + Swaret-like system (I know I and a bunch of people on the IRC channel did).
But I've heard you yourself say, "you shouldn't write a wrapper for that because srcpac is already a wrapper and we won't be able to use both of them at once." Paraphased, obviously.
The sorts of plugins to this wrapper would be added functionality that isn't necessary in pacman but that people bitch about. I'd rather not have a buggy pacman because some guy cried that his package manager didn't skip and jump.
This will cause issues, because each wrapper is essentially a hack, and it will always be that way until Pacman is made to a library, and the wrappers are written in a Language that can interface directly with C.
I hear that a lot too: "A wrapper is a hack, until it uses a library, then it's a good idea". So, how about it's first implemented as a hack, then when libpacman is released, the code is modified slightly to be an elegant solution? I figured it'd be written in python... python can do that sort of stuff...
Wrappers are also unofficial, by definition. You should never have to rely on two wrappers working together, because they are (in theory) written by completely independent people.
True, but with a framework and a properly defined API multiple modules could work together. That's the whole point. You're saying wrappers won't work together if they're written by independent people. Yes, I agree with that. But modules written for a pluggable framework could work together. That's why it's a pluggable framework...
Enter the pluggable wrapper script. It would be event based, interfacing (almost) seamlessly with pacman, and allow you to enabled and disable wrapper functionality, such as srcpac.
Going with my "wrappers are a hack" theory, you're just legitimizing a hack as the official "plugin" system to Pacman. Pacman should eventually allow plugins natively if you really want this. This would probably be the solution to all of our problems, as you choose the features you want.
I don't think pacman should eventually allow plugins. There should always be a simple way to get to the basics of package management. I also wouldn't want to write pacman plugins in C. And, to take one of your excuses, "suddenly we'd have to install 10 plugins just to get pacman up to snuff".
Of course Pacman couldn't be a plugin/library interface for a very long time, but I'm saying this is a bad road. I guarantee you that if you make a wrapper framework and begin writing lots of wrappers, this "hack" will become permanent, and cause innumerable issues (the slackware example again).
I believe you wanted to use the auxiliary verb "can't", "couldn't" meant that it'd only be a plugin/library interface for a short time.
You wouldn't be writing lots of wrappers, you'd be writing lots of modules.
I don't see how the slackware example fits here. We're not talking about different package managers and different repos. We're talking about added functionality.
My two example cases were srcpac and rollbacks. Rollbacks could be implementable through such a system, but I'm already told it'll be going right into pacman itself. Another example is regular expression searching.
I've already stated other places that I think RegExps should be in the pacman proper (if they're deemed worthy enough). Why? Because they conflict with an existent Pacman feature (Search). Now instead of just adding features, you're going to modify existing ones? This could become like Perl, with Wrapper dependency hell. Wrappers will start to implement "essential" features, so wrappers will depend on one another. Then you'll need 10 wrappers working together just to get Pacman up to suff.
Conflict with features? Did you look at how it was implemented originally? He added a new arg to pacman. -Sx would search using regular expressions.
The idea is never to implement an "essential" feature with a wrapper. If regexps aren't deemed worthy enough, then do they go the way of the dodo? Do we just ignore them because they're not pure enough to go into pacman proper?
I never said things that should go into pacman would go into a module instead. I'm saying things that don't need to go into pacman could be in a module instead.
What does everyone think of this idea? Is there enough demand for different functionalities that would be nice for some people but annoying, problematic or just stupid to add to pacman?
Plugins would be nice. Do it right or don't do it at all (:P). But like I said, most of the requested features I like, and think should go into pacman. The others I think shouldn't happen in any form. I do think Srcpac should eventually become part of the default Pacman distribution, just like Makepkg/ABS.
"right" is a relative term at this point. I don't disagree that most of the requested features probably should go into pacman, but there are others that don't need to.
Why is srcpac an exception this whole time? I don't see how you can say, "wrappers are evil and should never exist! but I love my srcpac!"
Jason
Basically, if a feature is important enough to write a wrapper for, it's important enough to be in Pacman, most of the time. Prove me wrong, but so far no one has.
SRCPAC is a bit different. All the source management tools in Arch are actually BASH scripts. So writing Srcpac in BASH was a logical step. Unless you plan on integrating ABS/Makepkg into Pacman (don't), keep Srcpac as BASH too (or Python :P).
I think that's contradictory. "Anything important enough to write a wrapper for should be integrated in pacman." "Srcpac is a wrapper but should stay a wrapper."
srcpac was written in bash because of the simplicity of the functionality not because it called other bash scripts.
I recognize this contradiction. I guess what Ben really meant is that there's an important set of functionality that should be integrated, and things that can be cleanly separated and benefit from being implemented as a wrapper can remain that way. I guess I'm not yet sure how I feel about that.
If you have to run more than one wrapper at once, things have gotten out of hand. That's my opinion. This is where the conflict exists. People like things light, and fear featuritis, but we've seen programs that need a whole mess of things on top just to be useful. This is messy, and it pisses people off. One example is Slackware + all the 3rd party package management tools. Look how many there are. They're not all compatible, and almost all slackware users download them right after installing. If so many people use the features, they should be included by default (though Pat has them on ftp.slackware.com). This "unofficial" requirement has made large, compatible Slackware repositories non-existent. Compare that to Arch, Debian, et al. In fact, many people switch to Arch just for a Slackware + Swaret-like system (I know I and a bunch of people on the IRC channel did).
I definitely agree that things can get too complex and deep, and it is one of the reasons I use Arch.. to avoid that.
But I've heard you yourself say, "you shouldn't write a wrapper for that because srcpac is already a wrapper and we won't be able to use both of them at once." Paraphased, obviously.
The sorts of plugins to this wrapper would be added functionality that isn't necessary in pacman but that people bitch about. I'd rather not have a buggy pacman because some guy cried that his package manager didn't skip and jump.
This will cause issues, because each wrapper is essentially a hack, and it will always be that way until Pacman is made to a library, and the wrappers are written in a Language that can interface directly with C.
I hear that a lot too: "A wrapper is a hack, until it uses a library, then it's a good idea". So, how about it's first implemented as a hack, then when libpacman is released, the code is modified slightly to be an elegant solution? I figured it'd be written in python... python can do that sort of stuff...
Wrappers are also unofficial, by definition. You should never have to rely on two wrappers working together, because they are (in theory) written by completely independent people.
True, but with a framework and a properly defined API multiple modules could work together. That's the whole point. You're saying wrappers won't work together if they're written by independent people. Yes, I agree with that. But modules written for a pluggable framework could work together. That's why it's a pluggable framework...
I agree with this, Jason. I think that the "module" becomes a "hack" when it doesn't follow an API, not because it's written by someone else or because it's written in another language.
Enter the pluggable wrapper script. It would be event based, interfacing (almost) seamlessly with pacman, and allow you to enabled and disable wrapper functionality, such as srcpac.
Going with my "wrappers are a hack" theory, you're just legitimizing a hack as the official "plugin" system to Pacman. Pacman should eventually allow plugins natively if you really want this. This would probably be the solution to all of our problems, as you choose the features you want.
I don't think pacman should eventually allow plugins. There should always be a simple way to get to the basics of package management. I also wouldn't want to write pacman plugins in C. And, to take one of your excuses, "suddenly we'd have to install 10 plugins just to get pacman up to snuff".
Jason, I don't think your first two statements are mutually exclusive. You can have both.
Here's the summary of my input: I agree that having 10 plugin packages to install would be annoying; however, just because the program is internally structured to use plugins doesn't mean you have to package it that way. If it makes sense for stability and simplicity to package the plugins into the single pacman package, that's fine.
So I guess my suggestion would be to define a C plugin API, define APIs for plugins in other languages where you feel they're necessary (structuring it as a "bridge plugin" between that language's plugin API and the C plugin API), and then carefully manage the plugins that you let into the main source tree. If you wanted, you could structure the system so that the list of plugins actually gets compiled into pacman, so people *can't* add plugins without distributing a whole new pacman package.. which they could still do for testing.
The thorny areas: How do you make sure you've got an interpreter for each kind of plugin API? Which languages do you choose as other languages for APIs? How do you structure the API to give the right level of flexibility?
So all this sounds like a a lot of work, though probably actually less than it sounds. Perhaps in the short term, a "wrapper API" might be the way to go, as Jason seems to be suggesting. My question is: exactly what do you mean by "event driven"? What exactly would be the interface between pacman and and the pluggable wrapper script? How would it survive changes to pacman itself? How would we keep the pluggable script in sync with the current version of pacman?
The really nice part about a plugin structure is that it encourages other people to extend pacman, because it will be easier to do so. I think ethereal is one of the classic examples of this phenomenon. Then the extensions that you like and are stable can be integrated into the main tree.
- P
Enter the pluggable wrapper script. It would be event based, interfacing (almost) seamlessly with pacman, and allow you to enabled and disable wrapper functionality, such as srcpac.
Going with my "wrappers are a hack" theory, you're just legitimizing a hack as the official "plugin" system to Pacman. Pacman should eventually allow plugins natively if you really want this. This would probably be the solution to all of our problems, as you choose the features you want.
I don't think pacman should eventually allow plugins. There should always be a simple way to get to the basics of package management. I also wouldn't want to write pacman plugins in C. And, to take one of your excuses, "suddenly we'd have to install 10 plugins just to get pacman up to snuff".
Jason, I don't think your first two statements are mutually exclusive. You can have both.
Here's the summary of my input: I agree that having 10 plugin packages to install would be annoying; however, just because the program is internally structured to use plugins doesn't mean you have to package it that way. If it makes sense for stability and simplicity to package the plugins into the single pacman package, that's fine.
So I guess my suggestion would be to define a C plugin API, define APIs for plugins in other languages where you feel they're necessary (structuring it as a "bridge plugin" between that language's plugin API and the C plugin API), and then carefully manage the plugins that you let into the main source tree. If you wanted, you could structure the system so that the list of plugins actually gets compiled into pacman, so people *can't* add plugins without distributing a whole new pacman package.. which they could still do for testing.
I'm pretty sure the plugins for other languages point is a sticky one... we don't want to have pacman compiling in a python interpreter just to use some plugins.
Though I do like the idea of being able to statically, at compile time, define your list of plugins to be shipped. That's kinda cool.
The thorny areas: How do you make sure you've got an interpreter for each kind of plugin API? Which languages do you choose as other languages for APIs? How do you structure the API to give the right level of flexibility?
Those are the tough questions. I already answered one by saying that building an interpreter into pacman is not an acceptable option. That still leaves options like pyrex though.
The API is an even more difficult problem. I only have a vague idea.
So then, if you were a plugin API for a package manager, what would you look like?
So all this sounds like a a lot of work, though probably actually less than it sounds. Perhaps in the short term, a "wrapper API" might be the way to go, as Jason seems to be suggesting. My question is: exactly what do you mean by "event driven"? What exactly would be the interface between pacman and and the pluggable wrapper script? How would it survive changes to pacman itself? How would we keep the pluggable script in sync with the current version of pacman?
It would be event driven because each module that wanted in would register itself to receive different events in the pacman process.
If you look at how srcpac wraps pacman, srcpac becomes a command of itself. It only calls pacman when it needs to. The design of srcpac allows it to ignore any functionality that doesn't affect its own functionality.
Any groundbreaking changes with pacman would need an upgrade to the wrapper framework.
The really nice part about a plugin structure is that it encourages other people to extend pacman, because it will be easier to do so. I think ethereal is one of the classic examples of this phenomenon. Then the extensions that you like and are stable can be integrated into the main tree.
I've never looked at the ethereal plugin system ;)
Jason
Just a question, is Judd on this list? He really should be. I mean, we are discussing the direction of his program.
Ben
On Tue, Aug 31, 2004 at 12:22:23PM -0500, Ben Mazer wrote:
Just a question, is Judd on this list? He really should be. I mean, we are discussing the direction of his program.
Yes, Judd is on this list. I'm sure he just wants to be a quiet bystander though.
I'd already mentioned the wrapper framework to him and he said as long as he doesn't have to change pacman in any way, he doesn't care.
Jason
On Tue, Aug 31, 2004 at 12:22:23PM -0500, Ben Mazer wrote:
Just a question, is Judd on this list? He really should be. I mean, we are discussing the direction of his program.
Yea, I lurk. :)
I'm kind of a one-thing-at-a-time type of guy. That way stuff gets done. In reality, it still turns into a 5-or-10-things-at-a-time process, but I'm very picky about which 5-10 things I work on, and other things have been in the loop for far longer, so they get precedence.
In my opinion, this discussion should stick to being just discussion, at least until libpacman is ready. Then we'll actually have an API we can build off of. Baby steps. :)
Although it is good to have some foresight. If we can all convince each other that an elaborate plugin system is what pacman really needs, then we can shape the libpacman API to prepare for that.
But I'm still not convinced. To me, a package manager should do a small number of things and do them very right. I don't know if a plugin system would be worth the time and effort. Maybe we should try and come up with a list of 5-10 solid pluggable ideas, ones that wouldn't deserve to go into pacman proper. We don't need to implement them all, but it would help to justify the need.
- J
But I'm still not convinced. To me, a package manager should do a small number of things and do them very right. I don't know if a plugin system would be worth the time and effort. Maybe we should try and come up with a list of 5-10 solid pluggable ideas, ones that wouldn't deserve to go into pacman proper. We don't need to implement them all, but it would help to justify the need.
This is a sage counterpoint. Though it's nice to have modular software, it would take substantial time to add the plugin API(s), and if there aren't enough plugins to justify it, our collective coding time could be put to better use elsewhere.
So I guess I'm waiting to hear the 5-10 ideas, too.
- P
On Tue, 2004-08-31 at 14:51, Judd Vinet wrote:
But I'm still not convinced. To me, a package manager should do a small number of things and do them very right. I don't know if a plugin system would be worth the time and effort. Maybe we should try and come up with a list of 5-10 solid pluggable ideas, ones that wouldn't deserve to go into pacman proper. We don't need to implement them all, but it would help to justify the need.
I think I know what Judd is hinting at. He wants a dancing Jesus plugin.
But anyway, if we want Pacman to really be integrated into the Arch experience, a "framework" mentality would be best. If we want Pacman to be just one tool, the current path is fine. To keep with my (crappy) example, The "tool" ideal is more in the Slackware direction. No one uses Slackware for its package management, it's just there to install/remove packages. Nothing special. Debian on the other hand really makes Apt/DPKG/Dselect part of why you choose Debian.
This may seem contradictory to my previous posts, but I like Pacman as an independent tool as well. Obviously there's such a things as underpowered or too independent, but Pacman isn't even close to there. As an example, I kept my Menu script agnostic of Pacman, because I felt there was no reason to create a big web of "integration". Another example is how I feel package maintenance should be done. Arch shouldn't try to "include everything". Just showing I'm not that much of a hypocrite :P.
I was just throwing out ideas. The practical side, which Judd has wisely taken, is that Pacman really isn't missing that much. So what if it won't do every magical thing. Even APT/DPKG doesn't do everything, and it's probably the most worked on package manager out there.
I like Arch Vanilla. Then again, I have lots of free time.
Ben
participants (4)
-
Ben Mazer
-
Jason Chu
-
Judd Vinet
-
Paul Mattal