Am Freitag, den 11.09.2009, 11:30 +0200 schrieb Xavier:

On Fri, Sep 11, 2009 at 11:18 AM, Marc - A. Dahlhaus [ Administration | Westermann GmbH ] wrote:

...

Hello List,

currently i'm working on adding a trigger infrastructure to libaplm/pacman to get rid of workload during upgrades or installations that runs more than a single time.

Good examples for such jobs would be:

- depmod after module installations - mkinitramfs after module installations - catalog-updates for fonts or infopages etc...

My current plan is to create a helper script called "alpmtrigger" wich handles the work of setting the triggers and executing the triggers after pacman is done with installation.

How would it work? Draft:

A package that adds a trigger to the system installs a trigger script that do all the work needed into a "/usr/share/pacman/alpmtrigger" dir.

inside of a packages install-helper we would call "alpmtrigger pkgtrigger" to set the trigger "pkgtrigger" active. alpmtrigger would do "touch /var/somewhere/alpmtrigger/pkgtrigger" wich would create the file if it not exists already.

A hook to execute "alpmtrigger" (without params?) would need to be added after the _alpm_run_chroot(root, "/sbin/ldconfig"); call in function _alpm_ldconfig inside of lib/libalpm/util.c to actualy scan the contents of /var/somewhere/alpmtrigger, and for each filename in it check if /usr/share/pacman/alpmtrigger/filename exists and if it is and is executable run the trigger. After the trigger is done and retuned without error, the file /var/somewhere/alpmtrigger/filename gets removed.

Any suggestions, comments or somebody else working on such a thing?

see http://wiki.archlinux.org/index.php/User:Allan/Pacman_Hooks

I think this design concept will not work in real world use cases. Problems i spotted: - Hooks should be provided by the packages that actualy contain the tools that the hook has to do. - Hooks should get installed by placing a file into an apropriate location and not by adding them to a single config-file (manual intervention of the user required or every new hook a new hooks package has to be made) - I don't think that so many points to add a hook are really needed in a real world... In the end it looks like this is a design concept to reimplement the install-helper functionality but decouple it from the actual package. Looks like a big step backwards for me...

Before implementing anything, I would like to see the maximum of actions which are currently done in real package scriptlets converted to the new hook system. So that we are sure we have a good coverage.

nothing to add here, as this is an arch only argument, i don't use arch and pacman development should done with other things than "how useful is XYZ for arch" in mind IMO...

Scriptlets are just another case of code duplication which is my big enemy. Once we are confident enough that the proposed system achieves that, I think I would be interested and motivated to do the implementation.

Yeah, and you would get rid of the hated duplication if you have one single point (a trigger script) to add this currently duplicated code to and select the things to do inside the packages that needs the work to be done, or wouldn't you? ;-) I think that a solution is needed that removes work from the package maintainers and not add another thing they have to think about and have to collaborate on like the single hook package what would be fruit of implementing the Packman_Hooks thing... By the way, i don't wanted to be rude with this, my english isn't that good... Marc

Hello Allan, Am Freitag, den 11.09.2009, 20:38 +1000 schrieb Allan McRae:

Marc - A. Dahlhaus [ Administration | Westermann GmbH ] wrote:

...

Am Freitag, den 11.09.2009, 11:30 +0200 schrieb Xavier:

...

On Fri, Sep 11, 2009 at 11:18 AM, Marc - A. Dahlhaus [ Administration | Westermann GmbH ] wrote:

...

Hello List,

currently i'm working on adding a trigger infrastructure to libaplm/pacman to get rid of workload during upgrades or installations that runs more than a single time.

Good examples for such jobs would be:

- depmod after module installations - mkinitramfs after module installations - catalog-updates for fonts or infopages etc...

My current plan is to create a helper script called "alpmtrigger" wich handles the work of setting the triggers and executing the triggers after pacman is done with installation.

How would it work? Draft:

A package that adds a trigger to the system installs a trigger script that do all the work needed into a "/usr/share/pacman/alpmtrigger" dir.

inside of a packages install-helper we would call "alpmtrigger pkgtrigger" to set the trigger "pkgtrigger" active. alpmtrigger would do "touch /var/somewhere/alpmtrigger/pkgtrigger" wich would create the file if it not exists already.

A hook to execute "alpmtrigger" (without params?) would need to be added after the _alpm_run_chroot(root, "/sbin/ldconfig"); call in function _alpm_ldconfig inside of lib/libalpm/util.c to actualy scan the contents of /var/somewhere/alpmtrigger, and for each filename in it check if /usr/share/pacman/alpmtrigger/filename exists and if it is and is executable run the trigger. After the trigger is done and retuned without error, the file /var/somewhere/alpmtrigger/filename gets removed.

Any suggestions, comments or somebody else working on such a thing?

see http://wiki.archlinux.org/index.php/User:Allan/Pacman_Hooks

I think this design concept will not work in real world use cases.

Note that the design document I started is far from the final version... it was written to get discussion going and see what it could be used for.

...because of this i called it "design concept"...

...

Problems i spotted:

- Hooks should be provided by the packages that actualy contain the tools that the hook has to do.

Sure, but how is the opposite implied by what is written. Texinfo could (and should) provide the hook for updating the info databse, etc. I think this is you main point throughout and that seems to be a misunderstanding.

...

- Hooks should get installed by placing a file into an apropriate location and not by adding them to a single config-file (manual intervention of the user required or every new hook a new hooks package has to be made)

From the design document, hook were added as a single file to the /etc/pacman.d/hooks.d/ directory. Adding the hook to a config file was to control if a user wanted that hook run. e.g. I really do not give a crap about updating the info page database so I would not set it to run. This can probably be improved (as could many things in that wiki page).

...

- I don't think that so many points to add a hook are really needed in a real world...

Here is a list of "hooks" that would be useful for Arch packaging. I know this is Arch specific but it should give an idea of real world use cases:

PostInstall: [gconf-install] - install gconf files [info-install] - add info pages to info directory file [man-install] - add man pages to db

PreRemove: [gconf-remove] - remove gconf files [info-remove] - remove info pages from info directory file [man-remove] - remove man pages from db

Transaction: [depmod] - can we do that as a hook [desktop] - update desktop database [font] - update the font cache [icon] - update gtk icon cache or xdg-icon-resource (how to choose?) [mime] - update mime database [tex-file] - texconfig-sys rehash [tex-font] - updmap-sys [vim] - update vim help tags [xpdf] - update /etc/xpdfrc

Install: [firefox] automatically run branding firefox script

The firefox one doesn't look like an example that would be used by anything other than the firefox package itself...

That covers actions that would use the majority of the proposed hook points.

<snip>

...

...

Before implementing anything, I would like to see the maximum of actions which are currently done in real package scriptlets converted to the new hook system. So that we are sure we have a good coverage.

nothing to add here, as this is an arch only argument, i don't use arch and pacman development should done with other things than "how useful is XYZ for arch" in mind IMO...

You are free to provide your non-Arch use cases so we can see real world examples.

...

<snip> I think that a solution is needed that removes work from the package maintainers and not add another thing they have to think about and have to collaborate on like the single hook package what would be fruit of implementing the Packman_Hooks thing...

As I said above, I really do not understand why you think we would have to provide a hooks package and not provide hooks with the appropriate package.

Got to this impression because there is a single config file in which a hook has to be configured before it could be activated. The only way to achieve such a configuration in a way that works without user-intervention would be to wrap it all inside of one hook-package or things such a full system installation would end up without working hooks to execute. I apologize if my impression is wrong... In the end, my opinion of the Pacman_Hooks stuff is that it is well to complicated. I think that the implementation would be very complicated with respect to the configuration stuff... The points to which hooks could be added are more or less the same points on which the install-helper would provide already a point to add stuff. Looks like duplication to me, would be better to use what we have to achieve the goal. The only outstanding hook-target would be the transactional one which if i understand that correctly gets fired up after the transaction has completed. This is exactly the functionality i want to get into pacman. I don't care which way it gets implemented upstream in the end, as it's just the functionality that i want to get and use. Also to Xavier: your comments and hints were very useful :-) Could you review the revised version of my plan that i send a hour ago and give me some feedback about it? I want to implement a proof of concept version of alpmtrigger over the weekend if i get enough spare time from my wife and the baby ;-) and some Feedback would be very useful for me. thanks, Marc

Am Freitag, den 11.09.2009, 21:50 +1000 schrieb Allan McRae:

Marc - A. Dahlhaus [ Administration | Westermann GmbH ] wrote:

...

Hello Allan,

Am Freitag, den 11.09.2009, 20:38 +1000 schrieb Allan McRae:

...

Marc - A. Dahlhaus [ Administration | Westermann GmbH ] wrote:

...

Am Freitag, den 11.09.2009, 11:30 +0200 schrieb Xavier:

...

On Fri, Sep 11, 2009 at 11:18 AM, Marc - A. Dahlhaus [ Administration | Westermann GmbH ] wrote:

...

Hello List,

currently i'm working on adding a trigger infrastructure to libaplm/pacman to get rid of workload during upgrades or installations that runs more than a single time.

Good examples for such jobs would be:

- depmod after module installations - mkinitramfs after module installations - catalog-updates for fonts or infopages etc...

My current plan is to create a helper script called "alpmtrigger" wich handles the work of setting the triggers and executing the triggers after pacman is done with installation.

How would it work? Draft:

A package that adds a trigger to the system installs a trigger script that do all the work needed into a "/usr/share/pacman/alpmtrigger" dir.

inside of a packages install-helper we would call "alpmtrigger pkgtrigger" to set the trigger "pkgtrigger" active. alpmtrigger would do "touch /var/somewhere/alpmtrigger/pkgtrigger" wich would create the file if it not exists already.

A hook to execute "alpmtrigger" (without params?) would need to be added after the _alpm_run_chroot(root, "/sbin/ldconfig"); call in function _alpm_ldconfig inside of lib/libalpm/util.c to actualy scan the contents of /var/somewhere/alpmtrigger, and for each filename in it check if /usr/share/pacman/alpmtrigger/filename exists and if it is and is executable run the trigger. After the trigger is done and retuned without error, the file /var/somewhere/alpmtrigger/filename gets removed.

Any suggestions, comments or somebody else working on such a thing?

see http://wiki.archlinux.org/index.php/User:Allan/Pacman_Hooks

I think this design concept will not work in real world use cases.

Note that the design document I started is far from the final version... it was written to get discussion going and see what it could be used for.

...because of this i called it "design concept"...

...

...

Problems i spotted:

- Hooks should be provided by the packages that actualy contain the tools that the hook has to do.

Sure, but how is the opposite implied by what is written. Texinfo could (and should) provide the hook for updating the info databse, etc. I think this is you main point throughout and that seems to be a misunderstanding.

...

- Hooks should get installed by placing a file into an apropriate location and not by adding them to a single config-file (manual intervention of the user required or every new hook a new hooks package has to be made)

From the design document, hook were added as a single file to the /etc/pacman.d/hooks.d/ directory. Adding the hook to a config file was to control if a user wanted that hook run. e.g. I really do not give a crap about updating the info page database so I would not set it to run. This can probably be improved (as could many things in that wiki page).

...

- I don't think that so many points to add a hook are really needed in a real world...

Here is a list of "hooks" that would be useful for Arch packaging. I know this is Arch specific but it should give an idea of real world use cases:

PostInstall: [gconf-install] - install gconf files [info-install] - add info pages to info directory file [man-install] - add man pages to db

PreRemove: [gconf-remove] - remove gconf files [info-remove] - remove info pages from info directory file [man-remove] - remove man pages from db

Transaction: [depmod] - can we do that as a hook [desktop] - update desktop database [font] - update the font cache [icon] - update gtk icon cache or xdg-icon-resource (how to choose?) [mime] - update mime database [tex-file] - texconfig-sys rehash [tex-font] - updmap-sys [vim] - update vim help tags [xpdf] - update /etc/xpdfrc

Install: [firefox] automatically run branding firefox script

The firefox one doesn't look like an example that would be used by anything other than the firefox package itself...

Yes - but it is an actual example a script many people do run everytime a package is updated. Obviously that would be a user provided hook and not provided by the firefox package.

Interesting idea...

...

...

That covers actions that would use the majority of the proposed hook points.

<snip>

...

...

Before implementing anything, I would like to see the maximum of actions which are currently done in real package scriptlets converted to the new hook system. So that we are sure we have a good coverage.

nothing to add here, as this is an arch only argument, i don't use arch and pacman development should done with other things than "how useful is XYZ for arch" in mind IMO...

You are free to provide your non-Arch use cases so we can see real world examples.

...

<snip> I think that a solution is needed that removes work from the package maintainers and not add another thing they have to think about and have to collaborate on like the single hook package what would be fruit of implementing the Packman_Hooks thing...

As I said above, I really do not understand why you think we would have to provide a hooks package and not provide hooks with the appropriate package.

Got to this impression because there is a single config file in which a hook has to be configured before it could be activated. The only way to achieve such a configuration in a way that works without user-intervention would be to wrap it all inside of one hook-package or things such a full system installation would end up without working hooks to execute. I apologize if my impression is wrong...

That was an idea put out there because I basically do not want my info page database updated (the most common install file task in Arch). This could probably be achieved without a config file by removing execute permissions from the hooks script or adding the script to the noextract in pacman.conf, or in a multitud of ways. And now that concern has been flagged, the wiki page can be adjusted.

This exact point was raised already back on Fri 23 Jan 2009 by me...

These are the first comments made on the hooks idea since Februrary despite much advertising of the potential of implementing this on my behalf. It is good to see some real discussion of implementation issues starting.

Hope this discussion remains as productive as it is :-) Marc

Allan McRae wrote:

<snip>

...

...

That was an idea put out there because I basically do not want my info page database updated (the most common install file task in Arch). This could probably be achieved without a config file by removing execute permissions from the hooks script or adding the script to the noextract in pacman.conf, or in a multitud of ways. And now that concern has been flagged, the wiki page can be adjusted.

This exact point was raised already back on Fri 23 Jan 2009 by me...

So it was... and here is the relevant part of my response: [quote]

...each hook need to provide the following information:

1. when to run it (per file or transaction) 2. what files trigger it running 3. a script that can be sourced and run by the shell.

It is _a lot_ easier if 1 and 2 are separate from 3. [end quote]

So I see what I was trying to do with the config file. We could always have a separate config file per hook or try to cleanly join these together.

Obviously this area needs more discussion and prototypes.

Looking at your suggestion at the start if this thread, it seems you wanted to avoid this config by adding a flag to run the script in the PKGUILD or install file. I see the advantage of the config is that I can forget about dealing with info pages at all and everything will be handled automatically. As you may notice, I hate packaging info pages :) Allan

Marc - A. Dahlhaus [ Administration | Westermann GmbH ] wrote:

Am Freitag, den 11.09.2009, 22:30 +1000 schrieb Allan McRae:

...

Allan McRae wrote:

...

<snip>

...

...

That was an idea put out there because I basically do not want my info page database updated (the most common install file task in Arch). This could probably be achieved without a config file by removing execute permissions from the hooks script or adding the script to the noextract in pacman.conf, or in a multitud of ways. And now that concern has been flagged, the wiki page can be adjusted.

This exact point was raised already back on Fri 23 Jan 2009 by me...

So it was... and here is the relevant part of my response: [quote]

...each hook need to provide the following information:

1. when to run it (per file or transaction) 2. what files trigger it running 3. a script that can be sourced and run by the shell.

It is _a lot_ easier if 1 and 2 are separate from 3. [end quote]

So I see what I was trying to do with the config file. We could always have a separate config file per hook or try to cleanly join these together.

Obviously this area needs more discussion and prototypes.

Looking at your suggestion at the start if this thread, it seems you wanted to avoid this config by adding a flag to run the script in the PKGUILD or install file.

To the install scriptlet because of the pre_ and post_ hooks we have there already i thought it would be a low hanging fruit for me... :o)

It would be very much like the apt trigger mechanisn in debian today with respect to the end-of-transaction hook i think...

We would get the decoupling of common tasks from the install-scriptlets that use them and remove the duplication point altogether from the desk and even gain new functionality...

Reading this, I am not sure I understand your proposal. You suggested: inside of a packages install-helper we would call "alpmtrigger pkgtrigger" to set the trigger "pkgtrigger" active. So each package needs an install file that contains just e.g. "alpmtrigger infopage". For every package that has info pages... That still sounds like duplication to me. Given all we need to currently do for the install file for info pages is to copy the prototype and list the info pages to install, you suggestion really is no less work.

<snip> Will get my hands dirty on this one over the weekend...

I would be waiting until some sort of consensus is met on how this should be implemented.... Allan

On Fri, Sep 11, 2009 at 05:05:07PM +0200, Marc - A. Dahlhaus [ Administration | Westermann GmbH ] wrote:

Add all info pages that got installed or updated in the last 24h:

find /usr/share/info -type f -ctime -1 \ -exec install-info '{}' /usr/share/info/dir 2>/dev/null

I have no opinion to offer on the hooks/trigger proposals in general, but as for the info page idea, arbitrary times are sloppy in this case when what you really want is "newer than /usr/share/info/dir". That will provide the proper list to the infodir command. As for removing info pages, that one can be done by either parsing the installed files list of a package and pulling out the names of the info pages, or just running infodir on the whole /usr/share/info directory after removal of said pages. The first is cleaner conceptually, but harder to implement correctly without making assumptions. As for the concept of pulling out redundant code from install scriptlets, is anyone suggesting the same thing for PKGBUILDs? For one, the compression of man/info pages is something that should either be done globally or not done globally. If implemented as a hook, then PKGBUILDs won't have to do it manually for those packages that install non-compressed files. -- Jeff My other computer is an abacus.

On Fri, Sep 11, 2009 at 3:27 PM, Allan McRae wrote:

So each package needs an install file that contains just e.g. "alpmtrigger infopage".  For every package that has info pages...  That still sounds like duplication to me. Given all we need to currently do for the install file for info pages is to copy the prototype and list the info pages to install, you suggestion really is no less work.

How would you even specify the info pages with this method ? If alpmtrigger infopage just does 'touch /var/somewhere/alpmtrigger/infopage' , how do you give the list of info files ? There is indeed a big difference between Marc proposal and the old Aaron proposal. The only thing Marc's proposal really provide is the ability to have Transaction hooks. The idea of Aaron was to have File Hooks, that is, actions which are triggered by installation or removal of files. Edit : now I saw Marc's answer, proposing that all kind of file hooks action could be handled by a transaction hook/trigger using find and ctime. But right now I cannot see which are the pros and cons of each method. Otherwise, it seems that Marc is saying that install scriptlets are fine, and he wants to add two functionality : 1) transaction level hook 2) sharing code 1 could be even more easily achieved by just defining a new function in scriptlets, eg transaction() 2 could be achieved with function libraries (see http://bugs.archlinux.org/task/10375) Again here, it is not clear to me which method is better, I am just throwing out ideas.

Xavier schrieb:

On Fri, Sep 11, 2009 at 3:27 PM, Allan McRae wrote:

...

Reading this, I am not sure I understand your proposal. You suggested:

inside of a packages install-helper we would call "alpmtrigger pkgtrigger" to set the trigger "pkgtrigger" active.

So each package needs an install file that contains just e.g. "alpmtrigger infopage". For every package that has info pages... That still sounds like duplication to me. Given all we need to currently do for the install file for info pages is to copy the prototype and list the info pages to install, you suggestion really is no less work.

Another important question here : how are user hooks defined then ?

I think we should distinct between triggers and what they are/provide and what hooks are/provide. The trigger approach is used to fire up well defined common tasks at a later point in the installation. Triggers are by design part of the package that provides them. They allow other packages to make usage of its functionality for common tasks at the end of a running transaction. They try to do such a common task one time during the whole transaction. There is no ordinary user of the distribution involved here by design as it has to just work for the user without user intervention. The hook approach is used to fire up tasks on specific events during the package installation. Hooks are placed at defined spots to the codebase of libalpm and provide chains with as much hook-rules as got configured for it. The rules consists of "filename that got installed" based fire up of scripts. (I think the remaining two hook types "once per transaction" and "for a given package" would be more or less like what i wrote about triggers above) The question is do we really need this chain of rules for each proposed hook in libalpm thing which needs to scan for all the defined filename or package-name rules and will it be worth the effort to implement it if just a few packages ever will make use of the hooks. How often will we look at matching hook rules during normal operation? I'll bet not quite often. I think this approach would also waste quite a few resources on every transaction to search for matches without even knowing if some package would trigger such an event. It has to do that every time as there is no "i want this hook to get fired up" action from the package-side as it would in the trigger approach. I think to list the content of one or more directories and fire up scriptlets based on files found would clearly outperform the scanning of every file that gets installed based on N rules we need to match against. I think we could achieve anything we could think of with the trigger approach from the packagers perspective except giving the user the ability to install hooks on this users self defined event-rules... If a user really wants to add a script which fires up if some filename based event is fulfilled than this user should take a look at the incron package and use it for what it was made IMO. Thats the way i look at the situation currently, but i'm open to any others opinion of course. Thanks, Marc

Allan McRae schrieb:

Marc - A. Dahlhaus wrote:

...

Xavier schrieb:

...

On Fri, Sep 11, 2009 at 3:27 PM, Allan McRae wrote:

...

Reading this, I am not sure I understand your proposal. You suggested:

inside of a packages install-helper we would call "alpmtrigger pkgtrigger" to set the trigger "pkgtrigger" active.

So each package needs an install file that contains just e.g. "alpmtrigger infopage". For every package that has info pages... That still sounds like duplication to me. Given all we need to currently do for the install file for info pages is to copy the prototype and list the info pages to install, you suggestion really is no less work.

Another important question here : how are user hooks defined then ?

I think we should distinct between triggers and what they are/provide and what hooks are/provide.

The trigger approach is used to fire up well defined common tasks at a later point in the installation.

Triggers are by design part of the package that provides them. They allow other packages to make usage of its functionality for common tasks at the end of a running transaction. They try to do such a common task one time during the whole transaction. There is no ordinary user of the distribution involved here by design as it has to just work for the user without user intervention.

How are other packages making use of the trigger if we are not letting filenames set it off? As I said in an earlier post, having to define the trigger in an install file or similar removes the automation of the hook which is pretty much the point here. It would just result in changing which .install file prototype is copied so would be no les work for packagers.

...

The hook approach is used to fire up tasks on specific events during the package installation.

Hooks are placed at defined spots to the codebase of libalpm and provide chains with as much hook-rules as got configured for it. The rules consists of "filename that got installed" based fire up of scripts. (I think the remaining two hook types "once per transaction" and "for a given package" would be more or less like what i wrote about triggers above)

So the hook case actually covers _updating_ info and man_db databases and installing gconf files. While "updating" the info database by deleting it and recreating it may be reasonably fast, it is a waste and doing that for e.g. gconf files would be slow.

Point taken. But what trigger try to achieve is a fully different thing: They delay a common workload that the packager wants to be done for the package and that could be done for more than one package (thats why i use the term common) during the transaction to the and of it so that it is possible to run this workload only one time. Nothing more. If we don't want to add it into an install sctiptlet, then we could even add an trigger array to PKGBUILD to give the packager the possibility to ask for a trigger. I argue that hooks try to achieve a very different thing and should be designed in a different way than the end-of-transaction workload triggers because they are different things.

...

The question is do we really need this chain of rules for each proposed hook in libalpm thing which needs to scan for all the defined filename or package-name rules and will it be worth the effort to implement it if just a few packages ever will make use of the hooks. How often will we look at matching hook rules during normal operation? I'll bet not quite often.

I think this approach would also waste quite a few resources on every transaction to search for matches without even knowing if some package would trigger such an event. It has to do that every time as there is no "i want this hook to get fired up" action from the package-side as it would in the trigger approach. I think to list the content of one or more directories and fire up scriptlets based on files found would clearly outperform the scanning of every file that gets installed based on N rules we need to match against.

Don't we already generate lists of files for conflict checking? So scanning that list for matches to hooks will be a relatively low cost operation.

Do we have information about what will get removed and what will get added to the filesystem after the conflict checking is done? So we need hook scripts that might take a second param and are fired up for every file for the info, man-db or gconf workload? $1 : filename $2 (file-action): add or remove

...

I think we could achieve anything we could think of with the trigger approach from the packagers perspective except giving the user the ability to install hooks on this users self defined event-rules...

`If a user really wants to add a script which fires up if some filename based event is fulfilled than this user should take a look at the incron package and use it for what it was made IMO.

Yes, and using incron does somewhat work for installing info pages. But it fails because it does not monitor changes in subdirectories. In the case of info pages, users would then need to have incron running for something a package manager for a distro should do.

Please correct me if I am wrong, but the reasons I do not like the "triggers" idea are: 1) I need to set off the trigger in the install file. So this just replaces copying one prototype with another. 2) It requires regenerating databases rather than updating them. That overhead will be large for some operations (e.g. gconf)

1: As stated above, triggers are activated by packages that want them by design as they try to achieve to do workload that is currently done many times on large transactions only once at the end of them. So there is a need to add the information to the package itself. 2: That's irrelevant for the trigger approach as i wanted to just show that you can achieve more or less the same results with triggers. The install-info thing was an example, nothing more. I fully agree with you on that for this kind of workload the hook system would be a clean implementation and possibly the right thing to do. I'm not against implementing the hook approach at all, but i think the triggers are a good functionality and even easy to implement. And as they don't operate on the "fired up by a file add or remove" paradigm that the hooks would use, they should be implemented in a different way. Also i still have the opinion that the configuration of a hook-script should be contained in the script itself as this is the only way that i could come up with at the moment that would activate the hook automatically on installation of the hook. Another way would be to add the hook configuration into the database and parse this info from there. But the user created hooks would only work if the user builds his own hook-package then... Another problem with hooks and the points on which they should be executed is that if you have a large transaction in which some packages add hooks that are needed for packages that are installed in the same transaction need to be parsed on installation time to give you the option to run the hooks on the point they should be executed. Also what about a package that wants hook X needs to be installed before the hook X is added by another package? Would be a classic chicken-egg problem i think. Trigges like i proposed them don't have this problem as they get executed after the transaction is completed. At this point all triggers are where they should be. Marc

Xavier schrieb:

On Sat, Sep 12, 2009 at 1:21 PM, Marc - A. Dahlhaus wrote:

...

Another problem with hooks and the points on which they should be executed is that if you have a large transaction in which some packages add hooks that are needed for packages that are installed in the same transaction need to be parsed on installation time to give you the option to run the hooks on the point they should be executed. Also what about a package that wants hook X needs to be installed before the hook X is added by another package? Would be a classic chicken-egg problem i think. Trigges like i proposed them don't have this problem as they get executed after the transaction is completed. At this point all triggers are where they should be.

We also wanted to support actions which are only executed after the transaction is completed : http://wiki.archlinux.org/index.php/User:Allan/Pacman_Hooks#Types_of_Hook

The differences if that they would be triggered automatically by a file, instead of manually triggered by the packages themselves.

I actually have read the text on the wiki and i know that there is a transaction hook mentioned there. Did you think about the case where a common task could be wanted if there isn't any filename based rule that could fire it up? An example could be that a package adds some functionality that is an optional dependency for a job. If the dependency is not installed on the jobs first execution than some thinks are not done. If the optional dependency get installed at a later point it could ask for the trigger of the dependency and activate the missing things. I argue that they should be differently handled because of situation where the "file X got installed so we need to start Y" will not work. Marc

Xavier schrieb:

On Sat, Sep 12, 2009 at 2:19 PM, Marc - A. Dahlhaus wrote:

...

I actually have read the text on the wiki and i know that there is a transaction hook mentioned there.

Did you think about the case where a common task could be wanted if there isn't any filename based rule that could fire it up?

An example could be that a package adds some functionality that is an optional dependency for a job. If the dependency is not installed on the jobs first execution than some thinks are not done. If the optional dependency get installed at a later point it could ask for the trigger of the dependency and activate the missing things.

I argue that they should be differently handled because of situation where the "file X got installed so we need to start Y" will not work.

I read your message like 10 times, and I am not sure I understand what you are saying. Maybe if you could provide a more concrete example, it would be clearer for everyone :)

I mean there are cases where you don't have a file that gets installed which is related to the hook scripts original target of files. As the packager of the package that provides hook X also lays down the rules on which hook X get executed, there is no way for package Y if it wants to activate hook X to do so if it isn't providing any file on which hook X gets executed. An example could be that you have a Hook for Apache httpd which reacts on the filename filter "/etc/httpd/conf.d/*.conf" and does "apachectl -t && apachectl -k restart" for example. Package mod_php for example installs /etc/httpd/conf.d/mod_php.conf and everything is working fine, php gets activated and anyone could have a nice day. But now the user adds php_apc to his system which in turn only installs the file /etc/php.d/apc.ini to the system which would not activate php_apc for mod_php because the httpd-restart hook doesn't get executed. In the trigger approach the trigger is like in the real world a thing you have to push to activate it... The packager of php_apc knows if httpd is installed and mod_php is installed then i have to trigger the httpd-restart hook for example. The packager of httpd could not know how the other packages that might relay on his hook want to use it and what files should activate it. Might be a bad example because in a sysvinit world the httpd.init would do that on "service httpd restart" but i think it makes my point very clear. A user could have a self created repo with some packages that should add something by activating some hook from some upstream package... You can't handle it this way. I think the trigger approach is useful even if it it requires activation from the packagers side. Marc

Dan McGee schrieb:

On Sat, Sep 12, 2009 at 8:52 AM, Marc - A. Dahlhaus wrote:

...

Xavier schrieb:

...

On Sat, Sep 12, 2009 at 2:19 PM, Marc - A. Dahlhaus wrote:

...

I actually have read the text on the wiki and i know that there is a transaction hook mentioned there.

Did you think about the case where a common task could be wanted if there isn't any filename based rule that could fire it up?

An example could be that a package adds some functionality that is an optional dependency for a job. If the dependency is not installed on the jobs first execution than some thinks are not done. If the optional dependency get installed at a later point it could ask for the trigger of the dependency and activate the missing things.

I argue that they should be differently handled because of situation where the "file X got installed so we need to start Y" will not work.

I read your message like 10 times, and I am not sure I understand what you are saying. Maybe if you could provide a more concrete example, it would be clearer for everyone :)

I mean there are cases where you don't have a file that gets installed which is related to the hook scripts original target of files. As the packager of the package that provides hook X also lays down the rules on which hook X get executed, there is no way for package Y if it wants to activate hook X to do so if it isn't providing any file on which hook X gets executed.

An example could be that you have a Hook for Apache httpd which reacts on the filename filter "/etc/httpd/conf.d/*.conf" and does "apachectl -t && apachectl -k restart" for example.

Package mod_php for example installs /etc/httpd/conf.d/mod_php.conf and everything is working fine, php gets activated and anyone could have a nice day.

But now the user adds php_apc to his system which in turn only installs the file /etc/php.d/apc.ini to the system which would not activate php_apc for mod_php because the httpd-restart hook doesn't get executed.

In the trigger approach the trigger is like in the real world a thing you have to push to activate it...

The packager of php_apc knows if httpd is installed and mod_php is installed then i have to trigger the httpd-restart hook for example.

The packager of httpd could not know how the other packages that might relay on his hook want to use it and what files should activate it.

Might be a bad example because in a sysvinit world the httpd.init would do that on "service httpd restart" but i think it makes my point very clear.

A user could have a self created repo with some packages that should add something by activating some hook from some upstream package... You can't handle it this way.

I think the trigger approach is useful even if it it requires activation from the packagers side.

Hahahaha restarting daemons automatically... you haven't been here long, have you? :)

A silly example, i know ;o) This chained dependency was the fist that got trough my mind and Xavier asked for an example... (Debian does such a thing IIRC...)

Anyway, isn't it easy enough to just have the package install a dummy file somewhere that the hook is watching in order to trigger it? As long as this magical httpd-restart hook exists and is watching something like /var/lib/pacman/hooks/httpd-restart/, there is no problem.

That would work, but is more an hack to get this functionality in the hook design draft instead of a clean design... How to solve the other problems that i spotted in the hook design draft? <quote> Also i still have the opinion that the configuration of a hook-script should be contained in the script itself as this is the only way that i could come up with at the moment that would activate the hook automatically on installation of the hook. Another way would be to add the hook configuration into the database and parse this info from there. But the user created hooks would only work if the user builds his own hook-package then... Another problem with hooks and the points on which they should be executed is that if you have a large transaction in which some packages add hooks that are needed for packages that are installed in the same transaction need to be parsed on installation time to give you the option to run the hooks on the point they should be executed. Also what about a package that wants hook X needs to be installed before the hook X is added by another package? Would be a classic chicken-egg problem i think. Triggers like i proposed them don't have this problem as they get executed after the transaction is completed. At this point all triggers are where they should be. </quote> Marc

Xavier schrieb:

On Sat, Sep 12, 2009 at 9:38 PM, Marc - A. Dahlhaus wrote:

...

How to solve the other problems that i spotted in the hook design draft?

I think these problems are the same for hooks and triggers, and they can both be solved.

...

<quote> Also i still have the opinion that the configuration of a hook-script should be contained in the script itself as this is the only way that i could come up with at the moment that would activate the hook automatically on installation of the hook. Another way would be to add the hook configuration into the database and parse this info from there. But the user created hooks would only work if the user builds his own hook-package then...

packages could just install filehook config in a hook.d directory

...

Another problem with hooks and the points on which they should be executed is that if you have a large transaction in which some packages add hooks that are needed for packages that are installed in the same transaction need to be parsed on installation time to give you the option to run the hooks on the point they should be executed. Also what about a package that wants hook X needs to be installed before the hook X is added by another package? Would be a classic chicken-egg problem i think. Triggers like i proposed them don't have this problem as they get executed after the transaction is completed. At this point all triggers are where they should be. </quote>

filehooks can also work at transaction level

Please take a closer look at the proposed events and what it means in an implementation to get them going on large transaction like a full system installation (copied from the wiki): * Install - run on installation of a matching file/package * PostInstall - run after installation of a matching file/package * PreRemove - run prior to removal of a matching file/package * PostRemove - run after to removal of a matching file/package * Transaction - run at end of transaction involving at least one matching file I think that to get there for all of them, you have to constantly rescan the hook.d directory during the whole transaction to load the freshly added configurations and hooks and also have to rescan the whole list of already installed or removed files in the running transaction for every freshly added hook to solve the chicken-egg problems and installation of new hooks during transaction. I think that this is a huge design problem and it would be very interesting to do a clean implementation of such a thing. ;o) I don't think that the implementation of this is a fun job to do, but i also hope that you get anywhere with this draft as it would be a nice thing to have for things like info-pages... As i couldn't convince you even a little bit at all that triggers would be a nice thing to have in pacman so far instead of having a draft for something that nobody wants to implement at the moment, i'll go ahead and implement it tomorrow as i want to use such a functionality better sooner than later... If anybody is interested in it and wants to test it, drop me a line... As i don't want to waste more of your time with the triggers approach, i will not post it for review but i'm happy and willing to discuss more about the hook approach. ;-) Marc

Xavier schrieb:

On Sat, Sep 12, 2009 at 10:29 PM, Marc - A. Dahlhaus wrote:

...

Please take a closer look at the proposed events and what it means in an implementation to get them going on large transaction like a full system installation (copied from the wiki):

* Install - run on installation of a matching file/package * PostInstall - run after installation of a matching file/package * PreRemove - run prior to removal of a matching file/package * PostRemove - run after to removal of a matching file/package * Transaction - run at end of transaction involving at least one matching file

I think that to get there for all of them, you have to constantly rescan the hook.d directory during the whole transaction to load the freshly added configurations and hooks and also have to rescan the whole list of already installed or removed files in the running transaction for every freshly added hook to solve the chicken-egg problems and installation of new hooks during transaction.

You are right, I was thinking about these problems. I guess I would just ignore the chicken-eggs problem (we already have to deal with this anyway with the current scriptlets, and it seems to work good enough ...), and also limit the features and functionality to make the implementation more doable :)

mhm... i have to think more about it... The chicken-egg problems in the current scriptlets, well... i solve them using a trigger :-P

We do not have to rescan the whole hooks.d directory, we could simply monitor files added there, and just reload this file. And this new hook would only be applied for the following packages.

inotify or FAM based? FAM would be the more portable way and gamin would use inotify on linux...

...

As i couldn't convince you even a little bit at all that triggers would be a nice thing to have in pacman so far instead of having a draft for something that nobody wants to implement at the moment, i'll go ahead and implement it tomorrow as i want to use such a functionality better sooner than later...

If anybody is interested in it and wants to test it, drop me a line...

this thread is getting huge, I think one of my mail got lost in the deep, when I was thinking about alternatives to implement your trigger idea :

That one was slipped into another mailbox here (by a keyword filter on the string "Edit : " as we use it in our svn commit mails)...

Otherwise, it seems that Marc is saying that install scriptlets are fine, and he wants to add two functionality : 1) transaction level hook 2) sharing code

1 could be even more easily achieved by just defining a new function in scriptlets, eg transaction()

This would make the case impossible where you want to run a common task only once for the whole transaction even if more than one package wants the workload (eg. mkinitramfs after kernel got updated and some modules got updated or installed during the same transaction) as transaction() would run for every package defining it...

2 could be achieved with function libraries (see http://bugs.archlinux.org/task/10375)

I thought of something like that for scriptlets also. My plan was to just source a script with common functions into the scriptlet execs before the scriptlet itself gets sourced into the running sh... For makepkg i had sourcing of some helper functions for placing sysvinit symlinks into rc?.d (was the most common code duplication here) in the PKGBUILDs but as we migrated over to upstart so we don't needed that anymore...

Again here, it is not clear to me which method is better, I am just throwing out ideas.

Well, so far i got much headwind against the trigger idea... Marc