[pacman-dev] [PATCH v2 1/3] bacman: allow for parallel packaging

[Dave Reisner]

On Wed, Aug 31, 2016 at 10:35:37PM +0200, Gordian Edenhofer wrote:
> On Tue, 2016-08-30 at 20:30 +1000, Allan McRae wrote:
> > On 15/08/16 06:39, Gordian Edenhofer wrote:
> > > 
> > > * move the actual assembly process into its own function
> > > * allow for packaging multiple packages with one command
> > > * handle SIGHUP SIGINT SIGTERM and remove working dirs accordingly
> > > * add some comments
> > > 
> > > Signed-off-by: Gordian Edenhofer <gordian.edenhofer at gmail.com>
> > > ---
> > > * Clean up working directories with force on abort
> > > * Sleep for 100ms in between parallelization work
> > 
> > Why sleep?
> 
> Sleep is invoked to reduce the CPU usage caused by busy-waiting.
> The approach for parallelization is straight forward and simple. It
> checks every n seconds whether enough jobs are running and in case not
> it starts a new one. Since this is not an elegant way to do it, I added
> a note which incorporates Dave Reisner hint to use wait -n at some
> point in the future when bash 4.3 is available at every system.
> 
> > Two patches are needed:
> > 
> > 1) allow bacman to take multiple package arguments
> > 2) parallelize...
> 
> I will submit a second patch very soon which will address your
> complaints concerning my commit division.
> 
> > The second probably would not be accepted...
> 
> I urge you to reconsider. Parallelization increases the speed of this

I don't think anyone is suggesting that packaging multiple things in
parallel isn't useful. I already suggested that nothing needs to be
implemented in bacman proper in order for you to parallelize the work.
You can write your own "pbacman" as simply as:

  for arg; do bacman "$arg" & done; wait

> tool immensely. To proof my point I made some test on an Intel NUC with
> an i5-4250U dual-core CPU. The results [1] clearly show that when using
> all available four threads the time needed to assemble various chunks
> of packages is sliced in half.
> The graphs show the time required for a group of packages to be
> recreated with bacman. To show the potential of parallelization I
> chose: base + base-devel, base + base-devel + gnome and finally I
> assembled all packages installed on the system. Packages contained in
> multiple groups were only assembled once. All chosen package groups
> were recreated using 1-6 jobs.
> The respective measured values and code can be seen here [2].
> 
> Best Regards,
> Gordian Edenhofer
> 
> [1] http://edh.ddns.net/pacman_ml_bacman_benchmarks/bacman:%20simple%20
> benchmark.svg
> [2] http://edh.ddns.net/pacman_ml_bacman_benchmarks/bacman:%20simple%20
> benchmark.R.txt