[arch-general] Ye Olde Package Manager

[C Anthony Risinger]

On Sat, Jan 15, 2011 at 3:34 PM, Brendan Long <korin43 at gmail.com> wrote:
> On 01/13/2011 12:12 PM, C Anthony Risinger wrote:
>> leverage a revisioning system for package files instead of tarballs,
>> even if only locally.  store metadata in a non-relational engine like
>> couchdb (peer replicaition), or at least something like sqlite, for
>> sane access.
>
> A relational engine is actually really helpful for packages. A while ago
> I tried writing a package manager like pacman but using sqlite, and it's
> MUCH faster and still easy to use. The huge pauses every time you need
> metadata are incredibly annoying, and they completely disappear when you
> store things in a real database. The problem is that it has to be used
> by the official package manager, because having package data stored in
> two formats causes issues (because any time you use pacman, the other
> database doesn't know what changed).

yeah it would help with speed, but that's not really my focus; i seek
greater intelligence.  i find the "package manager" to be an
interesting _core_ commonality between all Linuxes, with the potential
to really open up and become an incredible tool for
developing/sharing/swapping state with other users in a secure,
ad-hoc, distributed, and mostly distro-agnostic manner.  think "social
network", at the developer/distribution/packager/user level, but where
the focus is on creation and knowledge expansion vs. spamming your
friends all day and playing pointless games.

a lofty goal, but nothing's unreachable when everything's already
right next to you.

> Revisioning package files is also interesting; I don't see the point of
> doing it locally though. Once you have the package, installing it is
> fast. Checking if files are the same first seems like a waste of effort.
> There already is a mechanism for creating those .pacnew files (and I
> think auto-merging those into the existing file would mess with the
> "knowing what your system is doing" part of Arch). Using deltas for
> packages would be helpful though, especially in the case of huge
> packages with minor changes.

again speed is not relevant (though the tar backend thing from Allan's
msg sounds kinda cool for this).  awhile back i had a small script
that would update and "eat" my pacman cache nightly into a packed git
repository.  with this i could export an immediately installable
package or list of packages, from any day.  so even though pacman
itself was oblivious, having it local was still useful.

nowadays i just use my btrfs hooks and rsync scripts to snapshot my
systems at various points of interest (ie. before pacman upgrades, on
reboot, nightly, etc) to ensure stability.

i see no problem with "auto merging" if it's done in a controlled,
consistent, and reproducible manner.  look into redhat's `augeas` --
it loads any defined native config format into a tree than can be
merged/manipulated/inspected before being written back out in the
native format.  they'll use this to provide clean control over VM host
networks from virt-manager, on any distro with definitions.  i'd trust
that over sed any day; these sorts of config "suggestions" could be
made visible before commit, similar to gentoo.

ultimately, package managers should let the user decide their own
level of involvement.  for instance, all i want to do is verify
correctness... i really don't care to manually fuxxor around doing
what my system could handle on it's own.  i'm a jelly filled human
doughnut; my error rate is rather high.

> The rest of your changes sound like things that would make packaging
> harder, and you should know that some (most?) of us like Arch's packages
> because they're easy to make. It make seem overly simple, but that's
> exactly what I want out of a package: Give a name, version number,
> source, and dependencies, then the commands I'd use to build it, and I'm
> done. If Arch every became like Debian with it's fancy, huge-time-sink
> packaging, I'd find a different distro.

yeah i've been around long enough to get the general vibe...  but
really... do you actually _enjoy_ making packages?  do you like it
when things break (even if not often, i'm not bashing arch developers
or anyone else here) because of small version mismatches/typos/etc.
due to the constant requirement for human interaction every step of
the way?  do you appreciate the system requiring an unknown amount of
your (limited) time each day you decide to update?  don't you ever
wish you could just say "hey computer 'ol pal, aggressively follow
upstream source for package X and merge remote user Y's with the local
configuration, unless either requires changes to package Z -- then ask
me first, cuz i run the show here"?

i've written several packages and maintain numerous arch based
hosts/VMs servicing my network, with other distros preceding that;
maintaining packages is 100% annoying. it's an _absolute_ waste of
time from a productivity standpoint; you're neither improving the
source nor utilizing it... your just trying your best to make it play
nice with itself and others.  we normally use computers to perform
these sorts of tedious tasks as much as possible, but package managers
have barely changed since they we first conceived.... kind of like
"PID 1" (until recently! oh how ye welcometh new thought like upstart
and _especially_ systemd)... we grant these applications the most
awesomest superpowers then give it the IQ of 17 or so.

ehm... sense?  i'm simply calling for more vision on this front.

i see packages as vectors that collectively determine your system's
"plane"; they should be guided as needed, not break down every time
the upstream/peerstream changes.

bash is great and all, but a solid architecture implementing next-gen
system/package management, along with maybe enabling upstreams to
embed "hints" as needed (with verification/fallback via analysis and
local/peer heuristics), will be more reliable + maintainable than your
PKGBUILD will ever be.  possibly even simpler.  look to system state
managers like puppet for some inspiration behind these concepts.

-------------------------------------------------------------------------------

please note this is not something i expect arch or pacman to realize;
arch and the team behind it does a good job and i am pleased with the
results and most decisions.  i simply tend to rant now and again on
the topic because i think it's a major, if not the primary, barrier of
entry to Linux-based OSes at large.

bleh... i'll create my own distro soon enough and add to the problem,
just need to finish the state manager to manage it :-) progress be
slow, but it's regular; got an 19mo child at large around here.

well that was fun!  i'm out.  see you all next time.

/end whatever you want to call the 4AM writings above this line/

C Anthony