[arch-general] Location of the pacman database

newer
[arch-general] Moving chmsee to AUR

Yamakaky

10 Sep 2014 10 Sep '14

6:34 p.m.

Hi ! Why is the pacman local database location defaulted to /var/lib/pacman ? It should go in /usr as it reflects the installed packages and it is only updated when a package is updated. If it where in /usr, it would be easier to share the /usr partition between hosts. What do you think ?

Show replies by date

Joel Teichroeb

10 Sep 10 Sep

6:50 p.m.

Generally, things that are written to are not stored in /usr. This page might help https://wiki.archlinux.org/index.php/arch_filesystem_hierarchy

Yamakaky

7:05 p.m.

Le 10/09/2014 20:50, Joel Teichroeb a écrit :

...

Generally, things that are written to are not stored in /usr.

This page might help

https://wiki.archlinux.org/index.php/arch_filesystem_hierarchy

That's what I thought at first, but it's different. pacman db doesn't contain runtime stuff, it's a description of the currently installed packages. It's a bit like putting os-release in /usr/lib.

Leonid Isaev

7:21 p.m.

On Wed, Sep 10, 2014 at 09:05:03PM +0200, Yamakaky wrote:

...

Le 10/09/2014 20:50, Joel Teichroeb a écrit :

...
Generally, things that are written to are not stored in /usr.

This page might help

https://wiki.archlinux.org/index.php/arch_filesystem_hierarchy

That's what I thought at first, but it's different. pacman db doesn't contain runtime stuff, it's a description of the currently installed packages. It's a bit like putting os-release in /usr/lib.

I htink you are right about the local DB (/var/lib/pacman/local). However, /var/lib/pacman/sync should probably stay in /var (I don't need a rw root FS to resync package DB). Anyway, I think you should open a feature request at the bugtracker. Cheers, -- Leonid Isaev GPG fingerprints: DA92 034D B4A8 EC51 7EA6 20DF 9291 EE8A 043C B8C4 C0DF 20D0 C075 C3F1 E1BE 775A A7AE F6CB 164B 5A6D

Yamakaky

7:24 p.m.

...

I htink you are right about the local DB (/var/lib/pacman/local). However, /var/lib/pacman/sync should probably stay in /var (I don't need a rw root FS to resync package DB).

Yes, of course. I was talking about the local/ database.

Yamakaky

7:37 p.m.

...

Anyway, I think you should open a feature request at the bugtracker.

Just watched : there is already this bug report : https://bugs.archlinux.org/task/41863

Neven Sajko

9:59 p.m.

...

Just watched : there is already this bug report : https://bugs.archlinux.org/task/41863

But that task is not only about the filesystem location of the Pacman DB. It could be useful to open a separate bugreport just for that one issue. On 10 September 2014 21:37, Yamakaky wrote:

...

Anyway, I think you should open a feature request at the

...
bugtracker.

Just watched : there is already this bug report : https://bugs.archlinux.org/task/41863

Tobias Hunger

13 Sep 13 Sep

1:55 p.m.

On Wed, Sep 10, 2014 at 11:59 PM, Neven Sajko wrote:

...

But that task is not only about the filesystem location of the Pacman DB. It could be useful to open a separate bugreport just for that one issue.

For what reason should the pacman DB be moved if not to enable new features like the factory reset or image based installation? I can file separate bugs for all the items listed in https://bugs.archlinux.org/task/41863 , but I personally do not think it makes a lot of sense. Best Regards, Tobias

Leonid Isaev

5:12 p.m.

On Sat, Sep 13, 2014 at 03:55:26PM +0200, Tobias Hunger wrote:

...

On Wed, Sep 10, 2014 at 11:59 PM, Neven Sajko wrote:

...
But that task is not only about the filesystem location of the Pacman DB. It could be useful to open a separate bugreport just for that one issue.

For what reason should the pacman DB be moved if not to enable new features like the factory reset or image based installation?

I can file separate bugs for all the items listed in https://bugs.archlinux.org/task/41863 , but I personally do not think it makes a lot of sense.

Yeah, that's what the 1st response in the bug report basically said: pacman DB location is a cosmetic detail. Also, note that systemd features like factory reset and/or atomic updates make no sense in the context of a rolling-release distro like Arch (or any distribution for that matter), so I doubt that they can be a sufficient motivation for this DB move...

...

Best Regards, Tobias

Cheers, -- Leonid Isaev GPG fingerprints: DA92 034D B4A8 EC51 7EA6 20DF 9291 EE8A 043C B8C4 C0DF 20D0 C075 C3F1 E1BE 775A A7AE F6CB 164B 5A6D

Tobias Hunger

14 Sep 14 Sep

9:41 a.m.

On Sat, Sep 13, 2014 at 7:12 PM, Leonid Isaev wrote:

...

Yeah, that's what the 1st response in the bug report basically said: pacman DB location is a cosmetic detail.

No, it is not: /var will be wiped, so having the pacman DB there is not a good idea.

...

Also, note that systemd features like factory reset and/or atomic updates make no sense in the context of a rolling-release distro like Arch (or any distribution for that matter), so I doubt that they can be a sufficient motivation for this DB move...

I completely disagree: Factory reset is great, especially for a distro involving a lot of manual tweaking like arch:-) With factory reset you always know how to undo your own changes, getting back to the default state. That works for either all changes ever done to the system (factory reset) or selectively by just removing the configuration files you tweaked last. So factory reset is nice, but the image based installation is the best thing since sliced bread. I do that for a couple of month now (using ostree, not the new systemd way). It is a game changer. Basically I create new images of *all* my machines (physical machines, VMs and docker images) each night. That is really easy to do with arch's pacstrap and a bit of configuration that gets applied on top (and moving a couple of files around, e.g. the pacman DB;-). I then store the store the physical machine images in ostree (I am currently changing that to the subvolume approach systemd recently suggested). Now I can test the my new arch snapshot in the new image by either using systemd-nspawn or by starting it in a VM. I can also just reboot into the new image... if it breaks, the old, working image is just another reboot away. Moving the pacman DB is one step to make such a setup a bit more easy to create and It does not effect the traditional use case at all. You can even put the pacman db into e.g. /usr/lib/pacman/db for new installations and fall back to the current location in /var if there is no DB there. Best Regards, Tobias

Neven Sajko

2:06 p.m.

...

Moving the pacman DB is one step to make such a setup a bit more easy to create and It does not effect the traditional use case at all.

That's why I suggested putting it in a separate bugreport; it gets accepted more easily, and then less change is needed for 41863. On 14 September 2014 11:41, Tobias Hunger wrote:

...

On Sat, Sep 13, 2014 at 7:12 PM, Leonid Isaev wrote:

...
Yeah, that's what the 1st response in the bug report basically said: pacman DB location is a cosmetic detail.

No, it is not: /var will be wiped, so having the pacman DB there is not a good idea.

...
Also, note that systemd features like factory reset and/or atomic updates make no sense in the context of a rolling-release distro like Arch (or any distribution for that matter), so I doubt that they can be a sufficient motivation for this DB move...

I completely disagree:

Factory reset is great, especially for a distro involving a lot of manual tweaking like arch:-) With factory reset you always know how to undo your own changes, getting back to the default state. That works for either all changes ever done to the system (factory reset) or selectively by just removing the configuration files you tweaked last.

So factory reset is nice, but the image based installation is the best thing since sliced bread. I do that for a couple of month now (using ostree, not the new systemd way). It is a game changer.

Basically I create new images of *all* my machines (physical machines, VMs and docker images) each night. That is really easy to do with arch's pacstrap and a bit of configuration that gets applied on top (and moving a couple of files around, e.g. the pacman DB;-). I then store the store the physical machine images in ostree (I am currently changing that to the subvolume approach systemd recently suggested).

Now I can test the my new arch snapshot in the new image by either using systemd-nspawn or by starting it in a VM. I can also just reboot into the new image... if it breaks, the old, working image is just another reboot away.

Moving the pacman DB is one step to make such a setup a bit more easy to create and It does not effect the traditional use case at all. You can even put the pacman db into e.g. /usr/lib/pacman/db for new installations and fall back to the current location in /var if there is no DB there.

Best Regards, Tobias

Bigby James

3:46 p.m.

On 09/14, Tobias Hunger wrote:

...

Factory reset is great, especially for a distro involving a lot of manual tweaking like arch:-) With factory reset you always know how to undo your own changes, getting back to the default state. That works for either all changes ever done to the system (factory reset) or selectively by just removing the configuration files you tweaked last.

Frankly, with the existence of LVM and BTRFS snapshots, revision control systems and incremental backups, I can't for the life of me understand what use the overwhelming majority of Archers would have for a "factory reset" feature. The only people I've ever known to use a factory reset feature are Windows users who couldn't be bothered to make backups. Besides, maybe I'm missing something here, but since Arch uses upstream settings as its defaults in almost every case, "factory reset" provides the exact same functionality as reinstalling from scratch. There's really no sense in the Arch devs and all Arch users go through the trouble, from what I can see. You're asking everyone who uses Arch to make significant changes (to more than just the pacman db) to accomodate corner cases. -- "A common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools." - Douglas Adams

Yamakaky

5:17 p.m.

...

With factory reset you always know how to undo your own changes, getting back to the default state. That works for either all changes ever done to the system (factory reset) or selectively by just removing the configuration files you tweaked last.

Woh, I didn't thought about it, it's pretty cool !

Yamakaky

5:23 p.m.

Le 14/09/2014 19:17, Yamakaky a écrit :

...

...
With factory reset you always know how to undo your own changes, getting back to the default state. That works for either all changes ever done to the system (factory reset) or selectively by just removing the configuration files you tweaked last.

With default config files in /usr, it's easier to experiment !

Conrad Nelson

5:25 p.m.

On Sun, 2014-09-14 at 19:23 +0200, Yamakaky wrote:

...

Le 14/09/2014 19:17, Yamakaky a écrit :

...
...
With factory reset you always know how to undo your own changes, getting back to the default state. That works for either all changes ever done to the system (factory reset) or selectively by just removing the configuration files you tweaked last.

With default config files in /usr, it's easier to experiment !

Might also be useful for removing cruft after upgrades? Though in this case I'm thinking about discreet distributions like Debian Stable. Conrad

Nowaker

9:15 p.m.

Hey guys, I'm developing VirtKick (www.virtkick.io) and the very first supported hypervisor will be Arch Linux. "Factory reset" feature would really fit my use case.

...

There's really no sense in the Arch devs and all Arch users go through the trouble, from what I can see. You're asking everyone who uses Arch to make significant changes (to more than just the pacman db) to accomodate corner cases.

Let's first move /var/lib/pacman/local to /usr, as this very directory really belongs to /usr, and then see what we have left to do yet. At this point /var/lib/pacman/local defines the current state of /usr. It's not "variable" - you write to /var/lib/pacman/local if and only if you write to /usr. The description of /usr on wiki perfectly describes why /var/lib/pacman/local really belongs there:

...

This means that /usr shall be shareable between various hosts and must not be written to, except by the package manager (installation, update, upgrade)

What are these significant changes more than just the pacman database that would make users go through trouble? In #41863 I see three parts: - move /var/lib/pacman/local/ to /usr - move the default pacman.conf and mirrorlist to /usr/share - provide tmpfiles.d to copy those files to /etc If I'm not mistaken, /usr/share and tmpfiles.d are really trivial and wouldn't affect users in any way. That'd be a few additional files somewhere in the filesystem without any effect on existing machines. Or I'm wrong?

...

I then store the store the physical machine images in ostree (I am currently changing that to the subvolume approach systemd recently suggested).

Yamakaky, can you please provide some more info on this? And generally, could you show me your workflow - the shell commands in particular? That would explain things better than thousands of words. :-) -- Kind regards, Damian Nowak StratusHost www.AtlasHost.eu

Leonid Isaev

9:55 p.m.

On Sun, Sep 14, 2014 at 11:15:13PM +0200, Nowaker wrote:

...

At this point /var/lib/pacman/local defines the current state of /usr. It's not "variable" - you write to /var/lib/pacman/local if and only if you write to /usr. The description of /usr on wiki perfectly describes why /var/lib/pacman/local really belongs there:

So, files in (now) /usr/lib/pacman/local contain filelists of packages, yes? If you wipe /var, lots of packages will have missing files...

...

- move /var/lib/pacman/local/ to /usr - move the default pacman.conf and mirrorlist to /usr/share - provide tmpfiles.d to copy those files to /etc

What about pacman keyring? Also note that your custom keys should be packaged as well and resigned on-boot.

...

If I'm not mistaken, /usr/share and tmpfiles.d are really trivial and wouldn't affect users in any way. That'd be a few additional files somewhere in the filesystem without any effect on existing machines. Or I'm wrong?

This is madness. I remember sometime ago there was a witchhunt against daemons that write to /etc (cups is the worst offender). So why is it OK for systemd to do so? I personally don't want systemd to come anywhere near my /etc. Please package the tmpfiles.d/sysusers stuff with virtkick or whatever, but not with pacman. Cheers, -- Leonid Isaev GPG fingerprints: DA92 034D B4A8 EC51 7EA6 20DF 9291 EE8A 043C B8C4 C0DF 20D0 C075 C3F1 E1BE 775A A7AE F6CB 164B 5A6D

Nowaker

10:53 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

...

...
At this point /var/lib/pacman/local defines the current state of /usr. It's not "variable" - you write to /var/lib/pacman/local if and only if you write to /usr. The description of /usr on wiki perfectly describes why /var/lib/pacman/local really belongs there:

...

So, files in (now) /usr/lib/pacman/local contain filelists of packages, yes? If you wipe /var, lots of packages will have missing files...

Good point. I just did `pacman -Ql |grep -F ' /var'` to see how many files there are. 99.7% of them are directories only, though. Are tmpfiles.d supposed to create directories in /var too? Docs mention using tmpfiles.d to init /tmp or /run, not /var though. But I guess stateless systemd would always provide tmpfiles for that.

...

...
- move /var/lib/pacman/local/ to /usr - move the default pacman.conf and mirrorlist to /usr/share - provide tmpfiles.d to copy those files to /etc

...

What about pacman keyring? Also note that your custom keys should be packaged as well and resigned on-boot.

I wasn't aware of that. I only refer to what the OP requested and that didn't sound complicated at all. Now it does.

...

...
If I'm not mistaken, /usr/share and tmpfiles.d are really trivial and wouldn't affect users in any way. That'd be a few additional files somewhere in the filesystem without any effect on existing machines. Or I'm wrong?

...

This is madness. I remember sometime ago there was a witchhunt against daemons that write to /etc (cups is the worst offender). So why is it OK for systemd to do so? I personally don't want systemd to come anywhere near my /etc. Please package the tmpfiles.d/sysusers stuff with virtkick or whatever, but not with pacman.

Please note I'm not a huge fan of systemd. I'd rather Arch hadn't married systemd back then. But since it has already happened, many parts of Arch make use of systemd (e.g. netctl), there are several systemd contributors amongst Arch Linux developers, it'd be good to make use of various systemd features. As long as they are not costly to implement, of course. And this "factory reset" feature indeed seems to be costly. - -- Kind regards, Damian Nowak StratusHost www.AtlasHost.eu -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJUFhxzAAoJENUJYSy7yByfwkwP/jJkNBkXQZ/4nkq2/ZuVJHDL dRsaZ4GeTsgOQ6Q3JP7RQ67nvYiQhyUqIIQotvp+ZmT8rjNXkpOQBg9gUC3vycRk RoD38S2A4nDQcYeftM5nMjUYkSJnM01OBblrgw6+wK1NP1NNilgtklRX41Th5xmz bFGqD1b6QO7g5SrY0lelLJUYG3SkN4O8aFO5fKHpUSSaalvoIE6XlL3WtbLA7+nD NhVx4XwmcJMT9rqAlhhOYncm/dzssMdlkSAzTyIqrs4vOSxRvV1LUpcO294qeY31 jSO5vAGKjKMWANkeNUN5d0Ga/gLLEYXPsUNiA9UA4+N9PL9yRLeXO7AeOyG54pYl Ivp+iB5To9n98Vd9JPRThv3egjtod0vt8I75jB8VPcR2ZxFJjVHkA/9WR8yGsnYw 0IaQtmtz+f+DB1vJf+GEBVEZHpGukwOVcjN19qchs0Q0Ee59Kkzlr1jL7c1YbTzV WnYVmnnXIntXHNYBg+DbpJC/VOOmR0yt+tx8mpXToHK9u4rVwnOgw4J1qtS9vz1C 3a0gNdSqxuS4QSV1fTSreiUIuQVi5grFXXAkS2FD5Gh3gMx3kGmkLl/0VQj39puT /mKIkTXbt7hqqWDKSwX1dz7h6PrkYpD6gAeonzNi58UwdHEWIfBWHUSXkEzRnli3 MMBXhJdu6bInfO1wAaDd =ddj8 -----END PGP SIGNATURE-----

Leonid Isaev

11:55 p.m.

On Mon, Sep 15, 2014 at 12:53:40AM +0200, Nowaker wrote:

...

Good point. I just did `pacman -Ql |grep -F ' /var'` to see how many files there are. 99.7% of them are directories only, though. Are tmpfiles.d supposed to create directories in /var too? Docs mention using tmpfiles.d to init /tmp or /run, not /var though. But I guess stateless systemd would always provide tmpfiles for that.

Yes, those are mostly dirs. And you can't create them through tmpfiles because it's going to be a mess. But breaking pacman -Qk is relatively harmless, the bigger question is what to do with all the files in var (e.g. exim mailspools)? Wiping /var is not an option on a workstation/server and is OK only in special cases, like kiosk-type systems or Fedora installations (which are broken by default anyway) :). AFAIK, this has not been publicly discussed on systemd-devel.

...

...
What about pacman keyring? Also note that your custom keys should be packaged as well and resigned on-boot.

I wasn't aware of that. I only refer to what the OP requested and that didn't sound complicated at all. Now it does.

Of course, depending on your paranoia (and amount of free time), you can ask whether keyring should be shared among snapshots and similar questions...

...

Please note I'm not a huge fan of systemd. I'd rather Arch hadn't married systemd back then. But since it has already happened, many parts of Arch make use of systemd (e.g. netctl), there are several systemd contributors amongst Arch Linux developers, it'd be good to make use of various systemd features. As long as they are not costly to implement, of course. And this "factory reset" feature indeed seems to be costly.

systemd's factory reset and atomic upgrades were explicitly stated to be useful only in special situations, like embedded systems. Just because Archlinux systemd package enables them doesn't mean that the entire distribution should be change around. Implementation is easy, support in all usecases is hard. I mean, Archlinux is not CoreOS... Cheers, -- Leonid Isaev GPG fingerprints: DA92 034D B4A8 EC51 7EA6 20DF 9291 EE8A 043C B8C4 C0DF 20D0 C075 C3F1 E1BE 775A A7AE F6CB 164B 5A6D

Tobias Hunger

15 Sep 15 Sep

6:28 a.m.

On Sep 15, 2014 1:56 AM, "Leonid Isaev" wrote:

...

systemd's factory reset and atomic upgrades were explicitly stated to be useful only in special situations, like embedded systems. Just because Archlinux systemd package enables them doesn't mean that the entire distribution should be change around.

Yes, these are features for special use cases, I am not arguing that. I would still like these features to be easy to implement on Arch, provided the required changes do not harm the traditional setups.

...

Implementation is easy, support in all usecases is hard. I mean, Archlinux is not CoreOS...

I do not think the changes in my bug report carry a support overhead. Best Regards, Tobias

Leonid Isaev

5:30 p.m.

Hi, On Mon, Sep 15, 2014 at 08:28:22AM +0200, Tobias Hunger wrote:

...

On Sep 15, 2014 1:56 AM, "Leonid Isaev" wrote:

...
systemd's factory reset and atomic upgrades were explicitly stated to be useful only in special situations, like embedded systems. Just because Archlinux systemd package enables them doesn't mean that the entire distribution should be change around.

Yes, these are features for special use cases, I am not arguing that.

I would still like these features to be easy to implement on Arch, provided the required changes do not harm the traditional setups.

...
Implementation is easy, support in all usecases is hard. I mean, Archlinux is not CoreOS...

I do not think the changes in my bug report carry a support overhead.

My only point is that moving pacman DB around is the least difficult thing to deal with when it comes to bringing factory-reset and friends to Archlinux. I don't object to this move by itself, but the motivation seems ill-defined to me. Moreover, reshuffling things around will break lots of scripts our there that expect /var/lib/pacman/local to be populated... Cheers, -- Leonid Isaev GPG fingerprints: DA92 034D B4A8 EC51 7EA6 20DF 9291 EE8A 043C B8C4 C0DF 20D0 C075 C3F1 E1BE 775A A7AE F6CB 164B 5A6D

Yamakaky

7:04 a.m.

...

Yes, those are mostly dirs. And you can't create them through tmpfiles because it's going to be a mess. But breaking pacman -Qk is relatively harmless, the bigger question is what to do with all the files in var (e.g. exim mailspools)? Wiping /var is not an option on a workstation/server and is OK only in special cases, like kiosk-type systems or Fedora installations (which are broken by default anyway) :).

Let divide the problem, with a package containing /var/lib/prog/ and /etc/prog/conf created by tmpfiles : - If the program wrote to /var, then there is no problem, as pacman does not remove a non empty dir anyway. - If it doesn't, there is a problem as /var/lib/prog will not be deleted. - The config file will not be deleted by pacman anyway. I can't find a solution now, I will think at it.

...

systemd's factory reset and atomic upgrades were explicitly stated to be useful only in special situations, like embedded systems. Just because Archlinux systemd package enables them doesn't mean that the entire distribution should be change around.

Being able to just delete /etc/<package> and reboot/restart the program to reset its config isn't "only useful in special situations". As Tobias said, arch is configured by users who can make mistakes. Factory reset isn't only for the entire system. And it doesn't harm the other users.

Tobias Hunger

10:42 a.m.

On Mon, Sep 15, 2014 at 9:04 AM, Yamakaky wrote:

...

Let divide the problem, with a package containing /var/lib/prog/ and /etc/prog/conf created by tmpfiles :

- If the program wrote to /var, then there is no problem, as pacman does not remove a non empty dir anyway. - If it doesn't, there is a problem as /var/lib/prog will not be deleted. - The config file will not be deleted by pacman anyway.

I can't find a solution now, I will think at it.

I actually do not even see the problem:-) tempfiles.d for files in /etc are a bandaid in the first place: They are meant to work around daemons that absolutely insist on having a configuration file and do not fall back gracefully to a default configuration without one. A tempfiles.d for /var is more permanent since daemons will need to have directories set up for their use (with proper permissions, etc.). But even when a packge ships those tempfiles.d snippets, those will only take effect if the files/directories are *missing*. Nobody will mess with either /etc nor /var up to the point where you screwed up your system yourself by deleting things that are actually needed. So nothing prevents having /var/lib/whatever and /etc/whatever.conf in the pacman package and shipping a tempfiles.d snippet to create those if they ever go missing. Nobody will muck around in your /etc nor /var (at least as long as your system is sane;-), pacman -Qo /var/whatever will just work as well, even after a factory reset. If the user uninstalls the package, then the tmpfiles.d snippet will also be gone, so no problem there.

...

Being able to just delete /etc/<package> and reboot/restart the program to reset its config isn't "only useful in special situations". As Tobias said, arch is configured by users who can make mistakes. Factory reset isn't only for the entire system. And it doesn't harm the other users.

Well, I would not delete /etc/<package> and then reboot, I would just cp /usr/lib/factory/etc/<package> /etc/<package> and restart the service:-) Having a directory with the pristine distro-provided configuration is a really nice thing. Just being able to run a simple diff to see all the changes you ever did to /etc is really nice. Currently that is severely limited in a default arch setup by the factory being almost empty though:-) Best Regards, Tobias

Joakim Hernberg

11:02 a.m.

On Mon, 15 Sep 2014 12:42:41 +0200 Tobias Hunger wrote:

...

Having a directory with the pristine distro-provided configuration is a really nice thing. Just being able to run a simple diff to see all the changes you ever did to /etc is really nice. Currently that is severely limited in a default arch setup by the factory being almost empty though:-)

It's also not a bad practice to to copy a config file to say *.original before you edit it. Between that and .pacnew files it ought to be possible to see how you messed your config very easily. -- Joakim

Tobias Hunger

5:32 a.m.

Am 15.09.2014 00:54 schrieb "Nowaker" :

...

Good point. I just did `pacman -Ql |grep -F ' /var'` to see how many files there are. 99.7% of them are directories only, though. Are tmpfiles.d supposed to create directories in /var too? Docs mention using tmpfiles.d to init /tmp or /run, not /var though. But I guess stateless systemd would always provide tmpfiles for that.

As I understand this, systemd expects daemons to deal with no settings in /etc and /var. Tempfiles.d is the proposed clutch till that is actually the case.

...

...
...
- move /var/lib/pacman/local/ to /usr - move the default pacman.conf and mirrorlist to /usr/share - provide tmpfiles.d to copy those files to /etc

...
What about pacman keyring? Also note that your custom keys should be packaged as well and resigned on-boot.

I just copy my keyring into /usr/lib/factory/etc and restore them from there as needed. The private keys should stay on the server creating the image, but currently I just put those into the package as well... I need to change that ASAP. In my defense: There are no users on any of the machines running those images that I do not trust.

...

I wasn't aware of that. I only refer to what the OP requested and that didn't sound complicated at all. Now it does.

I do not consider this a problem. When you use somebodies images you need to trust that person. I do not consider trusting the keys that person provides to be a problem.

...

...
...
If I'm not mistaken, /usr/share and tmpfiles.d are really trivial and wouldn't affect users in any way. That'd be a few additional files somewhere in the filesystem without any effect on existing machines. Or I'm wrong?

...
This is madness. I remember sometime ago there was a witchhunt against daemons that write to /etc (cups is the worst offender). So why is it OK for systemd to do so? I personally don't want systemd to come anywhere near my /etc. Please package the tmpfiles.d/sysusers stuff with virtkick or whatever, but not with pacman.

Any privileged process can mess with /etc at any time. With factory reset at least you get a pristine copy to compare the files in /etc against. Arch did embrace systemd, it should make it easy to use all its features. I am not proposing to enable them by default. Best Regards, Tobias

Leonid Isaev

5:41 p.m.

Hi, On Mon, Sep 15, 2014 at 07:32:51AM +0200, Tobias Hunger wrote:

...

As I understand this, systemd expects daemons to deal with no settings in /etc and /var.

/var stores files, not settings. Most daemons will run OK with empty /var. But what do you do with the files? Some kind of rsync gluework?

...

I do not consider this a problem. When you use somebodies images you need to trust that person. I do not consider trusting the keys that person provides to be a problem.

But it is a problem which has already been discussed. If you have N images, and 1 has its key stolen, all N are in danger. So, it's not about (not) trusting the developer.

...

...
...
This is madness. I remember sometime ago there was a witchhunt against daemons that write to /etc (cups is the worst offender). So why is it OK for systemd to do so? I personally don't want systemd to come anywhere near my /etc. Please package the tmpfiles.d/sysusers stuff with virtkick or whatever, but not with pacman.

Any privileged process can mess with /etc at any time. With factory reset at least you get a pristine copy to compare the files in /etc against.

Sure, and then we call it malicious... What do you call pristine? The files shipped on a livecd? Or an empty default configs shipped with daemons? So far, there are only things like groups/users, but those are trivialities.

...

Arch did embrace systemd, it should make it easy to use all its features. I am not proposing to enable them by default.

We already have enabled by default ldconfig.service enabled, systemd-update-done.service, etc, which messed a number of my containers. Cheers, -- Leonid Isaev GPG fingerprints: DA92 034D B4A8 EC51 7EA6 20DF 9291 EE8A 043C B8C4 C0DF 20D0 C075 C3F1 E1BE 775A A7AE F6CB 164B 5A6D

Tobias Hunger

7:37 p.m.

Hi Leonid, On Mon, Sep 15, 2014 at 7:41 PM, Leonid Isaev wrote:

...

On Mon, Sep 15, 2014 at 07:32:51AM +0200, Tobias Hunger wrote:

...
As I understand this, systemd expects daemons to deal with no settings in /etc and /var.

/var stores files, not settings. Most daemons will run OK with empty /var. But what do you do with the files? Some kind of rsync gluework?

I do not clean up /var in my setup and just keep it.

...

...
I do not consider this a problem. When you use somebodies images you need to trust that person. I do not consider trusting the keys that person provides to be a problem.

But it is a problem which has already been discussed. If you have N images, and 1 has its key stolen, all N are in danger. So, it's not about (not) trusting the developer.

Well, I do not put the secret keyring into the images, so everything should be fine. Pacman can still validate images, so everything is well. Local installs are not possible anyway.

...

...
Any privileged process can mess with /etc at any time. With factory reset at least you get a pristine copy to compare the files in /etc against.

Sure, and then we call it malicious... What do you call pristine? The files shipped on a livecd? Or an empty default configs shipped with daemons? So far, there are only things like groups/users, but those are trivialities.

In the general case 'pristine' is probably the settings shipped by the various arch linux packages. In my case it obviously is the config for the usr-snapshot (== config from the archlinux packages + all my local changes).

...

We already have enabled by default ldconfig.service enabled, systemd-update-done.service, etc, which messed a number of my containers.

I guess that is the price we all have to pay occasionally for running fresh software. Best Regards, Tobias

Leonid Isaev

8:42 p.m.

Hi Tobias, Thanks for the explanation. A few questions though -- sorry. On Mon, Sep 15, 2014 at 09:37:40PM +0200, Tobias Hunger wrote:

...

Well, I do not put the secret keyring into the images, so everything should be fine.

So, you never run pacman from within an image, or have sig. validation disabled in pacman.conf?

...

Pacman can still validate images, so everything is well.

Do you mean packages in an image?

...

Local installs are not possible anyway.

What's a local install? I mean, if you treat images atomically, why do you need pacman (and associated DB) at all? You should have it only on the buildhost that generates the images (I couldn't find details in your previous emails in this thread).

...

In the general case 'pristine' is probably the settings shipped by the various arch linux packages.

But those do not usually provide sane defaults, e.g. smartd.conf, dnsmasq.conf, syslog-ng.conf, wpa_supplicant.conf, and must be edited anyway. Cheers, -- Leonid Isaev GPG fingerprints: DA92 034D B4A8 EC51 7EA6 20DF 9291 EE8A 043C B8C4 C0DF 20D0 C075 C3F1 E1BE 775A A7AE F6CB 164B 5A6D

Jakub Klinkovský

8:47 p.m.

On 15.09.14 at 16:42, Leonid Isaev wrote:

...

...
In the general case 'pristine' is probably the settings shipped by the various arch linux packages.

But those do not usually provide sane defaults, e.g. smartd.conf, dnsmasq.conf, syslog-ng.conf, wpa_supplicant.conf, and must be edited anyway.

There is a difference between "example documentation config" and "default config", see one of my previous posts: https://mailman.archlinux.org/pipermail/arch-general/2014-May/036386.html

Tobias Hunger

9:19 p.m.

Hi Leonid, On Mon, Sep 15, 2014 at 10:42 PM, Leonid Isaev wrote:

...

On Mon, Sep 15, 2014 at 09:37:40PM +0200, Tobias Hunger wrote:

...
Well, I do not put the secret keyring into the images, so everything should be fine.

So, you never run pacman from within an image, or have sig. validation disabled in pacman.conf?

I never run pacman -S ever. /usr is read-only anyway, so it would fail without remounting it first:-)

...

...
Pacman can still validate images, so everything is well.

...

Do you mean packages in an image?

Yes. pacman -Qo, -Ql and co. are immensely useful.

...

...
Local installs are not possible anyway.

What's a local install?

Sorry, I meant "pacman -S whatever".

...

I mean, if you treat images atomically, why do you need pacman (and associated DB) at all? You should have it only on the buildhost that generates the images (I couldn't find details in your previous emails in this thread).

Yes, I *could* strip the package DB. I could also strip lots of other things that make no sense, but then I am not pressed for disk space. So I prefer keeping the convenience of keeping pacman around. It is so nice to be able to do a quick check which version of the packages are installed, which package a file belongs to, etc.

...

But those do not usually provide sane defaults, e.g. smartd.conf, dnsmasq.conf, syslog-ng.conf, wpa_supplicant.conf, and must be edited anyway.

True. I just copy /etc over to /usr/lib/factory/etc on the buildhost and then make sure the /etc on the host gets wiped during early boot and replaced with the contents of /usr/lib/factory/etc. Yes, I have a pretty special use-case. It works already, so arch is flexible enough to accommodate even wierdos like me. It would still be nice to get some of the hard things I had to configure around into arch to make things easier for other wierdos;-) Best Regards, Tobias

Tobias Hunger

14 Sep 14 Sep

10:30 p.m.

Hi Nowaker, I am the one with the images, not Yamakaky:-) On Sun, Sep 14, 2014 at 11:15 PM, Nowaker wrote:

...

What are these significant changes more than just the pacman database that would make users go through trouble? In #41863 I see three parts:

- move /var/lib/pacman/local/ to /usr - move the default pacman.conf and mirrorlist to /usr/share - provide tmpfiles.d to copy those files to /etc

If I'm not mistaken, /usr/share and tmpfiles.d are really trivial and wouldn't affect users in any way. That'd be a few additional files somewhere in the filesystem without any effect on existing machines. Or I'm wrong?

Nope, these two are really minor tasks. I do not think it makes sense to split these up.

...

...
I then store the store the physical machine images in ostree (I am currently changing that to the subvolume approach systemd recently suggested).

Yamakaky, can you please provide some more info on this?

And generally, could you show me your workflow - the shell commands in particular? That would explain things better than thousands of words. :-)

I have everything in a git repo, and that repo can basically recreate (almost) all my machines from scratch -- minus the home directories and a couple of server files I do backups off. I think you will understand that I do not want that repository public:-) If there is interest I can put "splitting the actual scripts from the system configuration" onto my todo list. Basically (almost) all my machines are created from scratch each night. The simplified version is this: For each machine: 1) Create a temp directory T 2) mount a new usr:something:new-snapshot btrfs subvolume on $T/usr 3) Run pacstrap to install all packages defined for that system into $T, making sure the package DB ends in $T/usr/lib/pacman/db 4) Apply any configuration changes (create users, whatever) 5) Move $T/etc to $T/usr/share/factory/etc 6) Copy kernel and initrd to $T/usr/somewhere Now the new arch snapshot is ready to be used:-). I only need to copy that on the target machine it is intended for (using btrfs send/receive, but that is still untested), copy the kernel/initrd from that snapshot into /boot and create a new boot loader entry for the root:my-arch:foo root subvolume I already have on that machine with the new usr:something:new-snapshot subvol as its /usr. Reboot. As part of every reboot something like a "factory reset" happens (I wipe /etc and copy the stuff in the factory directory during early boot), so that /etc is always in the state it was intended to be. To get back to an earlier usr-snapshot I just reboot again and select one of the older bootloader entries. Does that help? Actually the process currently still involves ostree to store/distribute and install directory images. I did describe the process I am currently moving to, even though it is not yet fully tested. It is a lot simpler to describe. PS: Yeap, my base systems are pretty static: I do want my basic setup to always work. The mutable parts of my system are the chroots I develop in. I can create those with the exact same system as my base systems, but I can also just run pacman in there to add more stuff. Best Regards, Tobias

Nowaker

11:01 p.m.

Thanks Tobias, I think I understand it. I've got a few questions: 1. Where is your data stored? /home? Or is it stored remotely? 2. How about downtimes? Do you do something about it, or just don't need HA? If you do, do you keep a different VM running before your new VM starts? (Then I guess you need remote data, so as to switch seamlessly) -- Kind regards, Damian Nowak StratusHost www.AtlasHost.eu

Tobias Hunger

15 Sep 15 Sep

6:19 a.m.

On Sep 15, 2014 1:02 AM, "Nowaker" wrote:

...

1. Where is your data stored? /home? Or is it stored remotely?

It depends. My server has its RAID arrays mounted for its data. My desktop on the other hand basically just has /home for storage. That is exactly what I need to back up.

...

2. How about downtimes? Do you do something about it, or just don't need HA? If you do, do you keep a different VM running before your new VM starts? (Then I guess you need remote data, so as to switch seamlessly)

My machines are not important enough for HA. Basically I just reboot when there is any interesting upgrade in the new image. Examining HA options would be interesting. CoreOS shows that it is possible. Best Regards, Tobias

3522

Age (days ago)

3527

Last active (days ago)

List overview

Download

32 comments

10 participants

participants (10)

Bigby James
Conrad Nelson
Jakub Klinkovský
Joakim Hernberg
Joel Teichroeb
Leonid Isaev
Neven Sajko
Nowaker
Tobias Hunger
Yamakaky