[arch-general] Arch pkg user and group IDs?
Hi, what are Arch's conventions for the numeric ranges of system (package) user and grop IDs? And can the range be limited? I am working up to switching a few dozen Debian clients to Arch. They have user homes on NFS, and authenticate through NIS. So far, nis distributes user and group IDs > 500, making anything lower a local / system ID. After installing Arch, I see % awk -F: '{ if ( $3 > 500 ) print }' < /etc/passwd systemd-journal-remote:x:999:999:systemd Journal Remote:/:/sbin/nologin systemd-journal-upload:x:998:998:systemd Journal Upload:/:/sbin/nologin systemd-coredump:x:997:997:systemd Core Dumper:/:/sbin/nologin % awk -F: '{ if ( $3 > 500 ) print }' < /etc/group systemd-journal-remote:x:999: systemd-journal-upload:x:998: systemd-coredump:x:997: lightdm:x:620: % which collide with NIS users.[1] My question: Do Arch packages come with hardcoded numerical user/group IDs? If not, is there a way to limit the numeric range of system IDs? I guess I would still have to reinstall the distribution. The alternative - a flag day to change NIS user and group IDs as well as on 4 TB file storage and local storage on 50 client machines - wouldn't be pretty. And I don't really have the time, either. Cheerio, hauke [1] lightdm appears to be a pathological case: It hardcodes both name and ID of its user and group, and installation plows on witjout flagging an error even after failing to register the user ID. -- The ASCII Ribbon Campaign Hauke Fath () No HTML/RTF in email Institut für Nachrichtentechnik /\ No Word docs in email TU Darmstadt Respect for open standards Ruf +49-6151-16-21344
First of all, any UID less than 1000 is generally considered a system user/group, I would suspect this tradition or a variation thereof comes from elder days. Yes, arch system users are numbered automatically, but the groups and users you listed are a bit of special snowflakes for systemd. I'm not sure whether remapping them would be possible. What I would recommend to you is to use another range further beyond 1000 for your network users and groups because with regard to what we established in the first paragraph, those don't seem to qualify as system users IMO. And, just to mention it for completness, there is /etc/login.defs where you can theoretically adjust those numbers. cheers! mar77i
On Sun, 27 Nov 2016 14:52:25 +0100, Martin Kühne via arch-general wrote:
What I would recommend to you is to use another range further beyond 1000 for your network users and groups because with regard to what we established in the first paragraph, those don't seem to qualify as system users IMO.
Well, since we are talking about an existing system with a few dozen users and more machines, plus a few servers, moving existing users and groups is a last resort, and one that will create unhappiness and bad fallout. I would rather not go there.
And, just to mention it for completness, there is /etc/login.defs where you can theoretically adjust those numbers.
This <snip> # Min/max values for automatic uid selection in useradd # UID_MIN 1000 UID_MAX 60000 # System accounts SYS_UID_MIN 500 SYS_UID_MAX 999 </snip> might actually help, depending on how well it is enforced. Thanks! Cheerio, hauke -- The ASCII Ribbon Campaign Hauke Fath () No HTML/RTF in email Institut für Nachrichtentechnik /\ No Word docs in email TU Darmstadt Respect for open standards Ruf +49-6151-16-21344
On 11/27/2016 12:27 PM, Hauke Fath wrote:
And, just to mention it for completness, there is /etc/login.defs where you can theoretically adjust those numbers.
This
<snip> # Min/max values for automatic uid selection in useradd # UID_MIN 1000 UID_MAX 60000 # System accounts SYS_UID_MIN 500 SYS_UID_MAX 999 </snip>
might actually help, depending on how well it is enforced. Thanks!
"enforced"? It is the configuration file for useradd. Anything not explicitly hardcoded in the UID/GID database (or hardcoded but not in the database) will respect the useradd configuration (when you reinstall Arch and all those users are created from scratch again). Although really, whatever distribution was running on your NFS server shouldn't be configuring for users with UIDs below 1000 -- a network is exactly the wrong place to be allowing UIDs that can clash with other distros' UID reservations. It might not be a bad idea to report that as a bug. -- Eli Schwartz
On Sun, 27 Nov 2016 15:49:33 -0500, Eli Schwartz via arch-general wrote:
[login.defs] might actually help, depending on how well it is enforced. Thanks!
"enforced"? It is the configuration file for useradd. Anything not explicitly hardcoded in the UID/GID database (or hardcoded but not in the database)
... like lightdm?
will respect the useradd configuration (when you reinstall Arch and all those users are created from scratch again).
Although really, whatever distribution was running on your NFS server shouldn't be configuring for users with UIDs below 1000 -- a network is exactly the wrong place to be allowing UIDs that can clash with other distros' UID reservations.
Well, there's http://unix.stackexchange.com/questions/101313/what-are-the-dangers-of-creat... which discusses OS distributions' conventions of where to start non-system uids/gids. The installation in question is about 13 years old, and it has been merged a while back with a database that was even older. So the uid 1000 border that Arch uses (and also Debian, according to the link above, although at least Debian 7 is counting upward, and the highest system uid on our systems is 118), is by no means universal. I guess the average distro maintainer doesn't work in a larger, historically grown network...
It might not be a bad idea to report that as a bug.
It was an administrative decision, back more than ten years ago, when 500 IDs appeared to be enough for everyone. ;) [ systemd-journal-remote:x:999: systemd-journal-upload:x:998: systemd-coredump:x:997: ]
AFAIK those systemd users/groups are generated by sysusers.d
Aahh! Thanks for the missing puzzle piece. I guess I can find my way from there. Cheerio, Hauke -- The ASCII Ribbon Campaign Hauke Fath () No HTML/RTF in email Institut für Nachrichtentechnik /\ No Word docs in email TU Darmstadt Respect for open standards Ruf +49-6151-16-21344
On 11/27/2016 08:52 AM, Martin Kühne via arch-general wrote:
Yes, arch system users are numbered automatically, but the groups and users you listed are a bit of special snowflakes for systemd. I'm not sure whether remapping them would be possible.
AFAIK those systemd users/groups are generated by sysusers.d, which doesn't declare specific UIDs. So I assume that is read from /etc/login.defs, besides which I don't see how that would otherwise work on distros that use lower max values. -- Eli Schwartz
On 11/27/2016 04:04 PM, Eli Schwartz wrote:
On 11/27/2016 08:52 AM, Martin Kühne via arch-general wrote:
Yes, arch system users are numbered automatically, but the groups and users you listed are a bit of special snowflakes for systemd. I'm not sure whether remapping them would be possible.
AFAIK those systemd users/groups are generated by sysusers.d, which doesn't declare specific UIDs. So I assume that is read from /etc/login.defs, besides which I don't see how that would otherwise work on distros that use lower max values.
Actually, the systemd "eevulz maintainers" meme strikes again. Or at least, a rather bizarre design choice: https://github.com/systemd/systemd/issues/3855 On 8/2/2016 10:44 AM, Lennart Poettering commented on Github:
I am strongly against this. Sorry. These defines should be chosen by the distro vendor, not by users. And it's not realistic to change these definitions during runtime anyway (as you cannot reasonably move the IDs allocated during installation).
Quite frankly, these settings should never have been in /etc as this suggests all would be fine with randomly changing the values for this, but this cannot work. Sorry.
Completely missing the forest for the trees... So clearly, the systemd way is to install an Arch system/chroot, change the defaults, and re-roll your systemd packages for use in a brand-new installation, then repeat *in perpetuity* , if you intend to deviate *at install time* from the user-configurable choices of the distro maintainers. So much for end-user configurability of systems, and here I thought it would be as simple as initially populating that file on a blank partition. I don't know if that is evil, I simply do not know what the logic is at all. This breaks systemd-sysuser's promised holy grail of declarative system users -- the "imperative" is still there, but moved, and hidden inside the binary that translates those files. The user/package wants certain users, the system announces what a user should look like, and then systemd stomps all over that by memorizing how it created users the first time (regardless of which hardware that was), and then just doing that again. -- Eli Schwartz
On 11/27/2016 07:58 AM, Hauke Fath wrote:
Hi,
what are Arch's conventions for the numeric ranges of system (package) user and grop IDs? And can the range be limited?
I am working up to switching a few dozen Debian clients to Arch. They have user homes on NFS, and authenticate through NIS. So far, nis distributes user and group IDs > 500, making anything lower a local / system ID. After installing Arch, I see
By convention, Arch expects user UIDs to be greater than 1000. There is a list of explicitly reserved system UIDs by repo packages here: https://wiki.archlinux.org/index.php/DeveloperWiki:UID_/_GID_Database -- Eli Schwartz
On Sun, 27 Nov 2016 08:55:48 -0500, Eli Schwartz via arch-general wrote:
By convention, Arch expects user UIDs to be greater than 1000.
I understand that.
There is a list of explicitly reserved system UIDs by repo packages here: https://wiki.archlinux.org/index.php/DeveloperWiki:UID_/_GID_Database
Which, apart from qmail which is easily avoided, lists only IDs below 500. But it looks like /etc/login.defs additionally reserves a range of system IDs. Cheerio, hauke -- The ASCII Ribbon Campaign Hauke Fath () No HTML/RTF in email Institut für Nachrichtentechnik /\ No Word docs in email TU Darmstadt Respect for open standards Ruf +49-6151-16-21344
On Sun, Nov 27, 2016 at 06:31:01PM +0100, Hauke Fath wrote:
On Sun, 27 Nov 2016 08:55:48 -0500, Eli Schwartz via arch-general wrote:
By convention, Arch expects user UIDs to be greater than 1000.
I understand that.
There is no such think as "arch expects". It is a default setting in /etc/login.defs that is being used by systemd because it is likely that a devs don't change login.defs before building systemd...
There is a list of explicitly reserved system UIDs by repo packages here: https://wiki.archlinux.org/index.php/DeveloperWiki:UID_/_GID_Database
Which, apart from qmail which is easily avoided, lists only IDs below 500. But it looks like /etc/login.defs additionally reserves a range of system IDs.
I strongly suspect that the above wikipage is deprecated. Most likely all arch packages will move to sysuser.d sooner or later. BTW, have you read the manpage of sysuser.d? You can override those "default" IDs by specifying your own, just need to change a few files (on our NIS master there are only snippets supplied with systemd)... But out of curiosity, why is it difficult to change user IDs on all files? I assume that you control the storage? Isn't it just a chown -R away? For example, for our NIS passwd/shadow map we use 6-digit IDs... Cheers, -- Leonid Isaev GPG fingerprints: DA92 034D B4A8 EC51 7EA6 20DF 9291 EE8A 043C B8C4 C0DF 20D0 C075 C3F1 E1BE 775A A7AE F6CB 164B 5A6D
On 11/27/2016 09:16 PM, Leonid Isaev wrote:
There is no such think as "arch expects". It is a default setting in /etc/login.defs that is being used by systemd because it is likely that a devs don't change login.defs before building systemd...
Well, packages can have files that need to have a specific system user ownership. That is why the UID/GID database exists, right? Because the UID baked into the *.pkg.tar.xz has to match /etc/passwd, and systemd-sysusers can't inherently do anything that repetitive useradd + getent scripting wasn't always capable of.
I strongly suspect that the above wikipage is deprecated. Most likely all arch packages will move to sysuser.d sooner or later.
I don't see why, again, systemd-sysusers was designed as a declarative alternative to `useradd -r` and the Wikipage was never deprecated by useradd...
BTW, have you read the manpage of sysuser.d? You can override those "default" IDs by specifying your own, just need to change a few files (on our NIS master there are only snippets supplied with systemd)...
Do you mean, by manually specifying a hardcoded UID for each user??? -- Eli Schwartz
On Sun, Nov 27, 2016 at 09:50:22PM -0500, Eli Schwartz via arch-general wrote:
On 11/27/2016 09:16 PM, Leonid Isaev wrote:
There is no such think as "arch expects". It is a default setting in /etc/login.defs that is being used by systemd because it is likely that a devs don't change login.defs before building systemd...
Well, packages can have files that need to have a specific system user ownership. That is why the UID/GID database exists, right? Because the UID baked into the *.pkg.tar.xz has to match /etc/passwd, and systemd-sysusers can't inherently do anything that repetitive useradd + getent scripting wasn't always capable of.
For example, dnsmasq ships /usr/lib/sysusers.d/dnsmasq.conf which contains 'u dnsmasq - "dnsmasq daemon" /' and on my system the user dnsmasq has (randomly-generated) ID = 997. Such packages won't have any files owned by a non-root user because they don't know the UID.
I strongly suspect that the above wikipage is deprecated. Most likely all arch packages will move to sysuser.d sooner or later.
I don't see why, again, systemd-sysusers was designed as a declarative alternative to `useradd -r` and the Wikipage was never deprecated by useradd...
Maybe, but currently most config snippets in /usr/lib/sysusers.d/ do not specify UID (except qemu.conf shipped with qemu). So I assume those get assigned randomly.
BTW, have you read the manpage of sysuser.d? You can override those "default" IDs by specifying your own, just need to change a few files (on our NIS master there are only snippets supplied with systemd)...
Do you mean, by manually specifying a hardcoded UID for each user???
Yes, why not? You can override files in /usr/lib/sysusers.d with files in /etc/sysusers.d having identical names, no? For example, on my workstation, there are only 23 lines in total where UID need to be changed below 500. Of course, this needs to be done for all Arch machines. That is why I think that changing ownership in NFS share is a better idea... Cheers, -- Leonid Isaev GPG fingerprints: DA92 034D B4A8 EC51 7EA6 20DF 9291 EE8A 043C B8C4 C0DF 20D0 C075 C3F1 E1BE 775A A7AE F6CB 164B 5A6D
On 11/27/2016 10:03 PM, Leonid Isaev wrote:
Well, packages can have files that need to have a specific system user ownership. That is why the UID/GID database exists, right? Because the UID baked into the *.pkg.tar.xz has to match /etc/passwd, and systemd-sysusers can't inherently do anything that repetitive useradd + getent scripting wasn't always capable of.
For example, dnsmasq ships /usr/lib/sysusers.d/dnsmasq.conf which contains 'u dnsmasq - "dnsmasq daemon" /' and on my system the user dnsmasq has (randomly-generated) ID = 997. Such packages won't have any files owned by a non-root user because they don't know the UID.
Let's rephrase that. Some packages contain software which requires runtime files owned by a non-root user associated with the software. The fact that they don't know the UID during makepkg is precisely the reason why the UID/GID database exists... dnsmasq is an irrelevant example of software which has absolutely nothing to do with the topic at hand.
Maybe, but currently most config snippets in /usr/lib/sysusers.d/ do not specify UID (except qemu.conf shipped with qemu). So I assume those get assigned randomly.
Again, that is the whole point. useradd assigns them randomly, and sysusers assigns them randomly. But there is a range of UIDs reserved for stuff that needs a non-random UID (because the UID needs to be known during makepkg for packaging). So yes, Arch expects hardcoded UIDs to be available below 500, which means system user UIDs can be and should be allocated from 500-999 (as opposed to, say, preemptively clobbering any potentially reserved UIDs below that).
Yes, why not? You can override files in /usr/lib/sysusers.d with files in /etc/sysusers.d having identical names, no? For example, on my workstation, there are only 23 lines in total where UID need to be changed below 500.
Because that is the classic definition of working around buggy software? useradd/sysusers was carefully designed to be able to dynamically allocate UIDs and now you have to copy and override every single sysusers definition because systemd won't let you configure the allocation range...
Of course, this needs to be done for all Arch machines.
Plus that.
That is why I think that changing ownership in NFS share is a better idea...
Well, depending on the number of files... Anyway, modifying a single file in a very predictable manner on multiple *new* machines is an even better idea. Plus, Hauke Fath indicated that there were also plenty of files scattered across 50 other existing machines. -- Eli Schwartz
On Sun, Nov 27, 2016 at 10:32:38PM -0500, Eli Schwartz via arch-general wrote:
On 11/27/2016 10:03 PM, Leonid Isaev wrote:
Well, packages can have files that need to have a specific system user ownership. That is why the UID/GID database exists, right? Because the UID baked into the *.pkg.tar.xz has to match /etc/passwd, and systemd-sysusers can't inherently do anything that repetitive useradd + getent scripting wasn't always capable of.
For example, dnsmasq ships /usr/lib/sysusers.d/dnsmasq.conf which contains 'u dnsmasq - "dnsmasq daemon" /' and on my system the user dnsmasq has (randomly-generated) ID = 997. Such packages won't have any files owned by a non-root user because they don't know the UID.
Let's rephrase that. Some packages contain software which requires runtime files owned by a non-root user associated with the software.
The fact that they don't know the UID during makepkg is precisely the reason why the UID/GID database exists...
Yes, if such packages exist, then of course, they would require hardcoded UID.
Yes, why not? You can override files in /usr/lib/sysusers.d with files in /etc/sysusers.d having identical names, no? For example, on my workstation, there are only 23 lines in total where UID need to be changed below 500.
Because that is the classic definition of working around buggy software? useradd/sysusers was carefully designed to be able to dynamically allocate UIDs and now you have to copy and override every single sysusers definition because systemd won't let you configure the allocation range...
I kinda agree with Lennart Poettering here as to why systemd shouldn't read login.defs at runtime. Still, dealing with systemd ppl sucks... I mean once you do change ranges in login.defs, you'll have to fix user memberships as well, yes?
Of course, this needs to be done for all Arch machines.
Plus that.
Not only that, but also updates must be carefully monitored...
That is why I think that changing ownership in NFS share is a better idea...
Well, depending on the number of files... Anyway, modifying a single file in a very predictable manner on multiple *new* machines is an even better idea. Plus, Hauke Fath indicated that there were also plenty of files scattered across 50 other existing machines.
Hmm, I thought all /home's were on an NFS share... But I still don't understand why is it such a difficult task? Sure, migrating 10 year old installations this way is not pleasant, but at least this will be future-proof. Because how do you avoid clashes with hard-coded UID in packages now? Cheers, -- Leonid Isaev GPG fingerprints: DA92 034D B4A8 EC51 7EA6 20DF 9291 EE8A 043C B8C4 C0DF 20D0 C075 C3F1 E1BE 775A A7AE F6CB 164B 5A6D
On Sun, 27 Nov 2016 19:16:56 -0700, Leonid Isaev wrote:
But out of curiosity, why is it difficult to change user IDs on all files? I assume that you control the storage? Isn't it just a chown -R away? For example, for our NIS passwd/shadow map we use 6-digit IDs...
Because... users have files - on their NFS home - on public NFS shares - on a partition of the local harddrive (and not necessarily limited to one machine) - on their home on the web server - on their home on the mailserver - on their home on the computing cluster all of which makes a change of user and group id slightly more involved than a 'chmod -R'. Nothing that couldn't be done, mind you, given enough round tuits - both for me and my users. As I said, it would have to be either a flag day (deploy a script with old-new mapping to all machines involved, lock out users, shut down services, run script), or piecemeal change (negociate time slot with user, log them out, annoy other users because you have to temporarily disable imap and smtp services, run said script). Both would need to be planned, communicated and negociated, and so take more time than I have. Cheerio, Hauke -- The ASCII Ribbon Campaign Hauke Fath () No HTML/RTF in email Institut für Nachrichtentechnik /\ No Word docs in email TU Darmstadt Respect for open standards Ruf +49-6151-16-21344
Hmm, you could do the move per-server, though, at least for the network services that publicly can report a different UID/GID pair than is advertised on the file system, which is at least true for NFS. Did you look into that, already? cheers! mar77i
On Mon, 28 Nov 2016 12:28:23 +0100, Martin Kühne via arch-general wrote:
Hmm, you could do the move per-server, though, at least for the network services that publicly can report a different UID/GID pair than is advertised on the file system, which is at least true for NFS.
NFSv4, that is. We use v3, which by design cannot remap IDs - the server takes the client's information as gospel.
Did you look into that, already?
Yes... the NFS server is actually the least of my worries. Cheerio, hauke -- The ASCII Ribbon Campaign Hauke Fath () No HTML/RTF in email Institut für Nachrichtentechnik /\ No Word docs in email TU Darmstadt Respect for open standards Ruf +49-6151-16-21344
On Mon, Nov 28, 2016 at 11:04:53AM +0100, Hauke Fath wrote:
On Sun, 27 Nov 2016 19:16:56 -0700, Leonid Isaev wrote:
But out of curiosity, why is it difficult to change user IDs on all files? I assume that you control the storage? Isn't it just a chown -R away? For example, for our NIS passwd/shadow map we use 6-digit IDs...
Because... users have files
- on their NFS home - on public NFS shares - on a partition of the local harddrive (and not necessarily limited to one machine) - on their home on the web server - on their home on the mailserver - on their home on the computing cluster
all of which makes a change of user and group id slightly more involved than a 'chmod -R'. Nothing that couldn't be done, mind you, given enough round tuits - both for me and my users.
As I said, it would have to be either a flag day (deploy a script with old-new mapping to all machines involved, lock out users, shut down services, run script), or piecemeal change (negociate time slot with user, log them out, annoy other users because you have to temporarily disable imap and smtp services, run said script). Both would need to be planned, communicated and negociated, and so take more time than I have.
OK, if this is not an option, then I you have few options, but all of the suck: 1. Just go over your /etc/passwd, /etc/group etc. and manually assign UID/GID to systemd* users. The hope is that packages won't install new users. Also, put /usr/lib/sysusers.d in a version control, so you can track changes. And of course, change your login.defs appropriately. Systemd won't read it, but other programs might. 2. If this is too hackish for you, then override files in /usr/lib/sysusers.d/ by copying them to /etc/sysusers.d and putting fixed UIDs there. Of course, /etc/passwd et all need to be fixed manually, as above. 3. Rebuild systemd with a proper login.defs... Keep in mind though, that new stable releases of systemd are almost always broken in one way or another... 4. Reopen the bugreport mentioned in this thread and try to bring LP back to earth. Good luck with that though, in my experience it is impossible :) If I were you, I'd choose (1) but create every new user in a high-numbered IDs, so hopefully in some time, you can drop the hack. HTH, -- Leonid Isaev GPG fingerprints: DA92 034D B4A8 EC51 7EA6 20DF 9291 EE8A 043C B8C4 C0DF 20D0 C075 C3F1 E1BE 775A A7AE F6CB 164B 5A6D
participants (4)
-
Eli Schwartz
-
Hauke Fath
-
Leonid Isaev
-
Martin Kühne