[arch-dev-public] Kernel - vanilla vs patched?
Well, this kicked itself off in IRC this morning (timezone GMT+1), and as it seems most people didn't see it and/or didn't participate, we should probably do it here. Here's the question, as I see it - what do we patch kernel26 for? Bugfixes? Stability? Enhancements? New features? Other? Specific case under discussion: http://bugs.archlinux.org/task/8500 - a request, from a single user so far, for a new feature in the kernel. We also have, or had, a request for fbsplash (closed, of course) and there are probably others - Romashka? My understanding of The Arch Way, as it applies here, is that we patch for bugfixes e.g. ipw2x00, and for enhancements to existing functionality e.g. alsa. In the referenced bug report, there are differing dev views expressed - as I'm posting this, I'll include my view, which is that we should not add new features to kernel26. I'll also include a possible compromise: /me grabs big can of worms, opens it, pours it out onto DevLand :) Two kernels - a strictly as-vanilla-as-possible kernel26 in Core, and a more let's-try-new-features-here kernel26extra in Extra. It could be in Community either, but I would see it as an official Arch package, supported by the dev team. I would be happy to maintain it if necessary, along with the usual array of external module packages - in cooperation, of course, with the current kernel maintainers. Right now, anyone who has read the IRC log knows how Tobias P and Aaron feel about it, but I expect others have views on this topic too, so let's hear them. Regards Tom.
2007/11/9, Tom K <tom@archlinux.org>:
Well, this kicked itself off in IRC this morning (timezone GMT+1), and as it seems most people didn't see it and/or didn't participate, we should probably do it here.
Here's the question, as I see it - what do we patch kernel26 for? Bugfixes? Stability? Enhancements? New features? Other?
Specific case under discussion: http://bugs.archlinux.org/task/8500 - a request, from a single user so far, for a new feature in the kernel.
We also have, or had, a request for fbsplash (closed, of course) and there are probably others - Romashka?
My understanding of The Arch Way, as it applies here, is that we patch for bugfixes e.g. ipw2x00, and for enhancements to existing functionality e.g. alsa. In the referenced bug report, there are differing dev views expressed - as I'm posting this, I'll include my view, which is that we should not add new features to kernel26.
I'll also include a possible compromise:
/me grabs big can of worms, opens it, pours it out onto DevLand :)
Two kernels - a strictly as-vanilla-as-possible kernel26 in Core, and a more let's-try-new-features-here kernel26extra in Extra. It could be in Community either, but I would see it as an official Arch package, supported by the dev team. I would be happy to maintain it if necessary, along with the usual array of external module packages - in cooperation, of course, with the current kernel maintainers.
Right now, anyone who has read the IRC log knows how Tobias P and Aaron feel about it, but I expect others have views on this topic too, so let's hear them.
Two kernels is a bad idea. Even vanilla kernel does need patches, so "as vanilla as possible" will contain same bugfixes and alsa updates as another kernel. And it does need features like squashfs/unionfs/aufs because we need them for our LiveCDs. I'm fine with that additional undervolting feature because a) it touches only small pice of code and b) our kernel maintainer is fine with it. Creating extra kernel just because of this is a bad idea. I don't want to see another flame here. However, if someone wants much more patched kernel, e.g. with Reiser4 support - this deserves for a separate kernel package. I don't know many such features though. Suspend2/TuxOnIce package died because it was hard to maintain and often broke. ck died. beyound died, vesa-tng which was one of cool features here transformed to uvesafb and will be in kernel .24. So I really see no point why very small and unintrusive patches could be maintained on our default kernel. Sure, everyone has his own opinion about what is useful. I'm fine with any small features that are not hard to maintain, doesn't make difference for those who don't need them and are unintrusive to kernel code. -- Roman Kyrylych (Роман Кирилич)
2007/11/9, Roman Kyrylych <roman.kyrylych@gmail.com>:
2007/11/9, Tom K <tom@archlinux.org>:
Well, this kicked itself off in IRC this morning (timezone GMT+1), and as it seems most people didn't see it and/or didn't participate, we should probably do it here.
Here's the question, as I see it - what do we patch kernel26 for? Bugfixes? Stability? Enhancements? New features? Other?
Specific case under discussion: http://bugs.archlinux.org/task/8500 - a request, from a single user so far, for a new feature in the kernel.
We also have, or had, a request for fbsplash (closed, of course) and there are probably others - Romashka?
My understanding of The Arch Way, as it applies here, is that we patch for bugfixes e.g. ipw2x00, and for enhancements to existing functionality e.g. alsa. In the referenced bug report, there are differing dev views expressed - as I'm posting this, I'll include my view, which is that we should not add new features to kernel26.
I'll also include a possible compromise:
/me grabs big can of worms, opens it, pours it out onto DevLand :)
Two kernels - a strictly as-vanilla-as-possible kernel26 in Core, and a more let's-try-new-features-here kernel26extra in Extra. It could be in Community either, but I would see it as an official Arch package, supported by the dev team. I would be happy to maintain it if necessary, along with the usual array of external module packages - in cooperation, of course, with the current kernel maintainers.
Right now, anyone who has read the IRC log knows how Tobias P and Aaron feel about it, but I expect others have views on this topic too, so let's hear them.
Two kernels is a bad idea. Even vanilla kernel does need patches, so "as vanilla as possible" will contain same bugfixes and alsa updates as another kernel. And it does need features like squashfs/unionfs/aufs because we need them for our LiveCDs. I'm fine with that additional undervolting feature because a) it touches only small pice of code and b) our kernel maintainer is fine with it.
Creating extra kernel just because of this is a bad idea. I don't want to see another flame here.
However, if someone wants much more patched kernel, e.g. with Reiser4 support - this deserves for a separate kernel package. I don't know many such features though. Suspend2/TuxOnIce package died because it was hard to maintain and often broke. ck died. beyound died, vesa-tng which was one of cool features here transformed to uvesafb and will be in kernel .24.
So I really see no point why very small and unintrusive patches could be maintained on our default kernel.
oops, typo *could not*
Sure, everyone has his own opinion about what is useful. I'm fine with any small features that are not hard to maintain, doesn't make difference for those who don't need them and are unintrusive to kernel code.
-- Roman Kyrylych (Роман Кирилич)
-- Roman Kyrylych (Роман Кирилич)
Tom K wrote:
My understanding of The Arch Way, as it applies here, is that we patch for bugfixes e.g. ipw2x00, and for enhancements to existing functionality e.g. alsa. In the referenced bug report, there are differing dev views expressed - as I'm posting this, I'll include my view, which is that we should not add new features to kernel26.
I would add to this that we endeavor to make minor, non-invasive patches that obviously (apparently) do more good than bad. For instance, we generally frown on large patches, unless they're clearly related to a single subsystem and released by the maintainers of that subsystem (like alsa) as more stable. I would really like to hear from people who have a substantive objection to particular patches in kernel26. If there are substantial reasons why the patching interferes with activity that could be performed with an unpatched kernel, I think we should at least consider dropping those patches. In general, we should not break things that would work if the kernel were vanilla, and we try hard not to do that. If we must split, I would recommend we go about this differently, which is to say that we have a kernel26vanilla or, perhaps more appropriately, kernel26linus in [extra] which is maintained by those who feel that kernel26 with patches is somehow unsuited to their needs. - P
On Nov 9, 2007 5:58 PM, Tom K <tom@archlinux.org> wrote:
/me grabs big can of worms, opens it, pours it out onto DevLand :)
Two kernels - a strictly as-vanilla-as-possible kernel26 in Core, and a more let's-try-new-features-here kernel26extra in Extra. It could be in Community either, but I would see it as an official Arch package, supported by the dev team. I would be happy to maintain it if necessary, along with the usual array of external module packages - in cooperation, of course, with the current kernel maintainers.
How would we draw the line as to what gets into the kernel26extra package? Do you plan to include small but well proven functionality patches, or will you be including experimental support for, say, perhaps new filesystems or new wireless drivers and so on? To rephrase, if kernel26extra becomes a reality, what exactly would go into it, and how would this be decided?
On 11/10/07, ganja guru <ganja.guru.x64@gmail.com> wrote:
On Nov 9, 2007 5:58 PM, Tom K <tom@archlinux.org> wrote:
/me grabs big can of worms, opens it, pours it out onto DevLand :)
Two kernels - a strictly as-vanilla-as-possible kernel26 in Core, and a more let's-try-new-features-here kernel26extra in Extra. It could be in Community either, but I would see it as an official Arch package, supported by the dev team. I would be happy to maintain it if necessary, along with the usual array of external module packages - in cooperation, of course, with the current kernel maintainers.
How would we draw the line as to what gets into the kernel26extra package? Do you plan to include small but well proven functionality patches, or will you be including experimental support for, say, perhaps new filesystems or new wireless drivers and so on? To rephrase, if kernel26extra becomes a reality, what exactly would go into it, and how would this be decided?
kernel26extra sounds awfully like what beyond was. I had phc, fbsplash, tuxonice. I don't think adding an extra kernel will work, because then we'll get the people who only want fbsplash, because tuxonice causes them other problems or they don't like it. Thats why we ended up with kernel26ck and kernel26suspend2 in the repos on top of beyond. Put lightly, dumping all extras into on kernel isnt a solution. Lets look at this differently. To put it lightly, editing the kernel26 pkgbuild to add/remove things, can be daunting for most or annoying to do every release. This is why they want us to do it. Why don't we make the kernel26 PKGBUILD a bit more adaptable/friendly? James -- iphitus // Arch Developer // iphitus.loudas.com
I was about to start this thread just this morning, but didn't have the time. Here is what I think: The Arch way is to keep things as untouched as possible and frankly, our kernel has become full of stuff that I don't see a reason for. * acpi-dsdt-initrd-v0.8.4-2.6.21.patch My opinion on this one is unclear: There have been tons of fixes to ACPI and the kernel developers claim that this patch should not be needed, as the Linux ACPI implementation is supposed to support everything out of the box. Right now, many people who needed a custom dsdt years ago, still use it despite the fact that they don't need it. However, there are _some rare cases_ where only a dsdt can fix the problems. The majority of computers however should now work without it. What I definitely don't like about the patch is that the ACPI subsystem has changed substantially since 2.6.21 - the fact that the patch still applies doesn't mean it doesn't introduce unwanted and stupid behaviour, or hidden bugs. None of us would know the problems the patch may cause, as none of us knows anything about kernel/ACPI code. * toshiba-bluetooth.patch What does this patch even do? It has been sitting there since .21. What is the reason this hasn't been merged upstream? 1) It breaks other setups, which would mean we should remove it, 2) it has been merged in a different way and still applies, in which case it will only cause bugs, but don't fix any, so we should remove it or 3) it has never been submitted, in which case it shouldn't be our responsibility to keep it along while the author isn't even capable of submitting it. In any of the above cases, we should remove the patch. * usb-storage-unusual-devs.patch I submitted this patch after I created it, so either 1) or 2) of the above apply, so we should remove it as well. * mactel-linux-2.6.23.patch What does this patch even do? There have been Macbook patches sitting around since .20 or so, and again: why aren't they merged upstream? Any of the reasons for them not being merged imply we should remove them. * http://www.archlinux.org/~tpowa/alsa-patches/alsa-20071109.patch.bz2 *http://www.archlinux.org/~tpowa/alsa-patches/alsa-include-20071109.patch.bz2 Why do we update alsa all the time, especially to non-released versions? This may have been necessary, when a large number of chipsets were broken, but I don't think that is still the case. Is there any specific reasons we cannot stick with the alsa version that Linux provides? * acpi-buggy-bios.patch Like mactel, this patch has been there since .20 or so, the ACPI implementation has had many changes, and it is more likely that it introduces bugs than that it fixes any. Again, this patch should be removed for one of the above reasons. squashfs3.2-patch.bz2 http://download.filesystems.org/unionfs/unionfs-2.1/unionfs-2.1.8_for_2.6.23... Why are none of these merged upstream? Why do we have to add features ourselves? I know we had problems with externally built unionfs, but adding it in-kernel shouldn't be the solution, especially because it results in more work maintaining. * put_filp.patch * lhash-2.6.22.patch I don't know what these are, but these seem to be new bugfixes. Still, it should be evaluated if they are still needed before they are added again in 2.6.24. * adjust_current_level_to_closest_available.patch This is a bugfix that has already been merged for 2.6.24. This is an example of how things should work. I discovered this bug on my notebook, posted to the right mailing list and got a fix, confirmed it and now it will be merged in the next version. So we only need to take the patch along until the next version, and can then drop it. hibernate-saa7134.diff How long has this been sitting here? Again, the "why hasn't it been merged" question should be asked, and the patch removed, most likely. sata_sis-2.6.23.patch ipw2x00-2.6.23.patch zd1211rw-oops-2.6.23.patch ide-helper-2.6.23.patch I guess those are new, so again, we can keep them until .24, but they should be fixed by then and our patches removed. * linux-phc-0.3.0-kernel-vanilla-2.6.23rc3.patch This is an extra feature. One that a small minority of people can use. It is said to be unintrusive, but then, why isn't it in the upstream version? If it is so unintrusive, the upstream devs shouldn't object to adding it. So either it was never submitted or it was rejected for a good reason. # genpatches from gentoo * 2405_hostap-netdev-type.patch * 2525_usb-storage-nikon-d200-quirk.patch * 2530_usb-storage-nikon-d40x-quirk.patch What do these do? Why do we need them? * pre-2.6.23.2.patch Where does this patch come from? Where is it (not in our CVS)? The 2.6.23.y git tree shows no new patches since .23.1, so I would at least like a comment about it. My opinion is, we should remove most of the above patches as 1) we don't know what bugs they may introduce, 2) they haven't been merged upstream for a reason or 3) they have been merged but in a different way, introducing two fixes for the same problem, and the one in the vanilla kernel is most likely much cleaner. It appears to me that tpowa often blindly adds patches that are submitted, with the only condition that they apply and build. I want to reduce the number of patches applied to our kernel, and I want that any patch that adds a new feature or bugfix MUST be merged in a future version. If a patch has never been submitted, it should be submitted, if it has been submitted and rejected, it should be dropped.
* put_filp.patch * lhash-2.6.22.patch
I don't know what these are, but these seem to be new bugfixes. Still, it should be evaluated if they are still needed before they are added again in 2.6.24.
These are small patches that just expose kernel functionality so the AUFS modules will work. I will continue to monitor these as AUFS progresses and remove them if/when we can. Apparently, I should also make them say more loudly in the PKGBUILD that they have to do with AUFS, since that was difficult to tell. In general, +1 to James for the suggestion to make the kernel PKGBUILD more user friendly. - P
On Nov 9, 2007 7:45 AM, James <iphitus@gmail.com> wrote:
Lets look at this differently. To put it lightly, editing the kernel26 pkgbuild to add/remove things, can be daunting for most or annoying to do every release. This is why they want us to do it. Why don't we make the kernel26 PKGBUILD a bit more adaptable/friendly?
YES YES YES! Thank you for bringing this up. For that matter, I tried and failed at this because we use too much damn magic in our PKGBUILD! I've custom compiled my own TRUE vanilla kernel for one of my boxes, and I didn't use this PKGBUILD because it is just way too hard to decipher. Can someone please clean this up? I don't care if we have a select group of devs that are our "kernel maintainers"- it shouldn't take a rocket scientist to change this PKGBUILD and know what is going on. Why is there a 'yes "" | make config' in there? What is this sed EXTRAVERSION line? Do we even need the System.map file anymore? Add headers for lirc package, wtf? dm headers, inotify, blah blah, blah. This is a ridiculous PKGBUILD that two people know what is going on, and that is unacceptable. On Nov 9, 2007 8:59 AM, Thomas Bächler <thomas@archlinux.org> wrote:
I was about to start this thread just this morning, but didn't have the time. Here is what I think: The Arch way is to keep things as untouched as possible and frankly, our kernel has become full of stuff that I don't see a reason for.
dmcgee@dublin /var/abs/core/base/kernel26 $ ls | wc -l 38 Wow. I agree that this is getting out of hand. We need 38 files to build a "vanilla" kernel? As I said above, what happened to the make/make install concept?
* acpi-dsdt-initrd-v0.8.4-2.6.21.patch
What I definitely don't like about the patch is that the ACPI subsystem has changed substantially since 2.6.21 - the fact that the patch still applies doesn't mean it doesn't introduce unwanted and stupid behaviour, or hidden bugs. None of us would know the problems the patch may cause, as none of us knows anything about kernel/ACPI code.
And if it hasn't been merged upstream, then there is an issue here.
* toshiba-bluetooth.patch
Die.
* http://www.archlinux.org/~tpowa/alsa-patches/alsa-20071109.patch.bz2 *http://www.archlinux.org/~tpowa/alsa-patches/alsa-include-20071109.patch.bz2
Why do we update alsa all the time, especially to non-released versions? This may have been necessary, when a large number of chipsets were broken, but I don't think that is still the case. Is there any specific reasons we cannot stick with the alsa version that Linux provides?
I agree with Thomas here.
* acpi-buggy-bios.patch
Like mactel, this patch has been there since .20 or so, the ACPI implementation has had many changes, and it is more likely that it introduces bugs than that it fixes any. Again, this patch should be removed for one of the above reasons.
Die.
squashfs3.2-patch.bz2 http://download.filesystems.org/unionfs/unionfs-2.1/unionfs-2.1.8_for_2.6.23...
Why are none of these merged upstream? Why do we have to add features ourselves? I know we had problems with externally built unionfs, but adding it in-kernel shouldn't be the solution, especially because it results in more work maintaining.
* put_filp.patch * lhash-2.6.22.patch
I don't know what these are, but these seem to be new bugfixes. Still, it should be evaluated if they are still needed before they are added again in 2.6.24.
Document these patches please.
hibernate-saa7134.diff
How long has this been sitting here? Again, the "why hasn't it been merged" question should be asked, and the patch removed, most likely.
Agree.
sata_sis-2.6.23.patch ipw2x00-2.6.23.patch zd1211rw-oops-2.6.23.patch ide-helper-2.6.23.patch
I guess those are new, so again, we can keep them until .24, but they should be fixed by then and our patches removed.
* linux-phc-0.3.0-kernel-vanilla-2.6.23rc3.patch
This is an extra feature. One that a small minority of people can use. It is said to be unintrusive, but then, why isn't it in the upstream version? If it is so unintrusive, the upstream devs shouldn't object to adding it. So either it was never submitted or it was rejected for a good reason.
<sarcasm>Glad that it got added without much discussion.</sarcasm>
# genpatches from gentoo * 2405_hostap-netdev-type.patch * 2525_usb-storage-nikon-d200-quirk.patch * 2530_usb-storage-nikon-d40x-quirk.patch
What do these do? Why do we need them?
* pre-2.6.23.2.patch
Where does this patch come from? Where is it (not in our CVS)? The 2.6.23.y git tree shows no new patches since .23.1, so I would at least like a comment about it.
My opinion is, we should remove most of the above patches as 1) we don't know what bugs they may introduce, 2) they haven't been merged upstream for a reason or 3) they have been merged but in a different way, introducing two fixes for the same problem, and the one in the vanilla kernel is most likely much cleaner.
It appears to me that tpowa often blindly adds patches that are submitted, with the only condition that they apply and build.
I want to reduce the number of patches applied to our kernel, and I want that any patch that adds a new feature or bugfix MUST be merged in a future version. If a patch has never been submitted, it should be submitted, if it has been submitted and rejected, it should be dropped.
I agree with Thomas' sentiment here. We seem to be deviating from our own "strictly vanilla" kernel rules a bit too much. At every major kernel release, *every* patch should be reviewed and a decision made on future inclusion. I'm not asking for an all-dev vote here or anything, as that would be stupid, but I am asking for more than one person to decide. I don't mean to come across as critical here, as I appreciate you guys maintaining the kernel. But I think we need to continuously improve what we do, and this is one of those areas we could do that in. -Dan
* put_filp.patch * lhash-2.6.22.patch
I don't know what these are, but these seem to be new bugfixes. Still, it should be evaluated if they are still needed before they are added again in 2.6.24.
Document these patches please.
They were documented where used, not in the array.. which I did to keep the array from becoming unwieldy. I've now added it to the array, making it more unwieldy. We really need to agree on some conventions about how to do this stuff. If someone trying to balance the factors and do the right thing and do no wrong can't get it right, the solution isn't just to say "document these patches please". I already did that, just not where people were looking, apparently. The problem isn't the lack of documentation, it's, as James pointed out, that the PKGBUILD is a disaster and needs some normalization applied to it. - P
On Nov 9, 2007 8:59 AM, Thomas Bächler <thomas@archlinux.org> wrote:
I was about to start this thread just this morning, but didn't have the time. Here is what I think: The Arch way is to keep things as untouched as possible and frankly, our kernel has become full of stuff that I don't see a reason for.
Yes yes yes. I don't understand when this happened. When I first started using Arch, the kernel was barebones and vanilla. If a user wanted something exotic, they used ABS, and it's done. Now there seems to be this attitude of "why rebuild it, we'll do it for you" - which is shit. We've switch gears for the worse. The original view for Arch was selfish. The original view was "this is the distro *I* want, and screw you if you don't like it". Now we're switching over to "We'll make the distro you guys want'. We're losing touch. Thomas is right here - there no reason for us to support half the shit in the kernel that we do. There are some exceptions, yes (unionfs and squashfs were added to support _our_ tools, which again follows the "this is my distro" creed). When did this happen? When did all you guys stop making a distro for yourselves, and start making it for other people?
Am Freitag, 9. November 2007 schrieb Thomas Bächler:
* toshiba-bluetooth.patch
What does this patch even do? It has been sitting there since .21. What is the reason this hasn't been merged upstream? 1) It breaks other setups, which would mean we should remove it, 2) it has been merged in a different way and still applies, in which case it will only cause bugs, but don't fix any, so we should remove it or 3) it has never been submitted, in which case it shouldn't be our responsibility to keep it along while the author isn't even capable of submitting it. In any of the above cases, we should remove the patch. http://bugs.archlinux.org/task/5608 acpi_toshiba module is not developed anymore.
* mactel-linux-2.6.23.patch
It was requested on forum or bugtracker can't remember, it fixes issues on macbooks and is well maintained, most stuff merged upstream already if you look at the different sizes of the patches.
* acpi-buggy-bios.patch
Like mactel, this patch has been there since .20 or so, the ACPI implementation has had many changes, and it is more likely that it introduces bugs than that it fixes any. Again, this patch should be removed for one of the above reasons.
This patch was requested by someone that has issues with his acpi and bios. To get correct c states It was posted on kernel bugtracker and rejected because it violates the official acpi specifications. but it was needed for his pc to get it working properly. http://bugzilla.kernel.org/show_bug.cgi?id=7578 http://bugs.archlinux.org/task/6875 -- Tobias Powalowski Archlinux Developer & Package Maintainer (tpowa) http://www.archlinux.org tpowa@archlinux.org
Tobias Powalowski schrieb:
Am Freitag, 9. November 2007 schrieb Thomas Bächler:
* toshiba-bluetooth.patch http://bugs.archlinux.org/task/5608 acpi_toshiba module is not developed anymore.
Why is it not developed anymore? Has the patch been submitted. However, I think we can keep that one for now. It should be submitted though.
* mactel-linux-2.6.23.patch
It was requested on forum or bugtracker can't remember, it fixes issues on macbooks and is well maintained, most stuff merged upstream already if you look at the different sizes of the patches.
If the patch is actually shrinking and expected to be gone some time, I am completely fine with it.
* acpi-buggy-bios.patch This patch was requested by someone that has issues with his acpi and bios. To get correct c states It was posted on kernel bugtracker and rejected because it violates the official acpi specifications. but it was needed for his pc to get it working properly. http://bugzilla.kernel.org/show_bug.cgi?id=7578 http://bugs.archlinux.org/task/6875
I don't like the fact that the patch has been rejected at all. It seems to be reasonable though - and I have to say that the guys at Asus support seem to be real asses from what I read there. However, we already have the custom DSDT patch, can't this problem be solved with a custom DSDT? This way, we'd only have one patch that the upstream ACPI developers don't like instead of two. We should document both the toshiba and the speedstep patch very carefully in the PKGBUILD (not in a one-liner but in a long description) if they are going to stay.
On Fri, Nov 09, 2007 at 12:28:43PM +0000, Tom K wrote:
Well, this kicked itself off in IRC this morning (timezone GMT+1), and as it seems most people didn't see it and/or didn't participate, we should probably do it here.
Here's the question, as I see it - what do we patch kernel26 for? Bugfixes? Stability? Enhancements? New features? Other?
Specific case under discussion: http://bugs.archlinux.org/task/8500 - a request, from a single user so far, for a new feature in the kernel.
We also have, or had, a request for fbsplash (closed, of course) and there are probably others - Romashka?
My understanding of The Arch Way, as it applies here, is that we patch for bugfixes e.g. ipw2x00, and for enhancements to existing functionality e.g. alsa. In the referenced bug report, there are differing dev views expressed - as I'm posting this, I'll include my view, which is that we should not add new features to kernel26.
I'll also include a possible compromise:
/me grabs big can of worms, opens it, pours it out onto DevLand :)
Two kernels - a strictly as-vanilla-as-possible kernel26 in Core, and a more let's-try-new-features-here kernel26extra in Extra. It could be in Community either, but I would see it as an official Arch package, supported by the dev team. I would be happy to maintain it if necessary, along with the usual array of external module packages - in cooperation, of course, with the current kernel maintainers.
Right now, anyone who has read the IRC log knows how Tobias P and Aaron feel about it, but I expect others have views on this topic too, so let's hear them.
At the risk of sounding like an ass for a minute, I really don't understand how this is even an issue/discussion. We've pretty much always advertised packages "as vanilla as possible". I don't really see the need to be re-dicussing it. In short, this design desicion was already made when this distro was started, and was likely among the reasons you started using this distro. There is no need to rediscuss it, our philosophy on this is pretty clear, since it's always been "as vanilla as possible". -S PS: This is just one of the many sad incarnations of our current trend of throwing out or questioning everything that defines ARCH as ARCH. And quite frankly it kind of pisses me off that we're ever so slowly but surely gravitiating away from these core values. I don't know about you, but I started using Arch because I liked the way it did things, not because I wanted to change everything. If that were the case, I would have gone elsewhere.
Simo Leone wrote:
On Fri, Nov 09, 2007 at 12:28:43PM +0000, Tom K wrote:
Well, this kicked itself off in IRC this morning (timezone GMT+1), and as it seems most people didn't see it and/or didn't participate, we should probably do it here.
Here's the question, as I see it - what do we patch kernel26 for? Bugfixes? Stability? Enhancements? New features? Other?
Specific case under discussion: http://bugs.archlinux.org/task/8500 - a request, from a single user so far, for a new feature in the kernel.
We also have, or had, a request for fbsplash (closed, of course) and there are probably others - Romashka?
My understanding of The Arch Way, as it applies here, is that we patch for bugfixes e.g. ipw2x00, and for enhancements to existing functionality e.g. alsa. In the referenced bug report, there are differing dev views expressed - as I'm posting this, I'll include my view, which is that we should not add new features to kernel26.
I'll also include a possible compromise:
/me grabs big can of worms, opens it, pours it out onto DevLand :)
Two kernels - a strictly as-vanilla-as-possible kernel26 in Core, and a more let's-try-new-features-here kernel26extra in Extra. It could be in Community either, but I would see it as an official Arch package, supported by the dev team. I would be happy to maintain it if necessary, along with the usual array of external module packages - in cooperation, of course, with the current kernel maintainers.
Right now, anyone who has read the IRC log knows how Tobias P and Aaron feel about it, but I expect others have views on this topic too, so let's hear them.
At the risk of sounding like an ass for a minute, I really don't understand how this is even an issue/discussion. We've pretty much always advertised packages "as vanilla as possible". I don't really see the need to be re-dicussing it.
The problem is that we never really said what "as vanilla as possible" meant. In fact, we have tried to maintain vanilla behavior while still allowing the kernel to be flexed enough to support a full-featured distro. But I think I still agree with Simo on this one.. no need to change anything, but we should continue to scrutinize patches we add and be prepared to defend any patches we currently include, as we have been doing in one form or another for years. - P
Kernel maintaining is not easy (as probably those who done it know) , the main effort of the kernel is to satisfy the needs of the users/developers hardware and features. it's not about vanilla and or not vanilla it *HAS TO WORK* and must be maintainable in a reasonable amount of time, thats the real thing of a kernel. If something breaks it must be fixed or removed, plain simple. (my opinion) A maintainer of a package, who does the real work on it, reads bugreports, looks for fixes, tries to get a better package than others and tries to add userfriendlieness seems to not fit to the arch philosophy anymore, should probably do something else. If you say it must be plain vanilla as kernel.org is, go ahead do it, but without me. All this bikeshedding lately about everything isn't much fun and i don't need that. /me disappears and takes a full timeout for an uncertain amount of time. have fun guys, greetings tpowa -- Tobias Powalowski Archlinux Developer & Package Maintainer (tpowa) http://www.archlinux.org tpowa@archlinux.org
On Nov 9, 2007 11:05 AM, Tobias Powalowski <t.powa@gmx.de> wrote:
Kernel maintaining is not easy (as probably those who done it know) , the main effort of the kernel is to satisfy the needs of the users/developers hardware and features. it's not about vanilla and or not vanilla it *HAS TO WORK* and must be maintainable in a reasonable amount of time, thats the real thing of a kernel. If something breaks it must be fixed or removed, plain simple. (my opinion)
Ugh. Man. I don't know how to say this any different that the last 40 times I said this: we're not talking about bug fixes. Bug fixes are fine and expected. We're talking about miscellaneous feature requests
A maintainer of a package, who does the real work on it, reads bugreports, looks for fixes, tries to get a better package than others and tries to add userfriendlieness seems to not fit to the arch philosophy anymore, should probably do something else.
If you say it must be plain vanilla as kernel.org is, go ahead do it, but without me.
Are you saying that you do not wish to maintain the kernel? That's fine - no one is forcing your hand, and I'd be willing to do it in your stead.
On Fri, Nov 09, 2007 at 11:11:45AM -0600, Aaron Griffin wrote:
Are you saying that you do not wish to maintain the kernel? That's fine - no one is forcing your hand, and I'd be willing to do it in your stead.
I don't like the conversation style: It's not fine. Technical opinions differ but tpowa does the hard job and I'm glad he is doing a good job. Vanilla is our primary target but sometimes you have to take a pragmatic approach on special packages like the kernel. Jürgen
On Nov 9, 2007 11:36 AM, Jürgen Hötzel <juergen@hoetzel.info> wrote:
On Fri, Nov 09, 2007 at 11:11:45AM -0600, Aaron Griffin wrote:
Are you saying that you do not wish to maintain the kernel? That's fine - no one is forcing your hand, and I'd be willing to do it in your stead.
I don't like the conversation style: It's not fine. Technical opinions differ but tpowa does the hard job and I'm glad he is doing a good job. Vanilla is our primary target but sometimes you have to take a pragmatic approach on special packages like the kernel.
Let me rephrase that then: "If you do not want to maintain the kernel, no one will hold it against you" I understand tpowa does the hard work, but the issue here is that a large chunk of the developers are against this change, are against the feature-creep that's showing up in the kernel package. If that's something that cannot be lived with by the current maintainer, then we simply need a new maintainer - this is true of ANY package. If people are saying "oh god, you're screwing this up!" and a maintainer refuses to listen, well, that's a problem.
Tobias Powalowski schrieb:
it's not about vanilla and or not vanilla it *HAS TO WORK* and must be maintainable in a reasonable amount of time, thats the real thing of a kernel.
If you would have read my mail, you knew that this is exactly what I have doubts about: How do you know that a certain bugfix works and doesn't add even more bugs?
All this bikeshedding lately about everything isn't much fun and i don't need that.
/me disappears and takes a full timeout for an uncertain amount of time.
Maybe you should have added the missing patches to CVS first, I can't work with that incomplete stuff you left.
Thomas Bächler wrote:
Tobias Powalowski schrieb:
it's not about vanilla and or not vanilla it *HAS TO WORK* and must be maintainable in a reasonable amount of time, thats the real thing of a kernel.
If you would have read my mail, you knew that this is exactly what I have doubts about: How do you know that a certain bugfix works and doesn't add even more bugs?
You can't be sure. All you can do is make small modifications that you believe you have your head around, and try to do no harm. And fix/revert it if it breaks something. But I don't think we should look at something obviously wrong and say "no, we're going to leave that broken". It's a nice ideal, but it's just not practical, like Tpowa says. It's very easy to not apply bugfixes, until the day when Linus's kernel break your network driver. Then it's just a hassle for 1000 people to have to recompile the kernel package because we won't include the fix. We make things work, and we accommodate the needs of our community, as cleanly as possible. I think we should continue to do that. Per Jason's fine example on the CVS stuff, I'm going to try to pick something relevant to the situation and actually do it to improve things. I'm adding to my list to try to get unionfs to work outside the kernel. It may take a while, possibly even weeks, but I'm choosing that one and I'm going to work on it. In the end, it will probably need some in-kernel patches to expose functionality, but I will try to reduce them to the minimum. I will report back here on my progress. - P
/me disappears and takes a full timeout for an uncertain amount of time. have fun guys, greetings tpowa
Tobias, you have proved to do a very good job maintaining the kernel26 now for both architectures. And our community seems to be satisfied with your work. Yes, Arch's mission is to ship KISS packages. Mainly in the way to use and setup the system, not always KISS to package. The package should simply work in most cases. And when it's broken in certain parts we fix it. Even if this means backporting features(=sometimes hardware enhancements) and fixes not applied in Linus tree. Hey, we are Arch - means we are known to ship a binary metadistribution offering the base for latest and greatest OpenSource features and packages. Keeping that in mind I wonder why some of us want to break down all patching and feature enhancements. Some of us are working on LiveCDs, why not. Others work on Compiz'ing all desktops. Why do you want to forbid the master kernel26 maintainer to add features he wants to support? Me just wonders. As Linus said it's up to the distributions to make a good kernel out of what they ship "vanilla". It just depends on your target user group. We are not here to only satisfy our own! The only point I see tpowa should improve is a bit more documentation for all the added lines and patches. What I see more and more becoming a big problem is defining the Arch way. see dev /wiki/Goals/ /wiki/Objectives/ - both are just suggestions. Did we finish that discussion? The bigger we grow the harder it becomes to satisfy the "edge" (desktop) user at the same time together with those who want to run Arch on critical (server) systems. Many compromises we made and just are discussing about could be solved by splitting arch up into two dedicated parts: one for the edge and one for critical systems. Some seem to like it and some don't. Andy
On Nov 9, 2007 2:12 PM, Andreas Radke <a.radke@arcor.de> wrote:
What I see more and more becoming a big problem is defining the Arch way.
What's wrong with this? This is how I've always done things. http://wiki.archlinux.org/index.php/The_Arch_Way
The sources are documented in PKGBUILd and the things are all from stable kernel trees. We have no stability problem else you would have read much more on the ML,FORUM or BUGTRACKER. (You want to pretend this but it doesn't matter which road you drive you are never sure if it works or breaks something else, it's too complex. Also .x kernels can have issues, the perfect kernel is and will not be reality, this is what i can tell ya of doing it the last months/years) You never know when linus or greg decides to release .x kernels. So interim patching makes sense. The particular feature request is usable and works, i tested it here and if you don't give a damn about acpi-cpufreq(which is not autoloaded at all) you are not affected at all. If this patch existed 1.5 years ago i wouldn't have needed to change the dip switches of my mainboard to undervolt. I was flamed for not adding genpatches some time ago, now we have some of them, wham again flamed. Last time i was flamed for not adding particular filesystems to the kernel because i doubted they port their patches to higher kernels (in the beginning, after seeing it's maintained well it's not a big deal) Using latest alsa has big benefits of supporting latest hardware and get latest fixes for the drivers and it's a seperate kernel subsystem where this works. (and sorry to say that, alsa seems to have issues everytime, thats not the fault of alsa but the fault of the vendors that use so many different types of chips and bioses) Violating KISS is not the point here it has nothing to do with it, also the arch way is not offended. Telling that the majority of devs are against it, show it to me. (The loudest ones are not the majority) I always thought we trust each other in doing the work, but flaming and bikeshedding seems to be our new hobby here. have a nice day greetings tpowa -- Tobias Powalowski Archlinux Developer & Package Maintainer (tpowa) http://www.archlinux.org tpowa@archlinux.org
Tobias Powalowski schrieb:
The sources are documented in PKGBUILd and the things are all from stable kernel trees.
Not all of them. Can you do me a favor: Can you please reply to my first mail and comment on the patches that have been sitting in our kernel for a looong time? I really want to know why we need some hibernate fix, mactel, toshiba-bluetooth and "acpi buggy bios" patch for 3 or 4 _major_ kernel releases, while those patches should have been merged by now (if they have even been submitted). I remember talk about a patch lately, that fixed an issue but openend another (discussion took place on the bugtracker). How do you ensure this doesn't happen?
(You want to pretend this but it doesn't matter which road you drive you are never sure if it works or breaks something else, it's too complex. Also .x kernels can have issues, the perfect kernel is and will not be reality, this is what i can tell ya of doing it the last months/years)
Stability is one thing, but the work should not be completely on our backs. We should patch certain issues in our kernel, but they should be gone in the next major release, otherwise something is really wrong here!
You never know when linus or greg decides to release .x kernels. So interim patching makes sense.
Using the prepatch for 2.6.23.2 is a _very_ good idea, I just wondered where you got it from (turns out I was looking at the wrong git tree).
The particular feature request is usable and works, i tested it here and if you don't give a damn about acpi-cpufreq(which is not autoloaded at all) you are not affected at all. If this patch existed 1.5 years ago i wouldn't have needed to change the dip switches of my mainboard to undervolt.
I use acpi-cpufreq. And I have a very strong opinion about it, as I don't want it to be broken in _any_ way (I depend on it very much). While it doesn't seem to be broken here, it introduces very dangerous functionality. I don't like the idea of adding a feature that has explicitly been rejected by upstream developers. And I don't like the fact that you added it while we had an unfinished discussion about it on the bugtracker.
Using latest alsa has big benefits of supporting latest hardware and get latest fixes for the drivers and it's a seperate kernel subsystem where this works. (and sorry to say that, alsa seems to have issues everytime, thats not the fault of alsa but the fault of the vendors that use so many different types of chips and bioses)
I am not sure if this is really still necessary. Surprisingly, this has worked out well and people usually don't complain about the state of our alsa. So I am willing to go along with it. You should however document the way you are creating these alsa patches in case you are gone for a while.
Violating KISS is not the point here it has nothing to do with it, also the arch way is not offended. Telling that the majority of devs are against it, show it to me. (The loudest ones are not the majority) I always thought we trust each other in doing the work, but flaming and bikeshedding seems to be our new hobby here.
I wouldn't take this thread as an offense, but as an opportunity to rethink the way you are working things with the kernel. I think we should evaluate the old patches more often and maybe throw some away every now and then. I don't think anybody is implying that having a pure vanilla kernel is the way to go, but it would be a good thing if we could reduce the number of patches we apply to it. I know how much work is involved with a major kernel update in Arch and it is really appreciated that you do it. And I have always supported you when I could. I have always offered my opinion when I thought you did something stupid. Still, there is always room for improvement, and I will present some ideas for better management of the -ARCH kernel soon.
Tobias Powalowski schrieb:
The sources are documented in PKGBUILd and the things are all from stable kernel trees.
Not all of them. Can you do me a favor: Can you please reply to my first mail and comment on the patches that have been sitting in our kernel for a looong time? I really want to know why we need some hibernate fix, mactel, toshiba-bluetooth and "acpi buggy bios" patch for 3 or 4 _major_ kernel releases, while those patches should have been merged by now (if they have even been submitted). I remember talk about a patch lately, that fixed an issue but openend another (discussion took place on the bugtracker). How do you ensure this doesn't happen?
(You want to pretend this but it doesn't matter which road you drive you are never sure if it works or breaks something else, it's too complex. Also .x kernels can have issues, the perfect kernel is and will not be reality, this is what i can tell ya of doing it the last months/years)
Stability is one thing, but the work should not be completely on our backs. We should patch certain issues in our kernel, but they should be gone in the next major release, otherwise something is really wrong here!
You never know when linus or greg decides to release .x kernels. So interim patching makes sense.
Using the prepatch for 2.6.23.2 is a _very_ good idea, I just wondered where you got it from (turns out I was looking at the wrong git tree).
The particular feature request is usable and works, i tested it here and if you don't give a damn about acpi-cpufreq(which is not autoloaded at all) you are not affected at all. If this patch existed 1.5 years ago i wouldn't have needed to change the dip switches of my mainboard to undervolt.
I use acpi-cpufreq. And I have a very strong opinion about it, as I don't want it to be broken in _any_ way (I depend on it very much). While it doesn't seem to be broken here, it introduces very dangerous functionality. I don't like the idea of adding a feature that has explicitly been rejected by upstream developers. And I don't like the fact that you added it while we had an unfinished discussion about it on the bugtracker.
Using latest alsa has big benefits of supporting latest hardware and get latest fixes for the drivers and it's a seperate kernel subsystem where this works. (and sorry to say that, alsa seems to have issues everytime, thats not the fault of alsa but the fault of the vendors that use so many different types of chips and bioses)
I am not sure if this is really still necessary. Surprisingly, this has worked out well and people usually don't complain about the state of our alsa. So I am willing to go along with it. You should however document the way you are creating these alsa patches in case you are gone for a while.
Violating KISS is not the point here it has nothing to do with it, also the arch way is not offended. Telling that the majority of devs are against it, show it to me. (The loudest ones are not the majority) I always thought we trust each other in doing the work, but flaming and bikeshedding seems to be our new hobby here.
I wouldn't take this thread as an offense, but as an opportunity to rethink the way you are working things with the kernel. I think we should evaluate the old patches more often and maybe throw some away every now and then. I don't think anybody is implying that having a pure vanilla kernel is the way to go, but it would be a good thing if we could reduce the number of patches we apply to it. I know how much work is involved with a major kernel update in Arch and it is really appreciated that you do it. And I have always supported you when I could. I have always offered my opinion when I thought you did something stupid. Still, there is always room for improvement, and I will present some ideas for better management of the -ARCH kernel soon.
The particular feature request is usable and works, i tested it here and if you don't give a damn about acpi-cpufreq(which is not autoloaded at all) you are not affected at all. If this patch existed 1.5 years ago i wouldn't have needed to change the dip switches of my mainboard to undervolt.
I use acpi-cpufreq. And I have a very strong opinion about it, as I don't want it to be broken in _any_ way (I depend on it very much). then you are the perfect person to report issues if there will be any (which i hardly doubt). You didn't even tell us if the -7 kernel works on your system. It adds sysfs functionalty if you don't change something on it nothing happens.
-- Tobias Powalowski Archlinux Developer & Package Maintainer (tpowa) http://www.archlinux.org tpowa@archlinux.org
Tobias Powalowski schrieb:
I use acpi-cpufreq. And I have a very strong opinion about it, as I don't want it to be broken in _any_ way (I depend on it very much). then you are the perfect person to report issues if there will be any (which i hardly doubt). You didn't even tell us if the -7 kernel works on your system. It adds sysfs functionalty if you don't change something on it nothing happens.
I have crappy bandwidth right now and can only sync my mirror when I am at the university. I have synced on Friday and do have -7 here but there was no chance for me to update yet as I couldn't reboot.
I was never even near being worried about Tobias' decisions for kernel26. Most times when I actually touched the PKGBUILD was to get a rc-kernel up on my machine to try it out. Now that I have .24-rc2 running here, I have to say that if it weren't for Tobias' work, I wouldn't have a working system to this day. Even though I don't know what he fixed as there are no PKGBUILDs provided yet, a vanilla version just plainly fucked my machine even before it even tried to detect my disks. Anyway, all I'm saying is that I don't want to see anything to change in the way he maintains the kernel at the moment. Everything added with patches makes sense to me or doesn't affect me, so I am - and probably 98% of the users that use our stock kernel - very very satisfied. Tobias Powalowski wrote:
/me disappears and takes a full timeout for an uncertain amount of time. have fun guys, greetings tpowa
That's really too bad but I guess since for a while some of us are constantly moody about issues with communications. I feel that we really should have a piss up with all devs and tus rather sooner than later. Linuxtag seemed to bring us closer to understand the email communication of the other and also respect opinions of the other more. This whole thread should just be locked, too bad that's not possible... Cheers, -H
On Mon, Nov 12, 2007 at 08:42:58PM +0100, Alexander Baldeck wrote:
Anyway, all I'm saying is that I don't want to see anything to change in the way he maintains the kernel at the moment. Everything added with patches makes sense to me or doesn't affect me, so I am - and probably 98% of the users that use our stock kernel - very very satisfied.
I second this. I doubt any user had trouble because of tpowas patches. But I'm sure many users had a benefit because of more useful kernel on their hardware. And when kernel patches doesn't alter kernel internals but add optional functionality it's still like vanilla to me. Jürgen
On Nov 12, 2007 1:42 PM, Alexander Baldeck <kth5@archlinuxppc.org> wrote:
I was never even near being worried about Tobias' decisions for kernel26. Most times when I actually touched the PKGBUILD was to get a rc-kernel up on my machine to try it out. Now that I have .24-rc2 running here, I have to say that if it weren't for Tobias' work, I wouldn't have a working system to this day. Even though I don't know what he fixed as there are no PKGBUILDs provided yet, a vanilla version just plainly fucked my machine even before it even tried to detect my disks.
I agree. I didn't mean to imply (as someone else pointed out) that I disliked the decisions that tpowa makes. I just wanted to point out that the kernel is very important. It is super important, and if people take issue with something, one person shouldn't be able to pull a trump card like "it's my package, I'll do what I want". This is fine with something stupid like gtkpod, but with the kernel, or glibc, or one of those important packages, if something comes into question, we need to address it. As it stands, the patch that is the root of this was questioned by James, Thomas, and myself. None of us liked it, but it was added in anyway - ignoring the rest of us completely. It's unacceptable to act like no one else knows what they're talking about.
That's really too bad but I guess since for a while some of us are constantly moody about issues with communications. I feel that we really should have a piss up with all devs and tus rather sooner than later.
Erm? Random slang I don't get - what is a "piss up"? I have a feeling it has to do with drinking, but it sounds like some sort of fist fight to me.
Aaron Griffin wrote:
Erm? Random slang I don't get - what is a "piss up"? I have a feeling it has to do with drinking, but it sounds like some sort of fist fight to me.
"Piss up", english term used by british anarchist to say "let's get drunk".
On Nov 12, 2007 3:11 PM, Alexander Baldeck <kth5@archlinuxppc.org> wrote:
Aaron Griffin wrote:
Erm? Random slang I don't get - what is a "piss up"? I have a feeling it has to do with drinking, but it sounds like some sort of fist fight to me.
"Piss up", english term used by british anarchist to say "let's get drunk".
Bollocks!
On Tue, November 13, 2007 07:54, Aaron Griffin wrote:
On Nov 12, 2007 1:42 PM, Alexander Baldeck <kth5@archlinuxppc.org> wrote:
I was never even near being worried about Tobias' decisions for kernel26. Most times when I actually touched the PKGBUILD was to get a rc-kernel up on my machine to try it out. Now that I have .24-rc2 running here, I have to say that if it weren't for Tobias' work, I wouldn't have a working system to this day. Even though I don't know what he fixed as there are no PKGBUILDs provided yet, a vanilla version just plainly fucked my machine even before it even tried to detect my disks.
I agree. I didn't mean to imply (as someone else pointed out) that I disliked the decisions that tpowa makes. I just wanted to point out that the kernel is very important.
We're not out against tpowa, we're suggesting improvements. So far mentioned: 1) Documentation of patches. Where are they from? What they do? Why are they included? (mostly done now) 2) Re-evaluating patches with each release... deally we shouldnt see patches in 2.6.22 that have been updated from 2.6.12 or even 2.6.21. 3) Proper review of patches. tpowa's done an awesome job, and his judgement has been great so far -- he isn't infallible as the undervolting patch shows. James
On Nov 12, 2007 6:52 PM, James Rayner <iphitus@gmail.com> wrote:
On Tue, November 13, 2007 07:54, Aaron Griffin wrote:
On Nov 12, 2007 1:42 PM, Alexander Baldeck <kth5@archlinuxppc.org> wrote:
I was never even near being worried about Tobias' decisions for kernel26. Most times when I actually touched the PKGBUILD was to get a rc-kernel up on my machine to try it out. Now that I have .24-rc2 running here, I have to say that if it weren't for Tobias' work, I wouldn't have a working system to this day. Even though I don't know what he fixed as there are no PKGBUILDs provided yet, a vanilla version just plainly fucked my machine even before it even tried to detect my disks.
I agree. I didn't mean to imply (as someone else pointed out) that I disliked the decisions that tpowa makes. I just wanted to point out that the kernel is very important.
We're not out against tpowa, we're suggesting improvements. So far mentioned:
1) Documentation of patches. Where are they from? What they do? Why are they included? (mostly done now) 2) Re-evaluating patches with each release... deally we shouldnt see patches in 2.6.22 that have been updated from 2.6.12 or even 2.6.21. 3) Proper review of patches. tpowa's done an awesome job, and his judgement has been great so far -- he isn't infallible as the undervolting patch shows.
I'd like to (re)mention one improvement, now that the fire has cooled off. :) 4) I'd love to see our kernel PKGBUILD be a bit less scary. At least right now, I would not feel comfortable using it to build my own kernel, replacing the existing Arch one, and having faith that it would boot up. I think if it was a bit more transparent and made it easier to drop things in (such as new patches, a different config, etc) that would be great for both other developers AND the community, without sacrificing anything technical. -Dan
participants (14)
-
Aaron Griffin
-
Alexander Baldeck
-
Andreas Radke
-
Dan McGee
-
ganja guru
-
James
-
James Rayner
-
Jürgen Hötzel
-
Paul Mattal
-
Roman Kyrylych
-
Simo Leone
-
Thomas Bächler
-
Tobias Powalowski
-
Tom K