[arch-dev-public] Debug packages for Arch Linux
I'm very happy to announce that debug packages in Arch Linux has been deployed :) This work began after FOSDEM 2020 and was announced in November 2020. However because of time constraints it took quite a while before we managed to deploy it. Currently we have a debuginfod service which is capable of delivering source listings and debug symbols to users with gdb, delve and other debuggers. This can be enabled by installing the `debuginfod` package and setting the environment variable `DEBUGINFOD_URLS="https://debuginfod.archlinux.org"`. The `debuginfod` package is also going to be providing this variable :) The debug package repositories themself are not publicly accessible nor distributed to our mirrors. This is because there is a general concern around the repository size increase. The goal is to take a look at distributing them and/or making them accessible in the future. Packages are being rebuilt for debug symbols where deemed needed by the packager and I suspect we should have a discussion around enabling the debug option by default or not. However see the section below around creating debug packages as there are some caveats of the current approach. I have added some links to the work that has been done to accomplish all of this at the bottom of this email. For general information about the instance one can look at the service itself: https://debuginfod.archlinux.org/ Previous emails on this topic: * https://lists.archlinux.org/pipermail/arch-dev-public/2020-November/030222.h... * https://lists.archlinux.org/pipermail/arch-dev-public/2021-December/030583.h... # archlinux.org news announcement Suggestion for the news announcement on archweb. I'll post it in a day or two unless there are any objections. Subject: Debug packages and debuginfod We are very happy to announce that debug packages are now available in Arch Linux. Debug symbols and source listing are provided through our debuginfod instance which can be utilized by debuggers such as gdb and delve. In the future we are planning to make these repositories public on our mirrors. For more information on this please visit the [archwiki](https://wiki.archlinux.org/title/Debuginfod) article, Or the debuginfod service. https://debuginfod.archlinux.org/ # Creating debug packages Currently with pacman 6.0.1-3 debug package only works on C/C++ projects. This is mostly because pacman utilizes a fairly ugly awk hack to extract sources and if it encounters a binary from Go or Rust (as an example) it is simply unable to deal with them and produces invalid packages. This is fixed with a patch i wrote which replaces the AWK hack with debugedit. https://gitlab.archlinux.org/pacman/pacman/-/commit/ae2f506ddfd11d9becda7216... Until this patch is backported, or a pacman release is done, I would advise us to not build debug packages for anything that isn't using gcc/clangd. Saying that, I'm unsure how we should be documenting debug packages across our distro. They probably belong in package guidelines sections of each package, or we make a common wiki page for this. Generally Rust needs a environment variable and Golang needs a bit of work to disable decompressed DWARF headers. We can probably work something out as we go along, currently I think we should try enable debug packages on our [core] packages. When we have pacman with the above patch I can try compile a list of available methods to get debug packages from Rust, Go and other languages. Obviously I'd need some help with that :) Cheers! -- Morten Linderud PGP: 9C02FF419FECBE16 # infrastructure https://gitlab.archlinux.org/archlinux/infrastructure/-/merge_requests/168 # dbscripts https://gitlab.archlinux.org/archlinux/dbscripts/-/merge_requests/21 https://gitlab.archlinux.org/archlinux/dbscripts/-/merge_requests/33 # devtools https://gitlab.archlinux.org/archlinux/devtools/-/merge_requests/78
On 30.01.2022 12.08, Morten Linderud via arch-dev-public wrote:
I'm very happy to announce that debug packages in Arch Linux has been deployed :)
This work began after FOSDEM 2020 and was announced in November 2020. However because of time constraints it took quite a while before we managed to deploy it.
Currently we have a debuginfod service which is capable of delivering source listings and debug symbols to users with gdb, delve and other debuggers. This can be enabled by installing the `debuginfod` package and setting the environment variable `DEBUGINFOD_URLS="https://debuginfod.archlinux.org"`. The `debuginfod` package is also going to be providing this variable :)
The debug package repositories themself are not publicly accessible nor distributed to our mirrors. This is because there is a general concern around the repository size increase. The goal is to take a look at distributing them and/or making them accessible in the future.
We have three sponsored machines, which are used as package mirrors and archive mirrors[1]. At the time of writing they have 14TB available storage each, so we could mirror the debug packages to them. [1] {america,asia,europe}.{archive,mirror}.pkgbuild.com - Kristian
On 30/01/2022 12:59, Kristian Klausen via arch-dev-public wrote:
On 30.01.2022 12.08, Morten Linderud via arch-dev-public wrote:
I'm very happy to announce that debug packages in Arch Linux has been deployed :)
This work began after FOSDEM 2020 and was announced in November 2020. However because of time constraints it took quite a while before we managed to deploy it.
Currently we have a debuginfod service which is capable of delivering source listings and debug symbols to users with gdb, delve and other debuggers. This can be enabled by installing the `debuginfod` package and setting the environment variable `DEBUGINFOD_URLS="https://debuginfod.archlinux.org"`. The `debuginfod` package is also going to be providing this variable :)
The debug package repositories themself are not publicly accessible nor distributed to our mirrors. This is because there is a general concern around the repository size increase. The goal is to take a look at distributing them and/or making them accessible in the future.
We have three sponsored machines, which are used as package mirrors and archive mirrors[1]. At the time of writing they have 14TB available storage each, so we could mirror the debug packages to them.
Thanks for deploying debuginfod Kristian/Morten! I think we should just mirror debug packages to our own controlled mirrors. Greetings, Jelle
On Mon, Jan 31, 2022 at 10:20:35AM +0100, Jelle van der Waa via arch-dev-public wrote:
On 30/01/2022 12:59, Kristian Klausen via arch-dev-public wrote:
We have three sponsored machines, which are used as package mirrors and archive mirrors[1]. At the time of writing they have 14TB available storage each, so we could mirror the debug packages to them.
Thanks for deploying debuginfod Kristian/Morten! I think we should just mirror debug packages to our own controlled mirrors.
Should I mention in the news post that the debug repositories should be available on our pkgbuild.com mirrors in a few days? Or should I wait with the announcement until this has been deployed? Thoughts? -- Morten Linderud PGP: 9C02FF419FECBE16
Hi, This is an updated news entry for the debug package announcement. We have been working on the system and provided a few repositories for the debug repositories as well. I'll post this when we have an updated "Getting traces" article ready :) # Debug packages and debuginfod We are very happy to announce that debug packages are now available in Arch Linux. Debug symbols and source listing are provided through our debuginfod instance which can be utilized by debuggers such as gdb and delve. https://debuginfod.archlinux.org/ A couple of sponsored mirrors are providing the debug repositories while we figure out and communicate the new [mirror requirements](https://wiki.archlinux.org/title/DeveloperWiki:NewMirrors). * https://america.mirror.pkgbuild.com * https://asia.mirror.pkgbuild.com * https://europe.mirror.pkgbuild.com For more information on this, please visit the [Debuginfod](https://wiki.archlinux.org/title/Debuginfod) wiki article, and also our newly renovated [Debugging/Getting traces](https://wiki.archlinux.org/title/Debugging/Getting_traces) article. -- Morten Linderud PGP: 9C02FF419FECBE16
Hey all, On 30/01/2022 12:08, Morten Linderud via arch-dev-public wrote:
I'm very happy to announce that debug packages in Arch Linux has been deployed :)
This work began after FOSDEM 2020 and was announced in November 2020. However because of time constraints it took quite a while before we managed to deploy it.
Currently we have a debuginfod service which is capable of delivering source listings and debug symbols to users with gdb, delve and other debuggers. This can be enabled by installing the `debuginfod` package and setting the environment variable `DEBUGINFOD_URLS="https://debuginfod.archlinux.org"`. The `debuginfod` package is also going to be providing this variable :)
The debug package repositories themself are not publicly accessible nor distributed to our mirrors. This is because there is a general concern around the repository size increase. The goal is to take a look at distributing them and/or making them accessible in the future.
Packages are being rebuilt for debug symbols where deemed needed by the packager and I suspect we should have a discussion around enabling the debug option by default or not. However see the section below around creating debug packages as there are some caveats of the current approach.
I have added some links to the work that has been done to accomplish all of this at the bottom of this email.
There are two questions which keep popping up around debug packages so I thought it would be good to answer them here. * How do I create a debug package? Add 'debug' to the options array in your PKGBUILD, bump pkgrel and rebuild. This should result into a debug package based on the 'pkgbase' of the package/PKGBUILD so for linux it ends up creating linux-debug-$pkgver-$pkgrel.tar.zst. * How do I verify if the debug package is generated correctly? After installing the debug package, gdb $binary should reveal: [jelle@t14s][~]%gdb /usr/bin/cryptsetup Reading symbols from /usr/bin/cryptsetup... Reading symbols from /usr/lib/debug/usr/bin/cryptsetup.debug... Apart from that the debug package should not contain any other paths then /usr/lib/debug and /usr/src/debug. Ideally namcap would print a warning when this is not the case. Hope this helps, Jelle van der Waa
On 4/2/22 22:49, Jelle van der Waa via arch-dev-public wrote:
* How do I create a debug package?
Add 'debug' to the options array in your PKGBUILD, bump pkgrel and rebuild. This should result into a debug package based on the 'pkgbase' of the package/PKGBUILD so for linux it ends up creating linux-debug-$pkgver-$pkgrel.tar.zst.
Are we intending to make that the default in devtools? Allan
On Fri, Feb 04, 2022 at 11:07:29PM +1000, Allan McRae via arch-dev-public wrote:
On 4/2/22 22:49, Jelle van der Waa via arch-dev-public wrote:
* How do I create a debug package?
Add 'debug' to the options array in your PKGBUILD, bump pkgrel and rebuild. This should result into a debug package based on the 'pkgbase' of the package/PKGBUILD so for linux it ends up creating linux-debug-$pkgver-$pkgrel.tar.zst.
Are we intending to make that the default in devtools?
Yes, but not with the current pacman release as it would produce more broken debug packages then valid ones. I also need to fix up a few debugedit issues which has been found the past week in terms of non-unique source directory in the debug package. -- Morten Linderud PGP: 9C02FF419FECBE16
On 04/02/2022 12:49, Jelle van der Waa via arch-dev-public wrote:
* How do I verify if the debug package is generated correctly?
After installing the debug package, gdb $binary should reveal:
[jelle@t14s][~]%gdb /usr/bin/cryptsetup Reading symbols from /usr/bin/cryptsetup... Reading symbols from /usr/lib/debug/usr/bin/cryptsetup.debug...
I've noticed that some packages say such thing but followed right after with:
Reading symbols from /usr/bin/bonnie++... Reading symbols from /usr/lib/debug/usr/bin/bonnie++.debug... (No debugging symbols found in /usr/lib/debug/usr/bin/bonnie++.debug) (gdb)
The above is wrong - it doesn't work, so just make sure the gdb prompt is right after the reading symbols line like:
Reading symbols from /usr/bin/shotcut... Reading symbols from /usr/lib/debug/usr/bin/shotcut.debug... (gdb)
Cheers, -- Leonidas Spyropoulos PGP: 59E43E106B247368
On 2022-02-04 13:49, Jelle van der Waa via arch-dev-public wrote:
Hey all,
On 30/01/2022 12:08, Morten Linderud via arch-dev-public wrote:
I'm very happy to announce that debug packages in Arch Linux has been deployed :)
This work began after FOSDEM 2020 and was announced in November 2020. However because of time constraints it took quite a while before we managed to deploy it.
Currently we have a debuginfod service which is capable of delivering source listings and debug symbols to users with gdb, delve and other debuggers. This can be enabled by installing the `debuginfod` package and setting the environment variable `DEBUGINFOD_URLS="https://debuginfod.archlinux.org"`. The `debuginfod` package is also going to be providing this variable :)
The debug package repositories themself are not publicly accessible nor distributed to our mirrors. This is because there is a general concern around the repository size increase. The goal is to take a look at distributing them and/or making them accessible in the future.
Packages are being rebuilt for debug symbols where deemed needed by the packager and I suspect we should have a discussion around enabling the debug option by default or not. However see the section below around creating debug packages as there are some caveats of the current approach.
I have added some links to the work that has been done to accomplish all of this at the bottom of this email.
There are two questions which keep popping up around debug packages so I thought it would be good to answer them here.
* How do I create a debug package?
Add 'debug' to the options array in your PKGBUILD, bump pkgrel and rebuild. This should result into a debug package based on the 'pkgbase' of the package/PKGBUILD so for linux it ends up creating linux-debug-$pkgver-$pkgrel.tar.zst.
* How do I verify if the debug package is generated correctly?
After installing the debug package, gdb $binary should reveal:
[jelle@t14s][~]%gdb /usr/bin/cryptsetup Reading symbols from /usr/bin/cryptsetup... Reading symbols from /usr/lib/debug/usr/bin/cryptsetup.debug...
Apart from that the debug package should not contain any other paths then /usr/lib/debug and /usr/src/debug. Ideally namcap would print a warning when this is not the case.
Hope this helps,
Jelle van der Waa
Thanks for the tl;dr rundown, Jelle. This is very, very easy to do. All who worked on this should be very proud - this new functionality is like magic.
participants (6)
-
Allan McRae
-
Brett Cornwall
-
Jelle van der Waa
-
Kristian Klausen
-
Leonidas Spyropoulos
-
Morten Linderud