[arch-general] debug package repositories again

newer
[arch-general] weechat installs to...

Joe Julian

12 Aug 2015 12 Aug '15

7:36 p.m.

It's been hashed and rehashed, but I need to bring this up again. Every other popular distro has a way of debugging without completely rebuilding a package, saving countless combined hours. Why can't we just have a way of installing the symbols? When I find a bug, I don't want to spend another hour building python, for instance. I want to install the symbols, open up gdb, and figure out where the problem is. I'll report it, probably fix it, and we'll all be happy. Allan has already added the features necessary to make this possible. Why is this so hard? What can I do to help make this a reality? https://lists.archlinux.org/pipermail/arch-dev-public/2013-April/024812.html https://lists.archlinux.org/pipermail/arch-general/2015-April/038794.html

Show replies by date

Leonid Isaev

13 Aug 13 Aug

12:02 a.m.

On Wed, Aug 12, 2015 at 12:36:22PM -0700, Joe Julian wrote:

...

It's been hashed and rehashed, but I need to bring this up again. Every other popular distro has a way of debugging without completely rebuilding a package, saving countless combined hours.

Why can't we just have a way of installing the symbols?

Not enough manpower...

...

Allan has already added the features necessary to make this possible.

Why is this so hard? What can I do to help make this a reality?

Well, there has to be a separate [debug] repo which (most importantly) must be kept in-sync with the main ones, or more packages in current repos. Then, someone has to actually build those packages. I would personally prefer that most packages come with debugging enabled by default. Surely, there will be a performance cost, but speed is not crucial in most cases. Cheers, -- Leonid Isaev GPG fingerprints: DA92 034D B4A8 EC51 7EA6 20DF 9291 EE8A 043C B8C4 C0DF 20D0 C075 C3F1 E1BE 775A A7AE F6CB 164B 5A6D

Joe Julian

12:09 a.m.

...

On Wed, Aug 12, 2015 at 12:36:22PM -0700, Joe Julian wrote:

...
It's been hashed and rehashed, but I need to bring this up again. Every other popular distro has a way of debugging without completely rebuilding a package, saving countless combined hours.

Why can't we just have a way of installing the symbols? Not enough manpower... There would be no more manpower needed than there is now. Set the debug

On 08/12/2015 05:02 PM, Leonid Isaev wrote: option to makepkg and put the debug package created in the debug repo.

...

...
Allan has already added the features necessary to make this possible.

Why is this so hard? What can I do to help make this a reality? Well, there has to be a separate [debug] repo which (most importantly) must be kept in-sync with the main ones, or more packages in current repos. Then, someone has to actually build those packages.

I would personally prefer that most packages come with debugging enabled by default. Surely, there will be a performance cost, but speed is not crucial in most cases.

Cheers,

There's no performance impact, just disk space and bandwidth.

Evan Penner

1:11 p.m.

On 08/12/2015 05:02 PM, Leonid Isaev wrote:

...

...
On Wed, Aug 12, 2015 at 12:36:22PM -0700, Joe Julian wrote:

...
It's been hashed and rehashed, but I need to bring this up again. Every other popular distro has a way of debugging without completely rebuilding a package, saving countless combined hours.

Why can't we just have a way of installing the symbols?

Not enough manpower...

There would be no more manpower needed than there is now. Set the debug option to makepkg and put the debug package created in the debug repo.

I'll submit a feature request on the bugtracker.

...

...
Allan has already added the features necessary to make this possible.

...
Why is this so hard? What can I do to help make this a reality?

Well, there has to be a separate [debug] repo which (most importantly) must be kept in-sync with the main ones, or more packages in current repos. Then, someone has to actually build those packages.

I would personally prefer that most packages come with debugging enabled by default. Surely, there will be a performance cost, but speed is not crucial in most cases.

Cheers,

There's no performance impact, just disk space and bandwidth.

Bandwidth is probably the main problem, although anyone who wants to debug will probably be fine with that.

Leonid Isaev

1:47 p.m.

On Thu, Aug 13, 2015 at 03:11:58PM +0200, Evan Penner wrote:

...

...
...
I would personally prefer that most packages come with debugging enabled by default. Surely, there will be a performance cost, but speed is not crucial in most cases.

Cheers,

There's no performance impact, just disk space and bandwidth.

Bandwidth is probably the main problem, although anyone who wants to debug will probably be fine with that.

I think you guys misunderstood me. The biggest problem IMHO with building debug versions locally is not compiling itself, but setting up the environment. So, I meant that packages come with debugging enabled (compiled with gcc -O0 -g and perhaps ./configure options). This way, there will be not many new packages. Of course, this is not a good idea for things like FF/Gnome/KDE because of a slow-down, but a performance penalty for smaller programs like vim, links, XFCE4 etc. will not be noticeable (at least I don't see any for a self-compiled xfce4 desktop on a single-core Intel Atom based netbook). Cheers, -- Leonid Isaev GPG fingerprints: DA92 034D B4A8 EC51 7EA6 20DF 9291 EE8A 043C B8C4 C0DF 20D0 C075 C3F1 E1BE 775A A7AE F6CB 164B 5A6D

Evan Penner

1:55 p.m.

On Thu, Aug 13, 2015 at 3:47 PM, Leonid Isaev wrote:

...

On Thu, Aug 13, 2015 at 03:11:58PM +0200, Evan Penner wrote:

...
...
...
I would personally prefer that most packages come with debugging enabled by default. Surely, there will be a performance cost, but speed is not crucial in most cases.

Cheers,

There's no performance impact, just disk space and bandwidth.

Bandwidth is probably the main problem, although anyone who wants to debug will probably be fine with that.

I think you guys misunderstood me. The biggest problem IMHO with building debug versions locally is not compiling itself, but setting up the environment. So, I meant that packages come with debugging enabled (compiled with gcc -O0 -g and perhaps ./configure options). This way, there will be not many new packages.

Of course, this is not a good idea for things like FF/Gnome/KDE because of a slow-down, but a performance penalty for smaller programs like vim, links, XFCE4 etc. will not be noticeable (at least I don't see any for a self-compiled xfce4 desktop on a single-core Intel Atom based netbook).

Right, I got it now. So ideally, you'd want all packages to come with debug flags enabled? I see how that would cause performance issues in larger applications, I understand however the concern for those packages. The problem is performance as you said. However, creating a new repository would increase the Arch database by twofold. This is a concept the devs will have to take a look at. For now, I guess, your only option is to compile them yourself.

Joe Julian

5:13 p.m.

On 08/13/2015 06:55 AM, Evan Penner wrote:

...

On Thu, Aug 13, 2015 at 3:47 PM, Leonid Isaev wrote:

...
...
...
...
I would personally prefer that most packages come with debugging enabled by default. Surely, there will be a performance cost, but speed is not crucial in most cases.

Cheers,

There's no performance impact, just disk space and bandwidth.

Bandwidth is probably the main problem, although anyone who wants to debug will probably be fine with that. I think you guys misunderstood me. The biggest problem IMHO with building debug versions locally is not compiling itself, but setting up the environment. So, I meant that packages come with debugging enabled (compiled with gcc -O0 -g and

On Thu, Aug 13, 2015 at 03:11:58PM +0200, Evan Penner wrote: perhaps ./configure options). This way, there will be not many new packages.

Of course, this is not a good idea for things like FF/Gnome/KDE because of a slow-down, but a performance penalty for smaller programs like vim, links, XFCE4 etc. will not be noticeable (at least I don't see any for a self-compiled xfce4 desktop on a single-core Intel Atom based netbook).

Right, I got it now. So ideally, you'd want all packages to come with debug flags enabled? I see how that would cause performance issues in larger applications,

False, see my previous reply.

...

I understand however the concern for those packages. The problem is performance as you said. However, creating a new repository would increase the Arch database by twofold.

False. Plus, the debug symbols *could* be in a separate package database that would be manually enabled by the people with the skills to know how to use them. People that actually use this distro professionally to build some really cool stuff (plugging IO, my employer, here).

...

This is a concept the devs will have to take a look at. For now, I guess, your only option is to compile them yourself.

Joe Julian

5:09 p.m.

On 08/13/2015 06:47 AM, Leonid Isaev wrote:

...

...
...
...
I would personally prefer that most packages come with debugging enabled by default. Surely, there will be a performance cost, but speed is not crucial in most cases.

Cheers,

There's no performance impact, just disk space and bandwidth.

Bandwidth is probably the main problem, although anyone who wants to debug will probably be fine with that. I think you guys misunderstood me. The biggest problem IMHO with building debug versions locally is not compiling itself, but setting up the environment. So, I meant that packages come with debugging enabled (compiled with gcc -O0 -g and

On Thu, Aug 13, 2015 at 03:11:58PM +0200, Evan Penner wrote: perhaps ./configure options). This way, there will be not many new packages.

No, the biggest problem with building debug versions locally is that it takes hours of developer time multiplied by every bug found. Why are we wasting such a limited resource when it's so easy not to? Life is finite.

...

Of course, this is not a good idea for things like FF/Gnome/KDE because of a slow-down, but a performance penalty for smaller programs like vim, links, XFCE4 etc. will not be noticeable (at least I don't see any for a self-compiled xfce4 desktop on a single-core Intel Atom based netbook).

Cheers,

What is this slow-down you keep talking about? I'm not asking to do away with optimization, just give us a way to get the debug symbols without rebuilding. The debug symbols are located in totally different sections from the code/data sections. You can check it with objdump: $ objdump -h a.out a.out: file format elf64-x86-64 Sections: Idx Name Size VMA LMA File off Algn 0 .interp 0000001c 0000000000400200 0000000000400200 00000200 2**0 CONTENTS, ALLOC, LOAD, READONLY, DATA 1 .note.ABI-tag 00000020 000000000040021c 000000000040021c 0000021c 2**2 CONTENTS, ALLOC, LOAD, READONLY, DATA 2 .note.gnu.build-id 00000024 000000000040023c 000000000040023c 0000023c 2**2 CONTENTS, ALLOC, LOAD, READONLY, DATA 3 .hash 00000018 0000000000400260 0000000000400260 00000260 2**3 CONTENTS, ALLOC, LOAD, READONLY, DATA 4 .gnu.hash 0000001c 0000000000400278 0000000000400278 00000278 2**3 CONTENTS, ALLOC, LOAD, READONLY, DATA 5 .dynsym 00000048 0000000000400298 0000000000400298 00000298 2**3 CONTENTS, ALLOC, LOAD, READONLY, DATA 6 .dynstr 00000038 00000000004002e0 00000000004002e0 000002e0 2**0 CONTENTS, ALLOC, LOAD, READONLY, DATA 7 .gnu.version 00000006 0000000000400318 0000000000400318 00000318 2**1 CONTENTS, ALLOC, LOAD, READONLY, DATA 8 .gnu.version_r 00000020 0000000000400320 0000000000400320 00000320 2**3 CONTENTS, ALLOC, LOAD, READONLY, DATA 9 .rela.dyn 00000018 0000000000400340 0000000000400340 00000340 2**3 CONTENTS, ALLOC, LOAD, READONLY, DATA 10 .rela.plt 00000018 0000000000400358 0000000000400358 00000358 2**3 CONTENTS, ALLOC, LOAD, READONLY, DATA 11 .init 00000018 0000000000400370 0000000000400370 00000370 2**2 CONTENTS, ALLOC, LOAD, READONLY, CODE 12 .plt 00000020 0000000000400388 0000000000400388 00000388 2**2 CONTENTS, ALLOC, LOAD, READONLY, CODE 13 .text 000001c8 00000000004003b0 00000000004003b0 000003b0 2**4 CONTENTS, ALLOC, LOAD, READONLY, CODE 14 .fini 0000000e 0000000000400578 0000000000400578 00000578 2**2 CONTENTS, ALLOC, LOAD, READONLY, CODE 15 .rodata 00000004 0000000000400588 0000000000400588 00000588 2**2 CONTENTS, ALLOC, LOAD, READONLY, DATA 16 .eh_frame_hdr 00000024 000000000040058c 000000000040058c 0000058c 2**2 CONTENTS, ALLOC, LOAD, READONLY, DATA 17 .eh_frame 0000007c 00000000004005b0 00000000004005b0 000005b0 2**3 CONTENTS, ALLOC, LOAD, READONLY, DATA 18 .ctors 00000010 0000000000600630 0000000000600630 00000630 2**3 CONTENTS, ALLOC, LOAD, DATA 19 .dtors 00000010 0000000000600640 0000000000600640 00000640 2**3 CONTENTS, ALLOC, LOAD, DATA 20 .jcr 00000008 0000000000600650 0000000000600650 00000650 2**3 CONTENTS, ALLOC, LOAD, DATA 21 .dynamic 000001a0 0000000000600658 0000000000600658 00000658 2**3 CONTENTS, ALLOC, LOAD, DATA 22 .got 00000008 00000000006007f8 00000000006007f8 000007f8 2**3 CONTENTS, ALLOC, LOAD, DATA 23 .got.plt 00000020 0000000000600800 0000000000600800 00000800 2**3 CONTENTS, ALLOC, LOAD, DATA 24 .data 00000010 0000000000600820 0000000000600820 00000820 2**3 CONTENTS, ALLOC, LOAD, DATA 25 .bss 00000010 0000000000600830 0000000000600830 00000830 2**3 ALLOC 26 .comment 00000039 0000000000000000 0000000000000000 00000830 2**0 CONTENTS, READONLY 27 .debug_aranges 00000030 0000000000000000 0000000000000000 00000869 2**0 CONTENTS, READONLY, DEBUGGING 28 .debug_pubnames 0000001b 0000000000000000 0000000000000000 00000899 2**0 CONTENTS, READONLY, DEBUGGING 29 .debug_info 00000055 0000000000000000 0000000000000000 000008b4 2**0 CONTENTS, READONLY, DEBUGGING 30 .debug_abbrev 00000034 0000000000000000 0000000000000000 00000909 2**0 CONTENTS, READONLY, DEBUGGING 31 .debug_line 0000003b 0000000000000000 0000000000000000 0000093d 2**0 CONTENTS, READONLY, DEBUGGING 32 .debug_str 00000026 0000000000000000 0000000000000000 00000978 2**0 CONTENTS, READONLY, DEBUGGING 33 .debug_loc 0000004c 0000000000000000 0000000000000000 0000099e 2**0 CONTENTS, READONLY, DEBUGGING You can see the extra sections (27 through 33). These sections won't be loaded at runtime, so there won't be any performance penalty. Using gdb, you can also examine them at runtime $ gdb ./a.out (gdb) break main (gdb) run (gdb) info files // blah blah .... Local exec file: `/home/kghost/a.out', file type elf64-x86-64. Entry point: 0x4003b0 0x0000000000400200 - 0x000000000040021c is .interp 0x000000000040021c - 0x000000000040023c is .note.ABI-tag 0x000000000040023c - 0x0000000000400260 is .note.gnu.build-id 0x0000000000400260 - 0x0000000000400278 is .hash 0x0000000000400278 - 0x0000000000400294 is .gnu.hash 0x0000000000400298 - 0x00000000004002e0 is .dynsym 0x00000000004002e0 - 0x0000000000400318 is .dynstr 0x0000000000400318 - 0x000000000040031e is .gnu.version 0x0000000000400320 - 0x0000000000400340 is .gnu.version_r 0x0000000000400340 - 0x0000000000400358 is .rela.dyn 0x0000000000400358 - 0x0000000000400370 is .rela.plt 0x0000000000400370 - 0x0000000000400388 is .init 0x0000000000400388 - 0x00000000004003a8 is .plt 0x00000000004003b0 - 0x0000000000400578 is .text 0x0000000000400578 - 0x0000000000400586 is .fini 0x0000000000400588 - 0x000000000040058c is .rodata 0x000000000040058c - 0x00000000004005b0 is .eh_frame_hdr 0x00000000004005b0 - 0x000000000040062c is .eh_frame 0x0000000000600630 - 0x0000000000600640 is .ctors 0x0000000000600640 - 0x0000000000600650 is .dtors 0x0000000000600650 - 0x0000000000600658 is .jcr 0x0000000000600658 - 0x00000000006007f8 is .dynamic 0x00000000006007f8 - 0x0000000000600800 is .got 0x0000000000600800 - 0x0000000000600820 is .got.plt 0x0000000000600820 - 0x0000000000600830 is .data 0x0000000000600830 - 0x0000000000600840 is .bss // blah blah .... So the only penalty is that you need extra disk space to store this information.

Leonid Isaev

5:58 p.m.

On Thu, Aug 13, 2015 at 10:09:00AM -0700, Joe Julian wrote:

...

...
...
Bandwidth is probably the main problem, although anyone who wants to debug will probably be fine with that. I think you guys misunderstood me. The biggest problem IMHO with building debug versions locally is not compiling itself, but setting up the environment. So, I meant that packages come with debugging enabled (compiled with gcc -O0 -g and perhaps ./configure options). This way, there will be not many new packages.

No, the biggest problem with building debug versions locally is that it takes hours of developer time multiplied by every bug found. Why are we wasting such a limited resource when it's so easy not to? Life is finite.

That's just pure theory. In practice, the compilation itself is a minor inconvenience, unless you are talking about Gnome/KDE. But debugging those is hopeless anyway :) As a developer, you'll spend most time understanding the changes (looking at the code), not compiling.

...

...
Of course, this is not a good idea for things like FF/Gnome/KDE because of a slow-down, but a performance penalty for smaller programs like vim, links, XFCE4 etc. will not be noticeable (at least I don't see any for a self-compiled xfce4 desktop on a single-core Intel Atom based netbook).

Cheers,

What is this slow-down you keep talking about? I'm not asking to do away with optimization, just give us a way to get the debug symbols without rebuilding. The debug symbols are located in totally different sections from the code/data sections. You can check it with objdump:

I was under the impression that with C, -On (n > 0) is not recommended with -g. Now, I don't know how much -O2 (for example) speeds things up compared to -O1 etc, but probably not much on small applications. Granted, I think any compiler-level optimization is overrated, and I never really saw any measurable effect of it. But I use Fortran (not C) for all my projects. Cheers, -- Leonid Isaev GPG fingerprints: DA92 034D B4A8 EC51 7EA6 20DF 9291 EE8A 043C B8C4 C0DF 20D0 C075 C3F1 E1BE 775A A7AE F6CB 164B 5A6D

Joe Julian

6:09 p.m.

On 08/13/2015 10:58 AM, Leonid Isaev wrote:

...

On Thu, Aug 13, 2015 at 10:09:00AM -0700, Joe Julian wrote:

...
...
...
Bandwidth is probably the main problem, although anyone who wants to debug will probably be fine with that. I think you guys misunderstood me. The biggest problem IMHO with building debug versions locally is not compiling itself, but setting up the environment. So, I meant that packages come with debugging enabled (compiled with gcc -O0 -g and perhaps ./configure options). This way, there will be not many new packages. No, the biggest problem with building debug versions locally is that it takes hours of developer time multiplied by every bug found. Why are we wasting such a limited resource when it's so easy not to? Life is finite. That's just pure theory. In practice, the compilation itself is a minor inconvenience, unless you are talking about Gnome/KDE. But debugging those is hopeless anyway :)

As a developer, you'll spend most time understanding the changes (looking at the code), not compiling.

I'm not some newb, maybe google me before you make condescending statements. There is a reason the other distros publish the debug symbols. It's valuable and there's no valid reason not to.

...

...
...
Of course, this is not a good idea for things like FF/Gnome/KDE because of a slow-down, but a performance penalty for smaller programs like vim, links, XFCE4 etc. will not be noticeable (at least I don't see any for a self-compiled xfce4 desktop on a single-core Intel Atom based netbook).

Cheers, What is this slow-down you keep talking about? I'm not asking to do away with optimization, just give us a way to get the debug symbols without rebuilding. The debug symbols are located in totally different sections from the code/data sections. You can check it with objdump: I was under the impression that with C, -On (n > 0) is not recommended with -g. Now, I don't know how much -O2 (for example) speeds things up compared to -O1 etc, but probably not much on small applications.

Granted, I think any compiler-level optimization is overrated, and I never really saw any measurable effect of it. But I use Fortran (not C) for all my projects.

Cheers,

Yes, if you optimize you will lose some variable data when it's out of scope. If it's actually about debugging and seeing what's in variables and stepping through code, that's something that I would likely compile on my own with O0. Most of the time it's about seeing where the deadlock or segfault is and getting a meaningful stack trace. I've seen quite a bit of a performance increase in some cases. GlusterFS was being built with O0 back in the 2.0 days. Building with O2 proved about a 30% increase in throughput.

Doug Newgard

7:37 p.m.

On Thu, 13 Aug 2015 11:09:46 -0700 Joe Julian wrote:

...

There is a reason the other distros publish the debug symbols. It's valuable and there's no valid reason not to.

AFAIK, right now the reason it can't happen is because nobody has stepped up and added support to devtools for putting split packages in different repos. Nobody willing to do the work is a valid reason. Doug

Joe Julian

8:08 p.m.

On 08/13/2015 12:37 PM, Doug Newgard wrote:

...

On Thu, 13 Aug 2015 11:09:46 -0700 Joe Julian wrote:

...
There is a reason the other distros publish the debug symbols. It's valuable and there's no valid reason not to. AFAIK, right now the reason it can't happen is because nobody has stepped up and added support to devtools for putting split packages in different repos. Nobody willing to do the work is a valid reason.

Doug

Excellent. I'll tackle that then.

Damian Nowak

9:06 p.m.

...

...
AFAIK, right now the reason it can't happen is because nobody has stepped up and added support to devtools for putting split packages in different repos. Nobody willing to do the work is a valid reason.

...

Excellent. I'll tackle that then.

Good luck! Let us know about a progress. Or maybe you can create an issue in the bug tracker and track the progress there? Anyways, would be great to have debug symbols provided at some point. Debug symbols would have saved me a lot of time with reporting problems to Mesa/Gallium3D bugtracker. -- Damian Nowak

Florian Pritz

15 Aug 15 Aug

1:42 p.m.

On Thu, 13 Aug 2015 13:08:55 -0700 Joe Julian wrote:

...

On 08/13/2015 12:37 PM, Doug Newgard wrote:

...
AFAIK, right now the reason it can't happen is because nobody has stepped up and added support to devtools for putting split packages in different repos. Nobody willing to do the work is a valid reason.

Excellent. I'll tackle that then.

dbscripts also needs to be changed and I've started working on this quite some time ago, but then had other things to do and I haven't picked it up again since because the code is not pretty and it requires a bit of time to get up to speed. If you really, really want to work on this feel free to ping me on IRC (Bluewind in #archlinux-projects would be best) so we can discuss it in detail. My current progress can be seen here, though I'm not sure if all the information there is still accurate: https://wiki.archlinux.org/index.php/User:Bluewind/dbscripts-rewrite Florian

3180

Age (days ago)

3183

Last active (days ago)

List overview

Download

13 comments

6 participants

participants (6)

Damian Nowak
Doug Newgard
Evan Penner
Florian Pritz
Joe Julian
Leonid Isaev