[arch-dev-public] [PATCH 1/2] makepkg.conf: Change default compression to .zst (fast)
From: "Jan Alexander Steffens (heftig)" <heftig@archlinux.org> Most people create packages from the AUR for local installation. This allows these packages to be created more quickly. --- PKGBUILD | 2 +- makepkg.conf | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/PKGBUILD b/PKGBUILD index 8566270..846a970 100644 --- a/PKGBUILD +++ b/PKGBUILD @@ -27,7 +27,7 @@ source=(https://sources.archlinux.org/other/pacman/$pkgname-$pkgver.tar.gz{,.sig sha256sums=('bb201a9f2fb53c28d011f661d50028efce6eef2c1d2a36728bdd0130189349a0' 'SKIP' '3353f363088c73f1f86a890547c0f87c7473e5caf43bbbc768c2e9a7397f2aa2' - '9c769f13c09a6f24c393a9762474eded2f269d8966e7764d9160d62232a7919b') + 'd113252f97f019a13541237a4f4c7fbe9ffd0c3e71ecd7cd8d5d227b378819ab') build() { diff --git a/makepkg.conf b/makepkg.conf index d8bf59e..a277503 100644 --- a/makepkg.conf +++ b/makepkg.conf @@ -132,7 +132,7 @@ DBGSRCDIR="/usr/src/debug" COMPRESSGZ=(gzip -c -f -n) COMPRESSBZ2=(bzip2 -c -f) COMPRESSXZ=(xz -c -z -) -COMPRESSZST=(zstd -c -z -q -) +COMPRESSZST=(zstd -c -T0 -) COMPRESSLRZ=(lrzip -q) COMPRESSLZO=(lzop -q) COMPRESSZ=(compress -c -f) -- 2.27.0
From: "Jan Alexander Steffens (heftig)" <heftig@archlinux.org> I recently read [Fedora's documentation on build flags][1] and I think they have some useful ideas. 1. Move -D_FORTIFY_SOURCE=2 from CPPFLAGS to CFLAGS using -Wp: Unfortunately, there are still build systems (e.g. CMake, homegrown Makefile rules) which use CFLAGS but not CPPFLAGS. Ultimately, we can cover more code with this workaround. 2. -fexceptions: Slight hardening of C programs making use of automatic variable cleanup or pthread_cancel. Cost should be negligible. 3. -fstack-clash-protection: Hardening of large stack allocations. Cost should be negigible. We need to patch clang to ignore this, like we once did for -fno-plt. 4. -fcf-protection: Hardening which makes code compatible with Intel CET. Increases code size a bit but cost should be negligible. No processors supporting it are available yet, but the linker only marks binaries for CET when all code is compatible, so we could get a head-start on this. 5. -fasynchronous-unwind-tables: Generates DWARF unwinding information that doesn't get stripped. Increases binary size a bit. Should make sure tools like perf and gdb can unwind the stack completely even without debug symbols. This makes the debugger more useful if you only have debug symbols for some frames, since frames without symbols can no longer break unwinding. 6. -Wp,-D_GLIBCXX_ASSERTIONS: Enables some assertions in libstdc++. Hardening similar to _FORTIFY_SOURCE. 7. -grecord-gcc-switches: Useful information to record. But since we don't use `debug` yet, won't affect us much. [1]: https://src.fedoraproject.org/rpms/redhat-rpm-config/blob/master/f/buildflag... --- PKGBUILD | 2 +- makepkg.conf | 12 +++++++----- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/PKGBUILD b/PKGBUILD index 846a970..ed1d492 100644 --- a/PKGBUILD +++ b/PKGBUILD @@ -27,7 +27,7 @@ source=(https://sources.archlinux.org/other/pacman/$pkgname-$pkgver.tar.gz{,.sig sha256sums=('bb201a9f2fb53c28d011f661d50028efce6eef2c1d2a36728bdd0130189349a0' 'SKIP' '3353f363088c73f1f86a890547c0f87c7473e5caf43bbbc768c2e9a7397f2aa2' - 'd113252f97f019a13541237a4f4c7fbe9ffd0c3e71ecd7cd8d5d227b378819ab') + '3818559af64c11d9cda127ae75e48e5f8780bbe71513f5a3c484c38eb16a2b71') build() { diff --git a/makepkg.conf b/makepkg.conf index a277503..c8c917e 100644 --- a/makepkg.conf +++ b/makepkg.conf @@ -36,16 +36,18 @@ CARCH="x86_64" CHOST="x86_64-pc-linux-gnu" #-- Compiler and Linker Flags -CPPFLAGS="-D_FORTIFY_SOURCE=2" -CFLAGS="-march=x86-64 -mtune=generic -O2 -pipe -fno-plt" -CXXFLAGS="-march=x86-64 -mtune=generic -O2 -pipe -fno-plt" +#CPPFLAGS="" +CFLAGS="-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions \ + -fstack-clash-protection -fcf-protection -fasynchronous-unwind-tables \ + -Wp,-D_FORTIFY_SOURCE=2,-D_GLIBCXX_ASSERTIONS" +CXXFLAGS="$CFLAGS" LDFLAGS="-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now" #RUSTFLAGS="-C opt-level=2" #-- Make Flags: change this for DistCC/SMP systems #MAKEFLAGS="-j2" #-- Debugging flags -DEBUG_CFLAGS="-g -fvar-tracking-assignments" -DEBUG_CXXFLAGS="-g -fvar-tracking-assignments" +DEBUG_CFLAGS="-g -grecord-gcc-switches -fvar-tracking-assignments" +DEBUG_CXXFLAGS="-g -grecord-gcc-switches -fvar-tracking-assignments" #DEBUG_RUSTFLAGS="-C debuginfo=2" ######################################################################### -- 2.27.0
On 7/10/20 2:38 PM, Jan Alexander Steffens (heftig) via arch-dev-public wrote:
From: "Jan Alexander Steffens (heftig)" <heftig@archlinux.org>
I recently read [Fedora's documentation on build flags][1] and I think they have some useful ideas.
1. Move -D_FORTIFY_SOURCE=2 from CPPFLAGS to CFLAGS using -Wp: Unfortunately, there are still build systems (e.g. CMake, homegrown Makefile rules) which use CFLAGS but not CPPFLAGS. Ultimately, we can cover more code with this workaround.
Sounds like a job for build() { export CFLAGS="$CPPFLAGS $CFLAGS" ... } (I do not understand how -Wp, helps here, its purpose is only to prevent the compiler driver from reinterpreting it before passing it to the preprocessor, and only if you have special needs and believe it will mangle your flags. -D_FORTIFY_SOURCE sounds sufficiently boring to say it won't be mangled.) Our cmake build already solves this with: https://git.archlinux.org/svntogit/packages.git/tree/trunk/cmake-cppflags.pa...
2. -fexceptions: Slight hardening of C programs making use of automatic variable cleanup or pthread_cancel. Cost should be negligible.
3. -fstack-clash-protection: Hardening of large stack allocations. Cost should be negigible.
We need to patch clang to ignore this, like we once did for -fno-plt.
4. -fcf-protection: Hardening which makes code compatible with Intel CET. Increases code size a bit but cost should be negligible.
No processors supporting it are available yet, but the linker only marks binaries for CET when all code is compatible, so we could get a head-start on this.
5. -fasynchronous-unwind-tables: Generates DWARF unwinding information that doesn't get stripped. Increases binary size a bit.
Should make sure tools like perf and gdb can unwind the stack completely even without debug symbols. This makes the debugger more useful if you only have debug symbols for some frames, since frames without symbols can no longer break unwinding.
If I can finish splitdebug package support in dbscripts...
6. -Wp,-D_GLIBCXX_ASSERTIONS: Enables some assertions in libstdc++. Hardening similar to _FORTIFY_SOURCE.
7. -grecord-gcc-switches: Useful information to record. But since we don't use `debug` yet, won't affect us much.
I wanted to add this to .BUILDINFO based on the contents of makepkg.conf TBH. It would work independent of 'debug'.
[1]: https://src.fedoraproject.org/rpms/redhat-rpm-config/blob/master/f/buildflag... --- PKGBUILD | 2 +- makepkg.conf | 12 +++++++----- 2 files changed, 8 insertions(+), 6 deletions(-)
diff --git a/PKGBUILD b/PKGBUILD index 846a970..ed1d492 100644 --- a/PKGBUILD +++ b/PKGBUILD @@ -27,7 +27,7 @@ source=(https://sources.archlinux.org/other/pacman/$pkgname-$pkgver.tar.gz{,.sig sha256sums=('bb201a9f2fb53c28d011f661d50028efce6eef2c1d2a36728bdd0130189349a0' 'SKIP' '3353f363088c73f1f86a890547c0f87c7473e5caf43bbbc768c2e9a7397f2aa2' - 'd113252f97f019a13541237a4f4c7fbe9ffd0c3e71ecd7cd8d5d227b378819ab') + '3818559af64c11d9cda127ae75e48e5f8780bbe71513f5a3c484c38eb16a2b71')
build() { diff --git a/makepkg.conf b/makepkg.conf index a277503..c8c917e 100644 --- a/makepkg.conf +++ b/makepkg.conf @@ -36,16 +36,18 @@ CARCH="x86_64" CHOST="x86_64-pc-linux-gnu"
#-- Compiler and Linker Flags -CPPFLAGS="-D_FORTIFY_SOURCE=2" -CFLAGS="-march=x86-64 -mtune=generic -O2 -pipe -fno-plt" -CXXFLAGS="-march=x86-64 -mtune=generic -O2 -pipe -fno-plt" +#CPPFLAGS="" +CFLAGS="-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions \ + -fstack-clash-protection -fcf-protection -fasynchronous-unwind-tables \ + -Wp,-D_FORTIFY_SOURCE=2,-D_GLIBCXX_ASSERTIONS" +CXXFLAGS="$CFLAGS" LDFLAGS="-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now" #RUSTFLAGS="-C opt-level=2" #-- Make Flags: change this for DistCC/SMP systems #MAKEFLAGS="-j2" #-- Debugging flags -DEBUG_CFLAGS="-g -fvar-tracking-assignments" -DEBUG_CXXFLAGS="-g -fvar-tracking-assignments" +DEBUG_CFLAGS="-g -grecord-gcc-switches -fvar-tracking-assignments" +DEBUG_CXXFLAGS="-g -grecord-gcc-switches -fvar-tracking-assignments" #DEBUG_RUSTFLAGS="-C debuginfo=2"
#########################################################################
-- Eli Schwartz Bug Wrangler and Trusted User
On Fri, 10 Jul 2020 at 21:45, Jan Alexander Steffens (heftig) via arch-dev-public <arch-dev-public@archlinux.org> wrote:
3. -fstack-clash-protection: Hardening of large stack allocations. Cost should be negigible.
We need to patch clang to ignore this, like we once did for -fno-plt.
Apparently, Fedora didn't think it was a good idea to make this a no-op. [1] The good news is it will be in Clang 11 (out in two-ish months?). Definitely worth waiting. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1658311
On 7/10/20 2:38 PM, Jan Alexander Steffens (heftig) via arch-dev-public wrote:
From: "Jan Alexander Steffens (heftig)" <heftig@archlinux.org>
Most people create packages from the AUR for local installation. This allows these packages to be created more quickly.
This doesn't switch to .zst (we already set this to .zst), only COMPRESSZST. Anyway people creating packages from the AUR for local installation should use PKGEXT=.pkg.tar to disable compression, as this is instant, pacman -U can install it faster because it doesn't need to decompress, and it doesn't take up more disk space if the user will rm the package after installing anyway...
--- PKGBUILD | 2 +- makepkg.conf | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/PKGBUILD b/PKGBUILD index 8566270..846a970 100644 --- a/PKGBUILD +++ b/PKGBUILD @@ -27,7 +27,7 @@ source=(https://sources.archlinux.org/other/pacman/$pkgname-$pkgver.tar.gz{,.sig sha256sums=('bb201a9f2fb53c28d011f661d50028efce6eef2c1d2a36728bdd0130189349a0' 'SKIP' '3353f363088c73f1f86a890547c0f87c7473e5caf43bbbc768c2e9a7397f2aa2' - '9c769f13c09a6f24c393a9762474eded2f269d8966e7764d9160d62232a7919b') + 'd113252f97f019a13541237a4f4c7fbe9ffd0c3e71ecd7cd8d5d227b378819ab')
build() { diff --git a/makepkg.conf b/makepkg.conf index d8bf59e..a277503 100644 --- a/makepkg.conf +++ b/makepkg.conf @@ -132,7 +132,7 @@ DBGSRCDIR="/usr/src/debug" COMPRESSGZ=(gzip -c -f -n) COMPRESSBZ2=(bzip2 -c -f) COMPRESSXZ=(xz -c -z -) -COMPRESSZST=(zstd -c -z -q -) +COMPRESSZST=(zstd -c -T0 -) COMPRESSLRZ=(lrzip -q) COMPRESSLZO=(lzop -q) COMPRESSZ=(compress -c -f)
-- Eli Schwartz Bug Wrangler and Trusted User
participants (3)
-
Eli Schwartz
-
Evangelos Foutras
-
Jan Alexander Steffens (heftig)