- Arch-devops - lists.archlinux.org

Re: [arch-devops] [arch-projects] [dbscripts] [PATCH 2/4] Add reproducible archive of packages.
by Eli Schwartz 20 Dec '18

20 Dec '18

On 12/4/18 1:09 PM, Eli Schwartz wrote: > Whenever adding new package files to the pool of distributed packages, > hardlink a copy of every package it was built with, into a > "reproducible" pool, and log which file required it. The question becomes, where can I store these? As-is, this will burden the mirror network as well. Unsure how to handle this. Could this be configurable by the mirror, as ISOs are now? Should we exclusively self-host this, and if so, where? archive.archlinux.org is managed by another service with its own exclusively writable location. > Signed-off-by: Eli Schwartz <eschwartz(a)archlinux.org> > --- > config | 1 + > config.local.svn-community | 1 + > config.local.svn-packages | 1 + > db-functions | 49 +++++++++++++++++++++++++++++++------- > db-update | 4 ++++ > 5 files changed, 48 insertions(+), 8 deletions(-) > > diff --git a/config b/config > index 1cfe11f4..5144fca7 100644 > --- a/config > +++ b/config > @@ -3,6 +3,7 @@ > FTP_BASE="/srv/ftp" > PKGREPOS=() > PKGPOOL='' > +EXTRA_PKGPOOLS=() > SRCPOOL='' > TESTING_REPO='' > STABLE_REPOS=() > diff --git a/config.local.svn-community b/config.local.svn-community > index 5d61b5ea..15bcc17f 100644 > --- a/config.local.svn-community > +++ b/config.local.svn-community > @@ -2,6 +2,7 @@ > > PKGREPOS=('community' 'community-testing' 'community-staging' 'multilib' 'multilib-testing' 'multilib-staging') > PKGPOOL='pool/community' > +EXTRA_PKGPOOLS=('pool/packages') > SRCPOOL='sources/community' > SVNREPO='file:///srv/repos/svn-community/svn' > SVNUSER='svn-community' > diff --git a/config.local.svn-packages b/config.local.svn-packages > index 34aab35c..75986b65 100644 > --- a/config.local.svn-packages > +++ b/config.local.svn-packages > @@ -2,6 +2,7 @@ > > PKGREPOS=('core' 'extra' 'testing' 'staging' 'kde-unstable' 'gnome-unstable') > PKGPOOL='pool/packages' > +EXTRA_PKGPOOLS=('pool/community') > SRCPOOL='sources/packages' > SVNREPO='file:///srv/repos/svn-packages/svn' > SVNUSER='svn-packages' > diff --git a/db-functions b/db-functions > index 7aeedced..2b1ae87a 100644 > --- a/db-functions > +++ b/db-functions > @@ -165,20 +165,23 @@ repo_unlock () { #repo_unlock <repo-name> <arch> > fi > } > > +# usage: _grep_all_info pkgfile infofile key > +_grep_all_info() { > + local _ret=() > + > + mapfile -t _ret < <(/usr/bin/bsdtar -xOqf "$1" "${2}" | grep "^${3} = ") > + > + printf '%s\n' "${_ret[@]#${3} = }" > +} > + > # usage: _grep_pkginfo pkgfile pattern > _grep_pkginfo() { > - local _ret > - > - _ret="$(/usr/bin/bsdtar -xOqf "$1" .PKGINFO | grep "^${2} = " | tail -1)" > - echo "${_ret#${2} = }" > + _grep_all_info "${1}" .PKGINFO "${2}" | tail -1 > } > > # usage: _grep_buildinfo pkgfile pattern > _grep_buildinfo() { > - local _ret > - > - _ret="$(/usr/bin/bsdtar -xOqf "$1" .BUILDINFO | grep "^${2} = " | tail -1)" > - echo "${_ret#${2} = }" > + _grep_all_info "${1}" .BUILDINFO "${2}" | tail -1 > } > > # Get the package base or name as fallback > @@ -444,4 +447,34 @@ arch_repo_modify() { > REPO_MODIFIED=1 > } > > +# Build an index of dependent packages needed by a given pkgfile > +# usage: make_reproducible pkgfile [check] > +make_reproducible() { > + local pkg dir pkgs=() pkgfile pkgfiles=() > + > + mapfile -t pkgs < <(_grep_all_info "${1}" .BUILDINFO installed) > + > + for pkg in "${pkgs[@]}"; do > + for dir in "${FTP_BASE}/${PKGPOOL}" "${EXTRA_PKGPOOLS[@]/#/${FTP_BASE}/}" "${STAGING}"/**/; do > + if pkgfile="$(getpkgfile "${dir}/${pkg}"${PKGEXTS} 2>/dev/null)"; then > + pkgfiles+=("${pkgfile}") > + continue 2 > + fi > + done > + error "could not find existing package for %s" "${pkg}" > + return 1 > + done > + > + if [[ ${2} = check ]]; then > + return 0 > + fi > + > + for pkg in "${pkgfiles[@]}"; do > + if [[ ! -f ${FTP_BASE}/${PKGPOOL}-reproducible/${pkg##*/} ]]; then > + ln -L "${pkg}" "${FTP_BASE}/${PKGPOOL}-reproducible/${pkg##*/}" > + fi > + echo "${1}" >> "${FTP_BASE}/${PKGPOOL}-reproducible/${pkg##*/}.buildlinks" > + done > +} > + > . "$(dirname "$(readlink -e "${BASH_SOURCE[0]}")")/db-functions-${VCS}" > diff --git a/db-update b/db-update > index 313fb999..11ec185f 100755 > --- a/db-update > +++ b/db-update > @@ -61,6 +61,9 @@ for repo in "${repos[@]}"; do > if ! check_builddir "${pkg}"; then > die "Package %s was not built in a chroot" "$repo/${pkg##*/}" > fi > + if ! make_reproducible "${pkg}" "check"; then > + die "Package %s is not reproducible" "${pkg}" > + fi > done > if ! check_splitpkgs "${repo}" "${pkgs[@]}"; then > die "Missing split packages for %s" "$repo" > @@ -82,6 +85,7 @@ for repo in "${repos[@]}"; do > # any packages might have been moved by the previous run > if [[ -f ${pkg} ]]; then > mv "${pkg}" "$FTP_BASE/${PKGPOOL}" > + make_reproducible "${FTP_BASE}/${PKGPOOL}${pkg##*/}" > fi > ln -s "../../../${PKGPOOL}/${pkgfile}" "$FTP_BASE/$repo/os/${pkgarch}" > # also move signatures > -- Eli Schwartz Bug Wrangler and Trusted User

3 4

[arch-devops] Switch wiki git repo from tarball based updates to git merges?
by Florian Pritz 05 Nov '18

05 Nov '18

Hi, AFAICT, our wiki repo has been created with an extracted tarball and updated by just extracting the new tarballs on top of it and committing the result. Due to that we've once lost a change (a2bbd243c85ea0e425ee3e8c380aba9f254cee61) when an update had been commited (7c2a4c95e1d96bc56d00fd133fa537acc1346f5b). I've noticed that we have this open pull request[1] and since the Vector skin is part of the upstream tarball I'm worried that if I were to apply the change, it would again be lost with the next update. [1] https://github.com/archlinux/archwiki/pull/18 I've thus prepared a local branch where I merged our repo with the upstream repo. That means that we can update by merging the upstream tags and thus git can track our changes we get merge conflicts when necessary. The only issue I see with this setup is that we also pull in upstream tags, but those obviously refer to the upstream versions which do not include our changes and thus won't work. We could disable tag fetching, but this has to be done each time someone configures the upstream repo in their local clone and it can't be done automatically. We could either ship a small script that configures the remote correctly, but it has to be executed manually by the user, or we just live with the tags and prefix our own tags with 'archwiki-' or similar in the future. We could also stay with the tarball update method, but I think that's unnecessarily complicated and error prone. Thoughts? Florian

2 2

[arch-devops] Build tools and repo management
by Dumont, Joey 01 Nov '18

01 Nov '18

Hi everyone! Our group is thinking of maintaining a subset of the msys2 repo to be used internally. I am wondering if there is documentation on the Arch Linux side on what tools you are currently using to automatically build the packages in the officially supported repos. I believe there must be some kind of dependency management as well, since official repos are never cursed by soname bump issues. Is there documentation on this as well? Specifically, is there a tool similar to apt-mirror? Thank you, and please don't hesistate to ask questions if I didn't make myself clear. Joey Dumont Scientific Computing Technical Advisor National Research Council Canada joey.dumont(a)nrc-cnrc.gc.ca<mailto:joey.dumont@nrc-cnrc.gc.ca>

3 3

[arch-devops] Mirrorbrain
by Phillip Smith 24 Sep '18

24 Sep '18

It's been discussed in the past, but I thought it worth having a semi-formal and documented discussion about it. Do we think setting up mirrorbrain[0] would be a worthwhile service? For those not familiar, the elevator pitch is that it's an open-source download redirector to coordinate a simple CDN. It is used by a number of open-source projects such as vlc, Gnome, KDE, LibreOffice and OpenSUSE. Architecturally speaking, it maintains a list of our mirrors and local copy of our repos, and monitors those mirrors for availability and staleness. HTTP(s) clients (ie, pacman) are then redirected to an appropriate mirror based on the clients geographic location and mirror health. Benefits include: 1. Ensuring users always receive up-to-date packages (mirrorbrain won't redirect to a mirror if that mirrors version of that package is outdated compared to the authoritative repository). 2. Automated monitoring of our mirror network to proactively detect stale or broken mirrors. Mirrorlist files (ie, /etc/pacman.d/mirrorlist) can be automatically generated based on MirrorBrain's data. 3. Reduced load on core mirrors, and load-balancing (for example, a 1Gbit mirror can be weighted to receive 10x the traffic of a 100MBps mirror). 4. Automatic MetaLink and Torrent file generation, with web-seeds (currently handled by hefur on luna?). Exceptions to redirection can be applied, for example to ensure security-sensitive files (checksum files perhaps) are always served directly from the authoritative repo. Requirements are pretty basic; apache with mod_asn, postgresql, and some python+perl modules. It can be run behind a reverse-proxy if we wanted to hide apache behind nginx. It's not a particular fast-moving project (last release 2014) but that's a reflection of it's stability IMHO. What do others think? I'm happy to take on the implementation project. Cheers, ~p [0] http://mirrorbrain.org/

4 5

[arch-devops] Centralized log monitoring and alerting?
by Andrew Crerar 11 Sep '18

11 Sep '18

Hi all, On 2018-07-28 there were some discussions in #archlinux-devops around setting up some sort of centralized logging/monitoring/alerting solution for the various services on Apollo (and maybe other?) server(s). I had mentioned possibly using the ELK[1] stack for this task. There was some back and forth about it potentially being a bit heavy handed for what was needed and how we would most likely need to repurpose/dedicate something like nymeria to handle the stack. There was also the suggestion of possibly using something like tenshi[2] if we're aiming for a low overhead solution, however, there would be much writing of the regexes. With that being said, the purpose of this email is to have a more formal discussion around what we're trying to capture from the logs, the actions we want to have taken with what ends up being captured, and possibly come to a consensus on what tool(s) we could leverage. Thoughts? Regards, Andrew [1] https://www.elastic.co/de/elk-stack [2]https://github.com/inversepath/tenshi

6 5

[arch-devops] Shutting down dragon (backup server)
by Florian Pritz 31 Aug '18

31 Aug '18

Hi, I'll turn off dragon.archlinux.org in 2 weeks unless someone objects. It used to host our backups, but these have since moved to vostok.archlinux.org. Evangelos and me checked if there are any important files left on there, but we couldn't find any. If someone else wants to check, you'll have two weeks to speak up. Otherwise I'll null the disks and cancel the contract. Florian

1 1

[arch-devops] Github Domain Verification
by Giancarlo Razzolini 10 Aug '18

10 Aug '18

Hi All, I have added verification for our organization's domain on github [0]. This is an extra security feature that shows a "verified" badge [1] on our organization's profile. This is to help avoid impersonations to both our users and to our developers. Make sure you're always seeing the verified badge when using it. Regards, Giancarlo Razzolini [0] https://help.github.com/articles/verifying-your-organization-s-domain/ [1] https://paste.xinu.at/CgBNjk/

1 0

[arch-devops] Hetzner 2FA and CAA added to domains
by Giancarlo Razzolini 07 Aug '18

07 Aug '18

Hi All, I have enabled Certificate Authority Authorization (RFC 6844) on both our main domains, archlinux.org and pkgbuild.com. Since we only use letsencrypt to issue certificates, I have added a CAA record only allowing it to issue SSL certificates. We can easily change it if needed in the future. I have also enabled 2FA for hetzner. Since it allows more than one token, the idea is for each one of us in the devops team to have a separate token. I'm going to send you all an encrypted email with the recovery key. This key should only be used in the unlikely event that we have any of our 2FA compromised or we need to login to the account in and emergency. If we use this recovery key, a new one should be issued and emailed to all the members again. I have created a token for myself, and I'm going to add another "master" token that I'll add to the ansible vault. You guys can use this token to login and create another for yourself. Please add your ansible username to the description, like I have done for my token. Aaron, since you don't have root access to the ansible repository, I can either give you access or, if you don't want to handle ansible stuff, I can email separately to you both the recovery key and the "master" token seed so you can create your own. Cheers, Giancarlo Razzolini

1 0

[arch-devops] [RFC] Better security through Content Security Policy and other headers.
by Jelle van der Waa 04 Aug '18

04 Aug '18

Hi All, Below is a proposal to increase our websecurity for *.archlinux.org # Goal Better security through Content Security Policy and other headers. # How Either sending CSP headers via nginx or modifying the web application to send the correct headers. Configuring via nginx is probably the easiest way to distribute these changes. # Which sites bugs.archlinux.org bbs.archlinux.org wiki.archlinux.org git.archlinux.org pkgbuild.com pkgbuild.com/~* mailman.archlinux.org kanboard.archlinux.org planet.archlinux.org archlinux.org (django) aur.archlinux.org Excluded is security.archlinux.org, which already includes all of the below headers via flask-talisman. # Which headers ## X-XSS-Protection The HTTP X-XSS-Protection response header is a feature of Internet Explorer, Chrome and Safari that stops pages from loading when they detect reflected cross-site scripting (XSS) attacks. This is mostly superseded by CSP, but still is not harmfull to add. add_header X-Xss-Protection "1; mode=block" always; ## X-Content-Type-Options The X-Content-Type-Options response HTTP header is a marker used by the server to indicate that the MIME types advertised in the Content-Type headers should not be changed and be followed. Check bugtracker/archweb if this breaks attachments or other files which are download. add_header X-Content-Type-Options "nosniff" always; ## X-Frame-Options The X-Frame-Options HTTP response header can be used to indicate whether or not a browser should be allowed to render a page in a <frame>, <iframe> or <object> . Sites can use this to avoid clickjacking attacks, by ensuring that their content is not embedded into other sites. I don't believe we want to allow embedding our websites. add_header X-Frame-Options "SAMEORIGIN" always; ## Content-Security-Policy The HTTP Content-Security-Policy response header allows web site administrators to control resources the user agent is allowed to load for a given page. With a few exceptions, policies mostly involve specifying server origins and script endpoints. This helps guard against cross-site scripting attacks (XSS). This might differ a bit per site, Archweb for example requires unsafe-inline / unsafe-eval for JavaScript and CSS. But all our loaded content should be from the same origin and restricted. An example policy would be: add_header Content-Security-Policy "default-src 'self'; style-src 'self'; font-src 'self'; form-action 'self';" # Challenges * inline JavaScript => archweb/aurweb => use a nonce or rewrite. * inline CSS => archweb/aurweb => use a nonce or rewrite. * unsafe-eval => this should never be allowed and should be fixed # Verifying The following website nicely shows the enabled security headers. https://securityheaders.com/ More information about these headers can be found here. https://scotthelme.co.uk/hardening-your-http-response-headers/ -- Jelle van der Waa

2 1

[arch-devops] Secondary backup ideas (Task #50)
by Phillip Smith 27 Jul '18

27 Jul '18

(Sorry, I can't reply to the original thread as I wasn't a member of the list at the time) Can I throw restic[0] in to the mix? It's very similar to borg, but super fast[1]. I moved away from borg a while ago for various reasons. It's written in golang, so it's a simple ELF executable to deploy. Supports lots of destinations[2] and multiple keys [3]. Specifically, I'm thinking the multiple key feature would allow us to have the client push backups to the backup server over a read-only connection (SSH, S3, rsync, whatever), then be able to run restic server-side to handle the cleanups. Anyway, 'food for thought' that we can discuss. Cheers, ~p [0] https://restic.net/ [1] My home desktop backup: ~121gb of source data on spinning-rust (not SSD); incremental backup takes ~8 seconds: scanned 11482 directories, 72707 files in 0:01 [0:07] 100.00% 13.298 GiB/s 121.249 GiB / 121.249 GiB 84189 / 84189 items 0 errors ETA 0:02 duration: 0:07, 15798.45MiB/s snapshot 8c4aea6d saved [2] https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html [3] https://restic.readthedocs.io/en/stable/070_encryption.html

3 5