- Aur-dev - lists.archlinux.org

[aur-dev] [PATCH 0/3] More tests
by Lukas Fleischer 06 Aug '16

06 Aug '16

This is a follow-up to the series I submitted earlier today. All patches are based on pu. Lukas Fleischer (3): t0001: Add more git-auth tests t0002: Add more git-serve tests t0003: Add more git-update tests git-interface/test/setup.sh | 91 ++++++++-- git-interface/test/t0001-auth.sh | 9 + git-interface/test/t0002-serve.sh | 62 +++++++ git-interface/test/t0003-update.sh | 350 ++++++++++++++++++++++++++++++++++++- 4 files changed, 496 insertions(+), 16 deletions(-) -- 2.9.2

1 4

[aur-dev] [PATCH] git-update: Remove package details before updating
by Lukas Fleischer 05 Aug '16

05 Aug '16

Explicitly remove all package sources, dependencies, relations, licenses and groups before inserting new ones. Signed-off-by: Lukas Fleischer <lfleischer(a)archlinux.org> --- Based on pu. git-interface/git-update.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/git-interface/git-update.py b/git-interface/git-update.py index 2820720..e6f6410 100755 --- a/git-interface/git-update.py +++ b/git-interface/git-update.py @@ -78,8 +78,13 @@ def save_metadata(metadata, conn, user): conn.execute("UPDATE PackageBases SET MaintainerUID = ? " + "WHERE ID = ? AND MaintainerUID IS NULL", [user_id, pkgbase_id]) - conn.execute("DELETE FROM Packages WHERE PackageBaseID = ?", - [pkgbase_id]) + for table in ('Sources', 'Depends', 'Relations', 'Licenses', 'Groups'): + conn.execute("DELETE FROM Package" + table + " WHERE EXISTS (" + + "SELECT * FROM Packages " + + "WHERE Packages.PackageBaseID = ? AND " + + "Package" + table + ".PackageID = Packages.ID)", + [pkgbase_id]) + conn.execute("DELETE FROM Packages WHERE PackageBaseID = ?", [pkgbase_id]) for pkgname in srcinfo.utils.get_package_names(metadata): pkginfo = srcinfo.utils.get_merged_package(pkgname, metadata) -- 2.9.2

1 0

[aur-dev] [PATCH 0/8] Introduce a test suite for the Git/SSH interface
by Lukas Fleischer 05 Aug '16

05 Aug '16

This larger patch set adds a test suite for the Git/SSH interface. It also includes some very basic tests of git-auth, git-serve and git-update. The series consists of three major parts: Patches 1, 4, 5, 6 and 7 prepare the code and the database schema such that one can alternatively use SQLite as database backend. Patches 2 and 3 add support for overriding the path to the configuration file using an environment variable. The last patch finally adds the test suite and some tests. Since it makes use of the new SQLite backend, no MySQL server is required. The whole topic will be merged to pu and sit there for a while. Comments welcome! Lukas Fleischer (8): git-interface: Add database abstraction layer git-interface: Factor out configuration file parsing git-interface: Add AUR_CONFIG environment variable git-interface: Do not use rowcount git-interface: Do not use UNIX_TIMESTAMP git-interface: Support SQLite as database backend Cleanup database schema git-interface: Add test suite and basic tests conf/config.proto | 2 +- git-interface/config.py | 31 ++ git-interface/db.py | 51 +++ git-interface/git-auth.py | 33 +- git-interface/git-serve.py | 96 ++--- git-interface/git-update.py | 162 ++++--- git-interface/test/Makefile | 11 + git-interface/test/setup.sh | 121 ++++++ git-interface/test/sharness.sh | 851 +++++++++++++++++++++++++++++++++++++ git-interface/test/t0001-auth.sh | 19 + git-interface/test/t0002-serve.sh | 26 ++ git-interface/test/t0003-update.sh | 20 + schema/aur-schema.sql | 67 ++- 13 files changed, 1288 insertions(+), 202 deletions(-) create mode 100644 git-interface/config.py create mode 100644 git-interface/db.py create mode 100644 git-interface/test/Makefile create mode 100644 git-interface/test/setup.sh create mode 100644 git-interface/test/sharness.sh create mode 100755 git-interface/test/t0001-auth.sh create mode 100755 git-interface/test/t0002-serve.sh create mode 100755 git-interface/test/t0003-update.sh -- 2.9.2

1 8

[aur-dev] [PATCH 1/3] git-update: Fix some issues reported by pyflakes
by Lukas Fleischer 05 Aug '16

05 Aug '16

Signed-off-by: Lukas Fleischer <lfleischer(a)archlinux.org> --- git-interface/git-update.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/git-interface/git-update.py b/git-interface/git-update.py index 41b38a6..5fc5562 100755 --- a/git-interface/git-update.py +++ b/git-interface/git-update.py @@ -167,11 +167,13 @@ def save_metadata(metadata, db, cur, user): "PackageBaseID = %s AND UserID = %s", [pkgbase_id, user_id]) if cur.fetchone()[0] == 0: - cur.execute("INSERT INTO PackageNotifications (PackageBaseID, UserID) " + - "VALUES (%s, %s)", [pkgbase_id, user_id]) + cur.execute("INSERT INTO PackageNotifications " + + "(PackageBaseID, UserID) VALUES (%s, %s)", + [pkgbase_id, user_id]) db.commit() + def update_notify(db, cur, user, pkgbase_id): # Obtain the user ID of the new maintainer. cur.execute("SELECT ID FROM Users WHERE Username = %s", [user]) @@ -180,6 +182,7 @@ def update_notify(db, cur, user, pkgbase_id): # Execute the notification script. subprocess.Popen((notify_cmd, 'update', str(user_id), str(pkgbase_id))) + def die(msg): sys.stderr.write("error: {:s}\n".format(msg)) exit(1) -- 2.9.2

1 2

[aur-dev] [PATCH] Change default SSH options to "restrict"
by Lukas Fleischer 05 Aug '16

05 Aug '16

>From the sshd(8) man page: Enable all restrictions, i.e. disable port, agent and X11 forwarding, as well as disabling PTY allocation and execution of ~/.ssh/rc. If any future restriction capabilities are added to authorized_keys files they will be included in this set. Signed-off-by: Lukas Fleischer <lfleischer(a)archlinux.org> --- conf/config.proto | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/conf/config.proto b/conf/config.proto index 64af774..d5778a0 100644 --- a/conf/config.proto +++ b/conf/config.proto @@ -47,7 +47,7 @@ RSA = SHA256:Ju+yWiMb/2O+gKQ9RJCDqvRg7l+Q95KFAeqM5sr6l2s valid-keytypes = ssh-rsa ssh-dss ecdsa-sha2-nistp256 ecdsa-sha2-nistp384 ecdsa-sha2-nistp521 ssh-ed25519 username-regex = [a-zA-Z0-9]+[.\-_]?[a-zA-Z0-9]+$ git-serve-cmd = /srv/http/aurweb/git-interface/git-serve.py -ssh-options = no-port-forwarding,no-X11-forwarding,no-pty +ssh-options = restrict [serve] repo-path = /srv/http/aurweb/aur.git/ -- 2.9.2

1 0

[aur-dev] [PATCH] git-serve: Deprecate setup-repo
by Lukas Fleischer 05 Aug '16

05 Aug '16

Since 02dd9c5 (git-serve.py: Automatically create repositories, 2015-01-06), one can create new package bases by running `git push`. It is no longer necessary to run setup-repo manually. Signed-off-by: Lukas Fleischer <lfleischer(a)archlinux.org> --- git-interface/git-serve.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/git-interface/git-serve.py b/git-interface/git-serve.py index aa5f1c9..81b7e93 100755 --- a/git-interface/git-serve.py +++ b/git-interface/git-serve.py @@ -133,6 +133,10 @@ def die_with_help(msg): die(msg + "\nTry `{:s} help` for a list of commands.".format(ssh_cmdline)) +def warn(msg): + sys.stderr.write("warning: {:s}\n".format(msg)) + + user = os.environ.get('AUR_USER') privileged = (os.environ.get('AUR_PRIVILEGED', '0') == '1') ssh_cmd = os.environ.get('SSH_ORIGINAL_COMMAND') @@ -186,6 +190,7 @@ elif action == 'setup-repo': die_with_help("{:s}: missing repository name".format(action)) if len(cmdargv) > 2: die_with_help("{:s}: too many arguments".format(action)) + warn('{:s} is deprecated. Use `git push` to create new repositories.') create_pkgbase(cmdargv[1], user) elif action == 'restore': if len(cmdargv) < 2: -- 2.9.2

1 1

[aur-dev] [PATCH] git-update: Notify privileged users of forced uploads
by Lukas Fleischer 01 Aug '16

01 Aug '16

Show a warning when a Trusted User or a developer creates a package that is blacklisted or already provided by an official package. Signed-off-by: Lukas Fleischer <lfleischer(a)archlinux.org> --- git-interface/git-update.py | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/git-interface/git-update.py b/git-interface/git-update.py index 2c24e72..41b38a6 100755 --- a/git-interface/git-update.py +++ b/git-interface/git-update.py @@ -202,6 +202,7 @@ repo = pygit2.Repository(repo_path) user = os.environ.get("AUR_USER") pkgbase = os.environ.get("AUR_PKGBASE") privileged = (os.environ.get("AUR_PRIVILEGED", '0') == '1') +warn_or_die = warn if privileged else die if len(sys.argv) == 2 and sys.argv[1] == "restore": if 'refs/heads/' + pkgbase not in repo.listall_references(): @@ -338,11 +339,11 @@ for pkgname in srcinfo.utils.get_package_names(metadata): pkginfo = srcinfo.utils.get_merged_package(pkgname, metadata) pkgname = pkginfo['pkgname'] - if pkgname in blacklist and not privileged: - die('package is blacklisted: {:s}'.format(pkgname)) - if pkgname in providers and not privileged: + if pkgname in blacklist: + warn_or_die('package is blacklisted: {:s}'.format(pkgname)) + if pkgname in providers: repo = providers[pkgname] - die('package already provided by [{:s}]: {:s}'.format(repo, pkgname)) + warn_or_die('package already provided by [{:s}]: {:s}'.format(repo, pkgname)) cur.execute("SELECT COUNT(*) FROM Packages WHERE Name = %s AND " + "PackageBaseID <> %s", [pkgname, pkgbase_id]) -- 2.9.0

1 0

[aur-dev] [PATCH] git-serve: Refactor environment variable access
by Lukas Fleischer 01 Aug '16

01 Aug '16

Read all environment variables at the beginning of the script and immediately pre-process their values. Signed-off-by: Lukas Fleischer <lfleischer(a)archlinux.org> --- git-interface/git-serve.py | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/git-interface/git-serve.py b/git-interface/git-serve.py index 35c6b3a..aa5f1c9 100755 --- a/git-interface/git-serve.py +++ b/git-interface/git-serve.py @@ -108,15 +108,12 @@ def pkgbase_set_keywords(pkgbase, keywords): db.close() -def check_permissions(pkgbase, user): +def pkgbase_has_write_access(pkgbase, user): db = mysql.connector.connect(host=aur_db_host, user=aur_db_user, passwd=aur_db_pass, db=aur_db_name, unix_socket=aur_db_socket, buffered=True) cur = db.cursor() - if os.environ.get('AUR_PRIVILEGED', '0') == '1': - return True - cur.execute("SELECT COUNT(*) FROM PackageBases " + "LEFT JOIN PackageComaintainers " + "ON PackageComaintainers.PackageBaseID = PackageBases.ID " + @@ -136,15 +133,18 @@ def die_with_help(msg): die(msg + "\nTry `{:s} help` for a list of commands.".format(ssh_cmdline)) -user = os.environ.get("AUR_USER") -cmd = os.environ.get("SSH_ORIGINAL_COMMAND") -if not cmd: +user = os.environ.get('AUR_USER') +privileged = (os.environ.get('AUR_PRIVILEGED', '0') == '1') +ssh_cmd = os.environ.get('SSH_ORIGINAL_COMMAND') +ssh_client = os.environ.get('SSH_CLIENT') + +if not ssh_cmd: die_with_help("Interactive shell is disabled.") -cmdargv = shlex.split(cmd) +cmdargv = shlex.split(ssh_cmd) action = cmdargv[0] +remote_addr = ssh_client.split(' ')[0] if ssh_client else None if enable_maintenance: - remote_addr = os.environ["SSH_CLIENT"].split(" ")[0] if remote_addr not in maintenance_exc: die("The AUR is down due to maintenance. We will be back soon.") @@ -165,7 +165,7 @@ if action == 'git-upload-pack' or action == 'git-receive-pack': create_pkgbase(pkgbase, user) if action == 'git-receive-pack': - if not check_permissions(pkgbase, user): + if not privileged and not pkgbase_has_write_access(pkgbase, user): die('{:s}: permission denied: {:s}'.format(action, user)) os.environ["AUR_USER"] = user -- 2.9.0

1 0

[aur-dev] [PATCH] Make LangPreference DB field wide enough to fit es_419
by Johannes Löthberg 20 Jul '16

20 Jul '16

The language code for Latin American Spanish is es_419, which is longer than the 5 characters previously allowed. Signed-off-by: Johannes Löthberg <johannes(a)kyriasis.com> --- schema/aur-schema.sql | 2 +- upgrading/4.3.0.txt | 6 ++++++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/schema/aur-schema.sql b/schema/aur-schema.sql index 1affc25..9991129 100644 --- a/schema/aur-schema.sql +++ b/schema/aur-schema.sql @@ -31,7 +31,7 @@ CREATE TABLE Users ( Salt CHAR(32) NOT NULL DEFAULT '', ResetKey CHAR(32) NOT NULL DEFAULT '', RealName VARCHAR(64) NOT NULL DEFAULT '', - LangPreference VARCHAR(5) NOT NULL DEFAULT 'en', + LangPreference VARCHAR(6) NOT NULL DEFAULT 'en', Homepage TEXT NULL DEFAULT NULL, IRCNick VARCHAR(32) NOT NULL DEFAULT '', PGPKey VARCHAR(40) NULL DEFAULT NULL, diff --git a/upgrading/4.3.0.txt b/upgrading/4.3.0.txt index 6c1753d..78f9455 100644 --- a/upgrading/4.3.0.txt +++ b/upgrading/4.3.0.txt @@ -21,3 +21,9 @@ ALTER TABLE OfficialProviders ADD COLUMN Repo VARCHAR(64) NOT NULL; ---- ALTER TABLE Users ADD COLUMN Homepage TEXT NULL DEFAULT NULL; ---- + +5. Resize LangPreference to fit Latin American Spanish language code: + +-- +ALTER TABLE Users MODIFY LangPreference VARCHAR(6); +-- -- 2.9.0

1 0

[aur-dev] [PATCH] Remove wearout factor from translatable string
by Lukas Fleischer 15 Jul '16

15 Jul '16

Even though that number is hardcoded in the source code, it is better to make it not appear in the message catalog such that it can be made configurable easily later. Signed-off-by: Lukas Fleischer <lfleischer(a)archlinux.org> --- web/template/pkg_search_results.php | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/web/template/pkg_search_results.php b/web/template/pkg_search_results.php index 3046c25..37a9032 100644 --- a/web/template/pkg_search_results.php +++ b/web/template/pkg_search_results.php @@ -35,7 +35,7 @@ if (!$result): ?> <th><a href="?<?= mkurl('SB=n&SO=' . $SO_next) ?>"><?= __("Name") ?></a></th> <th><?= __("Version") ?></th> <th><a href="?<?= mkurl('SB=v&SO=' . $SO_next) ?>"><?= __("Votes") ?></a></th> - <th><a href="?<?= mkurl('SB=p&SO=' . $SO_next) ?>"><?= __("Popularity") ?></a>" class="hover-help">?</th> + <th><a href="?<?= mkurl('SB=p&SO=' . $SO_next) ?>"><?= __("Popularity") ?></a>" class="hover-help">?</th> <?php if ($SID): ?> <th><a href="?<?= mkurl('SB=w&SO=' . $SO_next) ?>"><?= __("Voted") ?></a></th> <th><a href="?<?= mkurl('SB=o&SO=' . $SO_next) ?>"><?= __("Notify") ?></a></th> -- 2.9.0

1 0