- Aur-dev - lists.archlinux.org

aurweb tests and ci
by Kevin Morris 02 Jul '20

02 Jul '20

Hello aur-dev, Working on some things in aurweb at the moment, and I noticed that on master tests are failing on my system, and .travis.yml doesn't seem to properly setup the environment needed to run tests (sqlalchemy is not installed, db tables missing) on travis. Just a few questions 1. What's the state of testing aurweb currently? 2. What are any test barriers to committing code and submitting patches? 2a. What are the testing goals? 3. Is there a thread on this mailing list about this topic? Thank you very much for your time. Regards, Kevin -- Kevin Morris Software Developer Personal Inquiries: kevr.gtalk(a)gmail.com Personal Phone: (415) 583-9687 Technologies: C, C++, Python, Django, Ruby, Rails, ReactJS, jQuery, Javascript, SQL, Redux

2 1

[PATCH 3/3] Open AUR sessions from SSO
by Frédéric Mangano-Tarumi 14 Jun '20

14 Jun '20

Only the core functionality is implemented here. See the TODOs. --- aurweb/routers/sso.py | 51 +++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 49 insertions(+), 2 deletions(-) diff --git a/aurweb/routers/sso.py b/aurweb/routers/sso.py index b16edffb..d0802c34 100644 --- a/aurweb/routers/sso.py +++ b/aurweb/routers/sso.py @@ -1,9 +1,18 @@ +import time +import uuid + import fastapi from authlib.integrations.starlette_client import OAuth +from fastapi import Depends, HTTPException +from fastapi.responses import RedirectResponse +from sqlalchemy.sql import select from starlette.requests import Request import aurweb.config +import aurweb.db + +from aurweb.schema import Sessions, Users router = fastapi.APIRouter() @@ -23,8 +32,46 @@ async def login(request: Request): return await oauth.sso.authorize_redirect(request, redirect_uri, prompt="login") +def open_session(conn, user_id): + """ + Create a new user session into the database. Return its SID. + """ + # TODO check for account suspension + # TODO apply [options] max_sessions_per_user + sid = uuid.uuid4().hex + conn.execute(Sessions.insert().values( + UsersID=user_id, + SessionID=sid, + LastUpdateTS=time.time(), + )) + # TODO update Users.LastLogin and Users.LastLoginIPAddress + return sid + + @router.get("/sso/authenticate") -async def authenticate(request: Request): +async def authenticate(request: Request, conn=Depends(aurweb.db.connect)): + """ + Receive an OpenID Connect ID token, validate it, then process it to create + an new AUR session. + """ + # TODO check for banned IPs token = await oauth.sso.authorize_access_token(request) user = await oauth.sso.parse_id_token(request, token) - return dict(user) + sub = user.get("sub") # this is the SSO account ID in JWT terminology + if not sub: + raise HTTPException(status_code=400, detail="JWT is missing its `sub` field.") + + aur_accounts = conn.execute(select([Users.c.ID]).where(Users.c.SSOAccountID == sub)) \ + .fetchall() + if not aur_accounts: + return "Sorry, we don’t seem to know you Sir " + sub + elif len(aur_accounts) == 1: + sid = open_session(conn, aur_accounts[0][Users.c.ID]) + response = RedirectResponse("/") + # TODO redirect to the referrer + response.set_cookie(key="AURSID", value=sid, httponly=True, + secure=request.url.scheme == "https") + return response + else: + # We’ve got a severe integrity violation. + raise Exception("Multiple accounts found for SSO account " + sub) -- 2.27.0

2 1

[PATCH] aurweb.l10n: Translate without side effects
by Frédéric Mangano-Tarumi 10 Jun '20

10 Jun '20

The install method in Python’s gettext API aliases the translator’s gettext method to an application-global _(). We don’t use that anywhere, and it’s clear from aurweb’s Translator interface that we want to translate a piece of text without affecting any global namespace. --- aurweb/l10n.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/aurweb/l10n.py b/aurweb/l10n.py index 492200b3..51b56abb 100644 --- a/aurweb/l10n.py +++ b/aurweb/l10n.py @@ -15,5 +15,4 @@ class Translator: self._translator[lang] = gettext.translation("aurweb", self._localedir, languages=[lang]) - self._translator[lang].install() - return _(s) # _ is not defined, what is this? # noqa: F821 + return self._translator[lang].gettext(s) -- 2.27.0

2 1

[PATCH 2/3] Integrate SQLAlchemy into FastAPI
by Frédéric Mangano-Tarumi 08 Jun '20

08 Jun '20

--- aurweb/db.py | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/aurweb/db.py b/aurweb/db.py index 1ccd9a07..02aeba38 100644 --- a/aurweb/db.py +++ b/aurweb/db.py @@ -10,6 +10,8 @@ except ImportError: import aurweb.config +engine = None # See get_engine + def get_sqlalchemy_url(): """ @@ -38,6 +40,34 @@ def get_sqlalchemy_url(): raise ValueError('unsupported database backend') +def get_engine(): + """ + Return the global SQLAlchemy engine. + + The engine is created on the first call to get_engine and then stored in the + `engine` global variable for the next calls. + """ + from sqlalchemy import create_engine + global engine + if engine is None: + engine = create_engine(get_sqlalchemy_url(), + # check_same_thread is for a SQLite technicality + # https://fastapi.tiangolo.com/tutorial/sql-databases/#note + connect_args={"check_same_thread": False}) + return engine + + +def connect(): + """ + Return an SQLAlchemy connection. Connections are usually pooled. See + <https://docs.sqlalchemy.org/en/13/core/connections.html>. + + Since SQLAlchemy connections are context managers too, you should use it + with Python’s `with` operator, or with FastAPI’s dependency injection. + """ + return get_engine().connect() + + class Connection: _conn = None _paramstyle = None -- 2.27.0

1 0

[PATCH 1/3] Add SSO account ID in table Users
by Frédéric Mangano-Tarumi 08 Jun '20

08 Jun '20

This column holds a user ID issed by the single sign-on provider. For Keycloak, it is an UUID. For more flexibility, we will be using a standardly-sized VARCHAR field. --- aurweb/schema.py | 1 + ...6e1cd_add_sso_account_id_in_table_users.py | 30 +++++++++++++++++++ 2 files changed, 31 insertions(+) create mode 100644 migrations/versions/ef39fcd6e1cd_add_sso_account_id_in_table_users.py diff --git a/aurweb/schema.py b/aurweb/schema.py index 20f3e5ce..a1d56281 100644 --- a/aurweb/schema.py +++ b/aurweb/schema.py @@ -67,6 +67,7 @@ Users = Table( Column('CommentNotify', TINYINT(1), nullable=False, server_default=text("1")), Column('UpdateNotify', TINYINT(1), nullable=False, server_default=text("0")), Column('OwnershipNotify', TINYINT(1), nullable=False, server_default=text("1")), + Column('SSOAccountID', String(255), nullable=True, unique=True), Index('UsersAccountTypeID', 'AccountTypeID'), mysql_engine='InnoDB', ) diff --git a/migrations/versions/ef39fcd6e1cd_add_sso_account_id_in_table_users.py b/migrations/versions/ef39fcd6e1cd_add_sso_account_id_in_table_users.py new file mode 100644 index 00000000..9e125165 --- /dev/null +++ b/migrations/versions/ef39fcd6e1cd_add_sso_account_id_in_table_users.py @@ -0,0 +1,30 @@ +"""Add SSO account ID in table Users + +Revision ID: ef39fcd6e1cd +Revises: f47cad5d6d03 +Create Date: 2020-06-08 10:04:13.898617 + +""" +from alembic import op +import sqlalchemy as sa + + +# revision identifiers, used by Alembic. +revision = 'ef39fcd6e1cd' +down_revision = 'f47cad5d6d03' +branch_labels = None +depends_on = None + + +def upgrade(): + # ### commands auto generated by Alembic - please adjust! ### + op.add_column('Users', sa.Column('SSOAccountID', sa.String(length=255), nullable=True)) + op.create_unique_constraint(None, 'Users', ['SSOAccountID']) + # ### end Alembic commands ### + + +def downgrade(): + # ### commands auto generated by Alembic - please adjust! ### + op.drop_constraint(None, 'Users', type_='unique') + op.drop_column('Users', 'SSOAccountID') + # ### end Alembic commands ### -- 2.27.0

1 0

[PATCH 3/4] Crude OpenID Connect client using Authlib
by Frédéric Mangano-Tarumi 06 Jun '20

06 Jun '20

Developers can go to /sso/login to get redirected to the SSO. On successful login, the ID token is displayed. --- .gitlab-ci.yml | 3 ++- TESTING | 3 ++- aurweb/asgi.py | 13 +++++++++++++ aurweb/routers/__init__.py | 5 +++++ aurweb/routers/sso.py | 30 ++++++++++++++++++++++++++++++ aurweb/spawn.py | 3 +++ conf/config.defaults | 8 ++++++++ conf/config.dev | 9 +++++++++ 8 files changed, 72 insertions(+), 2 deletions(-) create mode 100644 aurweb/routers/__init__.py create mode 100644 aurweb/routers/sso.py diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index f6260ebb..9dc951aa 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -11,7 +11,8 @@ before_script: base-devel git gpgme protobuf pyalpm python-mysql-connector python-pygit2 python-srcinfo python-bleach python-markdown python-sqlalchemy python-alembic python-pytest python-werkzeug - python-pytest-tap python-fastapi uvicorn nginx + python-pytest-tap python-fastapi uvicorn nginx python-authlib + python-itsdangerous python-httpx test: script: diff --git a/TESTING b/TESTING index 7261df92..d7df3672 100644 --- a/TESTING +++ b/TESTING @@ -13,7 +13,8 @@ INSTALL. # pacman -S --needed php php-sqlite sqlite words fortune-mod \ python python-sqlalchemy python-alembic \ - python-fastapi uvicorn nginx + python-fastapi uvicorn nginx \ + python-authlib python-itsdangerous python-httpx Ensure to enable the pdo_sqlite extension in php.ini. diff --git a/aurweb/asgi.py b/aurweb/asgi.py index 9bb71ecc..60c7ade7 100644 --- a/aurweb/asgi.py +++ b/aurweb/asgi.py @@ -1,3 +1,16 @@ from fastapi import FastAPI +from starlette.middleware.sessions import SessionMiddleware + +import aurweb.config + +from aurweb.routers import sso app = FastAPI() + +session_secret = aurweb.config.get("fastapi", "session_secret") +if not session_secret: + raise Exception("[fastapi] session_secret must not be empty") + +app.add_middleware(SessionMiddleware, secret_key=session_secret) + +app.include_router(sso.router) diff --git a/aurweb/routers/__init__.py b/aurweb/routers/__init__.py new file mode 100644 index 00000000..35d43c03 --- /dev/null +++ b/aurweb/routers/__init__.py @@ -0,0 +1,5 @@ +""" +API routers for FastAPI. + +See https://fastapi.tiangolo.com/tutorial/bigger-applications/ +""" diff --git a/aurweb/routers/sso.py b/aurweb/routers/sso.py new file mode 100644 index 00000000..b16edffb --- /dev/null +++ b/aurweb/routers/sso.py @@ -0,0 +1,30 @@ +import fastapi + +from authlib.integrations.starlette_client import OAuth +from starlette.requests import Request + +import aurweb.config + +router = fastapi.APIRouter() + +oauth = OAuth() +oauth.register( + name="sso", + server_metadata_url=aurweb.config.get("sso", "openid_configuration"), + client_kwargs={"scope": "openid"}, + client_id=aurweb.config.get("sso", "client_id"), + client_secret=aurweb.config.get("sso", "client_secret"), +) + + +(a)router.get("/sso/login") +async def login(request: Request): + redirect_uri = aurweb.config.get("options", "aur_location") + "/sso/authenticate" + return await oauth.sso.authorize_redirect(request, redirect_uri, prompt="login") + + +(a)router.get("/sso/authenticate") +async def authenticate(request: Request): + token = await oauth.sso.authorize_access_token(request) + user = await oauth.sso.parse_id_token(request, token) + return dict(user) diff --git a/aurweb/spawn.py b/aurweb/spawn.py index e86f29fe..5da8587e 100644 --- a/aurweb/spawn.py +++ b/aurweb/spawn.py @@ -60,6 +60,9 @@ def generate_nginx_config(): location / {{ proxy_pass http://{aurweb.config.get("php", "bind_address")}; }} + location /sso {{ + proxy_pass http://{aurweb.config.get("fastapi", "bind_address")}; + }} }} }} """) diff --git a/conf/config.defaults b/conf/config.defaults index 447dacac..49259754 100644 --- a/conf/config.defaults +++ b/conf/config.defaults @@ -68,6 +68,14 @@ username-regex = [a-zA-Z0-9]+[.\-_]?[a-zA-Z0-9]+$ git-serve-cmd = /usr/local/bin/aurweb-git-serve ssh-options = restrict +[sso] +openid_configuration = +client_id = +client_secret = + +[fastapi] +session_secret = + [serve] repo-path = /srv/http/aurweb/aur.git/ repo-regex = [a-z0-9][a-z0-9.+_-]*$ diff --git a/conf/config.dev b/conf/config.dev index d752f61f..27e981f8 100644 --- a/conf/config.dev +++ b/conf/config.dev @@ -20,6 +20,12 @@ aur_location = http://127.0.0.1:8080 disable_http_login = 0 enable-maintenance = 0 +; Single sign-on +[sso] +openid_configuration = http://127.0.0.1:8083/auth/realms/aurweb/.well-known/openid-configuration +client_id = aurweb +client_secret = + [php] ; Address PHP should bind when spawned in development mode by aurweb.spawn. bind_address = 127.0.0.1:8081 @@ -30,3 +36,6 @@ htmldir = YOUR_AUR_ROOT/web/html [fastapi] ; Address uvicorn should bind when spawned in development mode by aurweb.spawn. bind_address = 127.0.0.1:8082 + +; Passphrase FastAPI uses to sign client-side sessions. +session_secret = 極秘、訳すな！あ、遅過ぎた。 -- 2.27.0

2 2

[PATCH 4/4] Guide to setting up Keycloak for the SSO
by Frédéric Mangano-Tarumi 06 Jun '20

06 Jun '20

--- conf/config.dev | 2 +- doc/SSO | 38 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+), 1 deletion(-) create mode 100644 doc/SSO diff --git a/conf/config.dev b/conf/config.dev index 27e981f8..fb0cc99a 100644 --- a/conf/config.dev +++ b/conf/config.dev @@ -20,7 +20,7 @@ aur_location = http://127.0.0.1:8080 disable_http_login = 0 enable-maintenance = 0 -; Single sign-on +; Single sign-on. See doc/SSO [sso] openid_configuration = http://127.0.0.1:8083/auth/realms/aurweb/.well-known/openid-configuration client_id = aurweb diff --git a/doc/SSO b/doc/SSO new file mode 100644 index 00000000..1b0b1f7d --- /dev/null +++ b/doc/SSO @@ -0,0 +1,38 @@ +Single Sign-On (SSO) +==================== + +This guide will walk you through setting up Keycloak for use with aurweb. For +extensive documentation, see <https://www.keycloak.org/documentation.html>. + +Installing Keycloak +------------------- + +Keycloak is in the official Arch repositories: + + # pacman -S keycloak + +The default port is 8080, which conflicts with aurweb’s default port. You need +to edit `/etc/keycloak/standalone.xml`, looking for this line: + + <socket-binding name="http" port="${jboss.http.port:8080}"/> + +The default developer configuration assumes it is set to 8083. Alternatively, +you may customize [options] aur_location and [sso] openid_configuration in +`conf/config`. + +You may then start `keycloak.service` through systemd. + +See also ArchWiki <https://wiki.archlinux.org/index.php/Keycloak>. + +Configuring a realm +------------------- + +Go to <http://127.0.0.1:8083/auth> and log in as administrator. Then, hover the +text right below the Keycloak logo at the top left, by default *Master*. Click +*Add realm* and name it *aurweb*. + +Open the *Clients* tab, and create a new *openid-connect* client. Call it +*aurweb*, and set the root URL to <http://127.0.0.1:8080> (your aur_location). + +Create a user from the *Users* tab and try logging in from +<http://127.0.0.1:8083/auth/realms/aurweb/account/>. -- 2.27.0

2 1

[PATCH 2/4] aurweb.spawn: Fix isort errors
by Frédéric Mangano-Tarumi 04 Jun '20

04 Jun '20

--- aurweb/spawn.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/aurweb/spawn.py b/aurweb/spawn.py index 7fe59e65..e86f29fe 100644 --- a/aurweb/spawn.py +++ b/aurweb/spawn.py @@ -8,8 +8,8 @@ configuration anyway. """ -import atexit import argparse +import atexit import os import subprocess import sys @@ -20,7 +20,6 @@ import urllib import aurweb.config import aurweb.schema - children = [] temporary_dir = None verbosity = 0 -- 2.27.0

1 0

[PATCH 1/4] Remove the FastAPI /hello test route
by Frédéric Mangano-Tarumi 04 Jun '20

04 Jun '20

--- aurweb/asgi.py | 5 ----- aurweb/spawn.py | 3 --- 2 files changed, 8 deletions(-) diff --git a/aurweb/asgi.py b/aurweb/asgi.py index 5f30471a..9bb71ecc 100644 --- a/aurweb/asgi.py +++ b/aurweb/asgi.py @@ -1,8 +1,3 @@ from fastapi import FastAPI app = FastAPI() - - -(a)app.get("/hello/") -async def hello(): - return {"message": "Hello from FastAPI!"} diff --git a/aurweb/spawn.py b/aurweb/spawn.py index 0506afa4..7fe59e65 100644 --- a/aurweb/spawn.py +++ b/aurweb/spawn.py @@ -61,9 +61,6 @@ def generate_nginx_config(): location / {{ proxy_pass http://{aurweb.config.get("php", "bind_address")}; }} - location /hello {{ - proxy_pass http://{aurweb.config.get("fastapi", "bind_address")}; - }} }} }} """) -- 2.27.0

1 0

Tech stack for Python aurweb
by Lukas Fleischer 03 Jun '20

03 Jun '20

Context: In light of growing demand/requests for modernizing the aurweb code base and switching the code base to a more popular programming language, we did some preparatory research and work to allow for writing new subsystems in Python. Frédéric proposed a lightweight dual stack implementation that allows us to implement different parts of the website with different tech stacks. A full rewrite, as proposed before, can be carried out gradually and ported subsystems will be usable early. Independent of these efforts, we suggest that new subsystems will be written in Python and any components that undergo major changes should be reimplemented in Python if the extra effort is manageable. With a dual stack PoC and a first use case (SSO), I would like to briefly discuss some design decisions before we start implementing. Everybody involved in these efforts seemed to prefer Flask [1] so far. It's a mature micro framework and has a relatively low barrier of entry. Being the default template engine for Flask, Jinja [2] is a natural choice. PHP and Python versions of base templates will co-exist. Despite the fact that these base templates are rarely modified and maintaining two copies will not involve significant extra work, it is beneficial to deduplicate as much as possible. With this in mind, a language agnostic engine such as mustache [3] might be an option worth considering. This is open for discussion. As briefly discussed before, SQLAlchemy [4] will be used as database toolkit. We appreciate any comments and we would be happy to discuss alternative frameworks/toolkits. If I forgot any important aspect here, please feel free to chime in too! [1] https://flask.palletsprojects.com/ [2] https://palletsprojects.com/p/jinja/ [3] https://mustache.github.io/ [4] https://www.sqlalchemy.org/

4 7