Tech stack for Python aurweb
Context: In light of growing demand/requests for modernizing the aurweb code base and switching the code base to a more popular programming language, we did some preparatory research and work to allow for writing new subsystems in Python. Frédéric proposed a lightweight dual stack implementation that allows us to implement different parts of the website with different tech stacks. A full rewrite, as proposed before, can be carried out gradually and ported subsystems will be usable early. Independent of these efforts, we suggest that new subsystems will be written in Python and any components that undergo major changes should be reimplemented in Python if the extra effort is manageable. With a dual stack PoC and a first use case (SSO), I would like to briefly discuss some design decisions before we start implementing. Everybody involved in these efforts seemed to prefer Flask [1] so far. It's a mature micro framework and has a relatively low barrier of entry. Being the default template engine for Flask, Jinja [2] is a natural choice. PHP and Python versions of base templates will co-exist. Despite the fact that these base templates are rarely modified and maintaining two copies will not involve significant extra work, it is beneficial to deduplicate as much as possible. With this in mind, a language agnostic engine such as mustache [3] might be an option worth considering. This is open for discussion. As briefly discussed before, SQLAlchemy [4] will be used as database toolkit. We appreciate any comments and we would be happy to discuss alternative frameworks/toolkits. If I forgot any important aspect here, please feel free to chime in too! [1] https://flask.palletsprojects.com/ [2] https://palletsprojects.com/p/jinja/ [3] https://mustache.github.io/ [4] https://www.sqlalchemy.org/
On Mon, 2020-05-18 at 17:49 -0400, Lukas Fleischer wrote:
Everybody involved in these efforts seemed to prefer Flask [1] so far. It's a mature micro framework and has a relatively low barrier of entry.
I wouldn't be opposed to other framework, as long as it is mature and simple (rules out Django), but the obvious choice here seems to be Flask.
Being the default template engine for Flask, Jinja [2] is a natural choice. PHP and Python versions of base templates will co-exist. Despite the fact that these base templates are rarely modified and maintaining two copies will not involve significant extra work, it is beneficial to deduplicate as much as possible. With this in mind, a language agnostic engine such as mustache [3] might be an option worth considering. This is open for discussion.
Since we plan to entirely replace the PHP codebase with Python, and given the fact that we would still need to rewrite the existent templates in mustache, I would go for Jinja. We almost never touch the templates so the maintenance burden wouldn't be much, if any, so we really only have the porting effort to take into account.
As briefly discussed before, SQLAlchemy [4] will be used as database toolkit.
We will be using SQLAlchemy Core. I would like to potentially move to SQLAlchemy ORM in the future, but that would need to be discussed separately. Cheers, Filipe Laíns
On Mon, 18 May 2020 at 18:10:14, Filipe Laíns wrote:
On Mon, 2020-05-18 at 17:49 -0400, Lukas Fleischer wrote:
Everybody involved in these efforts seemed to prefer Flask [1] so far. It's a mature micro framework and has a relatively low barrier of entry.
I wouldn't be opposed to other framework, as long as it is mature and simple (rules out Django), but the obvious choice here seems to be Flask.
Just for the record, since the other email was detached from this thread: Sven made a case for FastAPI here [1]. I think it is an interesting alternative that we should consider before finalizing our decision. [1] https://lists.archlinux.org/pipermail/aur-dev/2020-May/004857.html
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On Mon, 2020-05-18 at 17:49 -0400, Lukas Fleischer wrote:
Everybody involved in these efforts seemed to prefer Flask [1] so far. It's a mature micro framework and has a relatively low barrier of entry.
I'm a python programmer for a while now and I really like the small size and simplicity of Flask. But let me give you one advice here. If you're dealing with a more complex application Flask tends to become more complex and very hard to manage. All of a sudden you integrate about 10 Flask addons. Some of them are not well maintained. Some droppen support for others etc. You have to manage all those dependencies. Personally I prefer Django in more complex projects as it comes with everything included. You rarely have to add something to your dependencies. I never understodd the argument about it being too big or heavy. Under normal workloads you don't see any difference. Yes Flasks anwsers a few 100 more requests per minute then Django, but is really important to your project? And even if it is, can't you just spawn another instance of it? My personal rule of thumb is to use Flask for everything simple. When users are involved it immediatly becomes complex and I switch to Django. So for example a small API without user database is good job for flask. A blog like app where I have to deal with users, registrations etc is a good job for Django. - -- Greetings Ricardo Band https:// www.ricardo.band mailto:// email@ricardo.band xmpp://jabber@ricardo.band -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEhyzgFNiV8yZuI1T5Gp1lfQa1gg4FAl7KvxMACgkQGp1lfQa1 gg6Nig//R57GeMhSnux2yqwJs1fJCNwot3tQJMdWTBBsYKAlEq/cEcONsoynDlRR 0TilISlsECN2T4GkdDk5aQGcacgx2CPvdlaBhDCbDd7ik7fjaU+g+/3itt7bduwy qIW83QndTNFAQh+dZO2LX1nzQIb0woPbrNTvA4yuDkoa4zNDQgqayBHP9AKsjvK+ 3qDSy2R2DsnvsXpJcppnGX5uk4q/3HL5AGBWTdO+FFidGwLVAFC4S3RTI2ptbSqL rsHccogOyA9ytTrKV3sILM2ryqVaQIsJUVfST3eyY3uxdnYqiCZEk+LPZdBZDsFU guyq4VfmLDgMvmMa+fCYwb5K3noXcgJtu0RBtFlfQYRKsXUmJioPwJzZu8uRPaGh LTbIPFI+4xqvDF93e2k0sI3zk88ZKVI5sHAohcAE1vghdha3sngsqZTzCR6mllWv QzyEUzyg4CfLAg2fEJeCYIqj3rviL/jonZYbp7rdeBlB6VMcDgLNj1f8d4RETI/e yE0LagwoEkgWFJ8IQV+AabmdsoRBX/XMObjPRolkKK7WlHNWRLsdiS1vY71JTVA0 wi9PhkRsqL2Kt1hLYK1MQk9golZ8ptvhtKYGtLdKEyIPRcoUrHgQyeH8LK8e1hTi fgfSCRDydA/k2LFlO0H7xub+5QV+P5+9lnxxIQwG7qbuqyl1gRk= =3c1A -----END PGP SIGNATURE-----
Hi, While I don't plan to work on the new AUR, I have experience with both Flask and Django. I generally don't like the kind of comments that go "it would be nicer if you do X and Y" from people that won't actually participate, but I still feel it's relevant: On 24-05-20, Ricardo Band wrote:
I'm a python programmer for a while now and I really like the small size and simplicity of Flask. But let me give you one advice here. If you're dealing with a more complex application Flask tends to become more complex and very hard to manage. All of a sudden you integrate about 10 Flask addons. Some of them are not well maintained. Some droppen support for others etc. You have to manage all those dependencies.
Personally I prefer Django in more complex projects as it comes with everything included. You rarely have to add something to your dependencies.
I completely agree with this. Flask is fantastic for simple projects. But once you start adding users, authentication, an admin panel, command-line scripts, and so on, it quickly becomes a mess. Django is much more structured, which is a big advantage for complex projects. Regarding databases and migrations, Django is much nicer to work with than SQLAlchemy / Flask-SQLAlchemy. For instance, with Flask-SQLAlchemy I've had serious consistency issues between PostgreSQL and Sqlite, something that I had never seen with Django. Also, the Flask ecosystem is not exactly bustling with activity. On the positive side, this is because things are remarkably stable and working as intended. On the negative side, it means some amount of bit-rot and unmaintained projects, although it's clearly not to the point of complete disrepair.
My personal rule of thumb is to use Flask for everything simple. When users are involved it immediatly becomes complex and I switch to Django.
Very nice rule of thumb indeed!
So for example a small API without user database is good job for flask. A blog like app where I have to deal with users, registrations etc is a good job for Django.
With all that being said, if the people that will be developping the new AUR don't know anything about Django, it would mean investing time to learn Django. Baptiste
Thank you for your comments Ricardo and Baptiste. On Tue, 02 Jun 2020 at 09:07:21, Baptiste Jonglez wrote:
I generally don't like the kind of comments that go "it would be nicer if you do X and Y" from people that won't actually participate, but I still feel it's relevant:
One of the reasons for starting this thread was to give non-core developers the opportunity to step in and make alternative proposals or raise concerns, so no need to worry about that.
On 24-05-20, Ricardo Band wrote:
I'm a python programmer for a while now and I really like the small size and simplicity of Flask. But let me give you one advice here. If you're dealing with a more complex application Flask tends to become more complex and very hard to manage. All of a sudden you integrate about 10 Flask addons. Some of them are not well maintained. Some droppen support for others etc. You have to manage all those dependencies.
Personally I prefer Django in more complex projects as it comes with everything included. You rarely have to add something to your dependencies.
I completely agree with this.
Flask is fantastic for simple projects. But once you start adding users, authentication, an admin panel, command-line scripts, and so on, it quickly becomes a mess. Django is much more structured, which is a big advantage for complex projects.
One thing I should have clarified much more in my first email in this thread is that this is part of a dual stack solution, with most of the code still being written in PHP and only specific pages being handled by the new framework. Everything we do (including database access, user and session management, ...) must be fully compatible with our PHP implementation, hence we would almost certainly not be able to use any of the more sophisticated features that these more comprehensive frameworks provide.
Regarding databases and migrations, Django is much nicer to work with than SQLAlchemy / Flask-SQLAlchemy. For instance, with Flask-SQLAlchemy I've had serious consistency issues between PostgreSQL and Sqlite, something that I had never seen with Django.
Same here, everything needs to be compatible with our existing PHP code base; and we're already using SQLAlchemy for migrations there (even though this would be not too hard to change at this point if there is a strong reason for doing so).
Also, the Flask ecosystem is not exactly bustling with activity. On the positive side, this is because things are remarkably stable and working as intended. On the negative side, it means some amount of bit-rot and unmaintained projects, although it's clearly not to the point of complete disrepair.
As you may have noticed from some of the patches submitted to the ML, we decided to use FastAPI over Flask. Best regards, Lukas
Hi, On 02-06-20, Lukas Fleischer wrote:
One thing I should have clarified much more in my first email in this thread is that this is part of a dual stack solution, with most of the code still being written in PHP and only specific pages being handled by the new framework. Everything we do (including database access, user and session management, ...) must be fully compatible with our PHP implementation, hence we would almost certainly not be able to use any of the more sophisticated features that these more comprehensive frameworks provide.
Right, I had not understood this would be such a strong design constraint. In that case, yes, Django is clearly out of the loop. I understand the advantages of a gradual rollout. However there's a risk that the new code can end up being inconsistent, with possible security impacts. You will have to really double-check any code related to authentication, autorization, privilege, permissions, etc. "Heavyweight" frameworks usually take care of that for you in a consistent way, but here you will be mostly on your own.
As you may have noticed from some of the patches submitted to the ML, we decided to use FastAPI over Flask.
Indeed, I didn't know FastAPI. It looks like a more fancy/modern Flask, which is a good sign. Baptiste
On Wed, 03 Jun 2020 at 17:37:00, Baptiste Jonglez wrote:
Right, I had not understood this would be such a strong design constraint. In that case, yes, Django is clearly out of the loop.
Yes, should definitely have been more clear about this in the initial email, especially since I was asking for comments from non-contributors.
I understand the advantages of a gradual rollout. However there's a risk that the new code can end up being inconsistent, with possible security impacts. You will have to really double-check any code related to authentication, autorization, privilege, permissions, etc.
Good point, we are aware of that and we will be trying to minimize the amount of duplicate code. For example, logins will always only be handled in either the PHP code or the Python code, but we need to validate session cookies in both code bases of course.
Indeed, I didn't know FastAPI. It looks like a more fancy/modern Flask, which is a good sign.
Thanks again for your comments!
participants (4)
-
Baptiste Jonglez
-
Filipe Laíns
-
Lukas Fleischer
-
Ricardo Band