Package updating database schema
Hello AUR community, I have a very annoying issue which I have not been able to find a solution to. So conventionally Arch Linux packages which require migration and updating of database schemas tend to have the following pattern: - Stop service - Update the package - Run some script which migrates the database manually - Start service However I have been packaging https://code.onedev.io/onedev/server with Bengcooper for the AUR and we have hit into a major issue. The upgrade guide, as defined in their wiki page (https://docs.onedev.io/upgrade-guide) requires the old code and the new code. This is due to the use of JSW (a proprietary wrapper) which they ship their binaries with. This does not comply with how you traditionally package software, and thus I have no clue how I am meant to integrate this into the AUR. I have in my own time been working on packaging the raw java code without JSW (cause f*ck the proprietary wrapper), however this is taking longer than I thought it would and I really need an AUR package to make things easier for me running my public OneDev instance over at https://onedev.polarian.dev Does anyone have any ideas of how I can at least get the package semi-working with database migrations, even if it is unconventional and annoying to do so? Thanks, -- Polarian GPG signature: 0770E5312238C760 Website: https://polarian.dev JID/XMPP: polarian@polarian.dev
Hi, how about a PKGBUILD that contains a script similar to if an old instance isn't already installed then install it for the very first time and exit else install it to a different path /path/to/new-onedev/bin/upgrade.(sh|bat) /opt/onedev launch /opt/onedev if exit status is 0 remove /path/to/new-onedev else echo "Everything went wrong." fi fi ? Name the "{old,new}-onedev" paths. onedev-v1.0.1 onedev-v2.7.8 etc. Regards, Ralf
Hm, This could work, install to /opt/new-onedev and then if it is an initial install (not an upgrade) I can use the post_install hook to move the directory to /opt/onedev, if it is a upgrade I can use the post_upgrade hook to execute update.sh on /opt/onedev and then delete the new-onedev (as the update.sh script copies all the data over into the installation directory). What do you think of this approach? Thanks, -- Polarian GPG signature: 0770E5312238C760 Website: https://polarian.dev JID/XMPP: polarian@polarian.dev
On Fri, 2023-01-13 at 14:48 +0000, Polarian wrote:
copies all the data over into the installation directory
I would use either "mv -i" or "ln -s".
Hello, I would like to point out this is what the script provided by the developer does, not what I write. Thanks, -- Polarian GPG signature: 0770E5312238C760 Website: https://polarian.dev JID/XMPP: polarian@polarian.dev
On 2023-01-13 10:12, Ralf Mardorf wrote:
Hi,
how about a PKGBUILD that contains a script similar to
if an old instance isn't already installed then install it for the very first time and exit else install it to a different path
No, you can't do this. At the time a package is created it doesn't necessarily have access to even inspect the eventual target system. Sometimes it looks like you can when hacking on a PKGBUILD locally, but it is incorrect usage that is guaranteed *NOT* to work in the field. I suggest a split package (or two package) approach with careful use of version numbers in depends/provides statements such that the current package always requires the one-prior version provided by the legacy package. The legacy package should be setup to run in an alternative location & name such that it doesn't conflict with the current version. Then tweak the upgrade script for use in post_install to use the legacy package locations appropriately. The PostreSQL packaging has a similar setup.
Hello, This would work but the complexity of this packaging is ridiculous, not only that, the maintenance of this would be difficult too. I will see what I can do, thanks for the help, -- Polarian GPG signature: 0770E5312238C760 Website: https://polarian.dev JID/XMPP: polarian@polarian.dev
Hey Caleb, I have been thinking this over and I think this could work, but how would it depend on itself one version behind, because the AUR does not have historic versioning as far as I am aware, so it will be depending on a dependency it could not fetch? It could be installed that you symlink the binary you want to use to the path which the systemd service uses, similar to how java does it. But I am not sure if this will work, but I guess it is worth a go... Thanks, -- Polarian GPG signature: 0770E5312238C760 Website: https://polarian.dev JID/XMPP: polarian@polarian.dev
On Mon, 16 Jan 2023, at 11:29, Polarian wrote:
Hey Caleb,
I have been thinking this over and I think this could work, but how would it depend on itself one version behind, because the AUR does not have historic versioning as far as I am aware, so it will be depending on a dependency it could not fetch?
The previous version needs to be a separate package. Check what postgres is doing for migrating data. The current version is 14, but version 13 is needed when upgrading, which is a separate package here: https://archlinux.org/packages/extra/x86_64/postgresql-old-upgrade/ -- Hugo Osvaldo Barrera
January 12, 2023 6:53 PM, "Polarian"
The upgrade guide, as defined in their wiki page (https://docs.onedev.io/upgrade-guide) requires the old code and the new code.
Postgres has this same requirement on major-to-major upgrades; I suggest you look at how it's handled in the official repositories, with separate packages for postgresql[1] and postgresql-old-upgrade[2]. ~Celti [1]: https://archlinux.org/packages/extra/x86_64/postgresql/ [2]: https://archlinux.org/packages/extra/x86_64/postgresql-old-upgrade/
Hello, I have briefly taken a look at this, but I do not think this would be ideal for how I would like to package onedev, the other suggestions seem to be better for the purpose. I will read the postgresql package some more later and see if I can use any of the ideas from there. Can I just highlight how much developers make it difficult to package code, there are some codebases which are easy to package, simple build and install, and then there are these complex installations which need less than ideal methods of installation which are very unconventional. Hope packaging this package isn't decreasing my already bad reputation for AUR packages xD Thanks for the help, -- Polarian GPG signature: 0770E5312238C760 Website: https://polarian.dev JID/XMPP: polarian@polarian.dev
I took a look at the upgrade.sh script that it uses and it seems to me that it more just handles migrating files while preserving config and handling the start/stop scripts for the service. That makes me wonder if you can get away with just not using that at all in your PKGBUILD as the PKGBUILD handles all of that stuff for you. There is the Upgrade command here [1] on line 22 that it looks like the Docker version uses to run the application upgrades. My assumption would be you could re-purpose that to run the upgrades after a new package install. [1] https://code.onedev.io/onedev/server/~files/main/server-product/docker/entry...
Hello, Yes I have seen this script already, but that would require for me to write out a script to install the package which is a little annoying to do so, this is something I plan to do when I remove the annoying proprietary Java service wrapper in onedev-oss package I have been working on (not yet pushed to the aur). I might merge -oss into the main onedev package later on if benqcooper agrees it is worth it. I have discussed how the entrypoint.sh script works with the OneDev developer already, it requires both the new binary and the old binary to patch, thus is not much different to using the JSW, just the JSW is distributed with the binaries. Thus I would rather save this for later when I work on the -oss version than worry about this now. Is there any way to reference files within the package from pre_upgrade hook? If not I might install onedev to a new location and then just patch it in the post_upgrade hook, and then delete the files after patching xD of course that would be very unconventional but it is how the codebase was written to function, even doing it the same way as it has been done for docker, the developers docker contains contain both the new update and the old version, and they patch the old with the new. Thanks, -- Polarian GPG signature: 0770E5312238C760 Website: https://polarian.dev JID/XMPP: polarian@polarian.dev
Hello, When upgrading a package pacman will remove the old files before installing the new ones, this causes issues because I cant trick pacman into upgrading the database BEFORE removing the old files (the update script attempts to detect the old code). Anyone know how to stop pacman from doing this? Thanks, -- Polarian GPG signature: 0770E5312238C760 Website: https://polarian.dev JID/XMPP: polarian@polarian.dev
participants (6)
-
Caleb Maclennan
-
Celti Burroughs
-
Hugo Osvaldo Barrera
-
Matt Harrison
-
Polarian
-
Ralf Mardorf