tl;dr: follow standard practices — there is nothing special about passwords for private keys.
I want to publish a package repository with some packages that I need and only want to build once for all my systems.
I want to make the packages available for general use. I have server space for that so I only have to rsync my final repo to my server after compiling my packages.
I have my autobuild set up and signing seems to work, too.
For convenience, I decided to make the passphrase not too long.
This alone makes me raise an eyebrow and wonder, if the security is already compromised.
I have 10 characters with both, alphanumeric and "special characters".
Is it coming from a proper CSPRNG or an unbiased random source?
If not — in particular if was your brain that generated it, you have applied any changes to „make it easier to remember” or chosen one from a set of random passwords — you are close to having no password at all. But if it properly generated, it is meeting the often repeated password criteria: 8 characters in the past, becoming 10 nowadays.
But that doesn’t mean it is fine. Random, compact passwords are hard to remember. Unless you’re using a password manager, you’re going to either make mistakes (like writing down the password) or you’ll undetake an unneccessary effort for little gain (remembering it). There are better ways. See diceware and friends: it lets you generate a password with very good entropy, but being easy to remember.
If you’re using a password manager, you should not care about the password being “too long”. After all it’s not you who type it. Go for 16 or 20 random chars.
I think if the passphrase is meant to be uncrackable alone, then we wouldn't need the big private key file, right?
Those topics are unrelated. The password is only used to protect the key in case of a leak and plays no role in security based on that key. If the key is breakable, whether it is protected by a strong or weak pasword, or not protected at all is insignificant. The attack will not even consider the password.