I believe your problem comes from upstream vendoring the dependencies. If that’s the case, trying to separate libraries into other packages will require continuous effort from you. Could get even worse. If you fail, the users — unaware of external dependencies being your doing — will start bugging upstream and everybody will be unhappy.
While I do understand reasons for vendoring, I am myself against the idea. But if upstream decides to do that, going against their decision is pain.