Doesn't the actual key get derived using pbkdf2 with many iterations making brute force of even fairly weak passphrases time consuming?
Arguing that weak passphrases are okay because the hash is strong is making the assumption that a password cracker will perform a naive iterative search over the space of all possible passphrases.
In practice, I believe any decent password cracker would start with a dictionary of the most common words and passphrases, based on databases of leaked passwords. See  for examples of what might be tried first.
If your passphrase is "123456" then you can expect it to be cracked instantly, regardless of how strong the hash is.