I think a significant part of this is rooted in how those two sentences are usually used socially to denote different things. Men routinely describe their wife being upset at them in any capacity as yelling at them. When a woman says her husband is yelling at her she usually doesn’t mean that he got upset that she said something insensitive about him, she usually means that her husband had a bad day and came home to hurl misogynistic slurs and vague threats at his wife. Or that she didn’t want to have sex and he became enraged at her.

Verbal abuse is a serious issue that also happens to men but would usually come with some kind of clarification from the speaker. As it is socially less common for men to feel comfortable enough to talk about their experiences with physical and emotional abuse. Women by and large suffer on an almost entire class basis from these things, and our language reflects that. Nobody decided it is that way, it’s a byproduct of violence against women. The most accurate application of the language would be “my spouse is verbally/mentally/emotionally abusing me” and it does yield pretty consistent results for husband/wife.