The best part of the fediverse is that anyone can run their own server. The downside of this is that anyone can easily create hordes of fake accounts, as I will now demonstrate.
Fighting fake accounts is hard and most implementations do not currently have an effective way of filtering out fake accounts. I’m sure that the developers will step in if this becomes a bigger problem. Until then, remember that votes are just a number.
Arriving and Departing are also part of ActivityPub but I don’t see anyone sharing their travels here. It’s up to the server software or instance itself to determine what kind of ActivityStreams activities are supported or not.
In fact, there are explicit Like and Dislike activities, yet some software chooses to implement upvotes through boosts/favourites. Both are valid options in my opinion, especially if interoperability with tools like Mastodon are desired, but this goes to show how you can easily deviate from the spec.
If you really want, you can make a perfectly compliant ActivityPub server without any kind of text content. I can imagine a Strava-like app sending location updates and using departure/arrival to mark the start and end of a run, for example.
Disabling votes and relying purely on comment count wouldn’t be that hard. On the other hand, comments can be generated automatically by copying them from other articles/servers and users can be generated on the fly, so that wouldn’t solve anything, it’d just make it slightly harder to deal with the issue.