If you regulate the proxy system accordingly (felonies for fraud, paying for votes, etc.) then you could many of those pitfalls of which you speak. In any event, our current system favors the rich and those with privilege to vote so I'm not sure it's much different than what you envision.
Who says that the punishment for compulsory voting is police or military enforced voting? Compulsory voting could be a legal requirement with little to no teeth, similar to the ACA tax. It would push folks to vote even if the consequences were minor.
The catastrophizing you speak of isn't material imo, and shouldn't be excuses for us to not explore how we can improve the existing democracies of today.