Men shouldn't be labeled feminists because they're not women, it's as simple as that. The majority of them, especially extremely privileged ones like Justin Trudeau, only reap the benefits of patriarchy and could never even begin to comprehend what it's like to live as a woman in a world that considers you inferior and second class just because of the biology you were born with, biology, I might add, that is essential to nurturing and bringing life into the world.
Men can help out behind the scenes, be allies, but they can be as much a feminist as a white person can be a Black Lives Matter activist. Like men with feminism, white people can only be allies and support BLM's message and speak out against racism in any and every way possible. True allies for both causes wouldn't expect a pat on the head from the groups they're supporting, nor would they claim themselves members of these groups just because they support them.
I agree with the rest of the points you've made in your post.