Who decides what average is?
I’d say that it’s easier to be below average male, than it is to be below average female. Women have more pressure from society to be pretty. Men can be rugged, interesting, weather-beaten, etc instead of pretty.
Looks isn’t everything. Ugly people often become comedians, musicians, singers, actors, thereby moving the focus from their looks to what they are doing instead.
Geoffrey Rush (married), Steve Buscemi (married), Joe Pesci (divorced), Rowan Atkinson (divorced), Paul Giamatti (married), Ron Perlman (married), Luis Guzman (married), Marty Feldman (was married), Vincent Sciavelli (one divorce, was married at his death). Those are all odd-looking blokes who made something of their lives. Some are even famous FOR their odd appearance, and none are/were hermits or monks. And all of them found love, at least once (and no, I did not remove anyone from my list after finding out that they weren't married; all the ones I chose were married).
To function in society, you need social skills, not looks
To be a friend with women (and men, for that matter; there's no big diff), you need to be friendly, not pretty.
To be invited to parties, you only need booze. Lots of parties are “everyone’s invited” and few check invites at the door.
To be treated with respect, you have to show others respect
Proof that you’re an imbecile.