If someone were to say to you "You cannot announce the fact you are homosexual or society will segregate you to the extent of creating a petition to kick you out of their town", would you feel it fair? Because, this is the same thing. A pedophile cannot "come out" about how they feel.
Any sexuality (exluding being heterosexual, obviously) has had this stigma at some point. Seriously, if people are into hentai, they get accused of all sorts of ****. Even if the characters in that hentai are consenting, it's all because most people cant understand why someone might watch hentai.
The "Anime is for kids" argument works the same way. People dont understand why an adult would watch a 'cartoon', so they question it, while pushing it away.
Homosexuals, until recently were heavily segregated against. There was even legislation allowing the arrest of homosexuals who kissed in public! Homosexuals had a very hard time being accepted by society. I have some case studies i've done that were frankly ludicrous if you need examples, but I really doubt you do.
Pedophiles are getting the same treatment. They're heavily segregated against, and if they openly said "I'm attracted to children", they'd probably get flogged in the street. That is what is wrong. They get punished for feeling a natural urge. They are forced to be repressed, depressed, and lonely. No individual should be forced to feel this way in a civilized society.