I think the problem Mr. La Jeunesse has is with the word "public." For such an innocent-sounding word, it actually has two very different definitions:
1) The general population; people at large.
2) Those parts of the country which are maintained by the government for the people at large.
So, for instance, a "public" statement can be made by a "private" individual, if there's some person speaking out loud on the street. But this is quite different from a statement made by a "public official," which is a government employee.
Religion will NEVER be banned with regard to the first definition of "public"; it would be blatantly unconstitutional, not to mention extremely oppressive. It would mean that individuals ("private" individuals; see how confusing the terms can be?) aren't able to practice their religion freely, which is indeed horrible.
But in the areas described by the second definition, there should be no religion. Take a public library, for instance. How would you feel if a librarian refused to allow any books on, say, bacon in the library, because she happened to be Jewish, and thus it conflicted with her religious beliefs? You might argue (rightly so) that since the library is a public facility, there should be no influence of religion in its operation. Now in our hypothetical example the librarian was Jewish, but the same principle applies regardless of the actual religion in question.