A competition launched on the health ministry's website offers up to $1,000 (£780) for the best videos tackling "gender confusion".
Responding to an outcry from LGBT activists, the health ministry said that its aim was to promote creativity, not to discriminate.
The contest is for 13- to 24-year-olds.
Their entries should explore prevention and control; issues and consequences; and how to get help, said the ministry's website.
In a statement, Deputy Director-General of Health Lokman Hakim Sulaiman said: "This creative video competition is purely to tap knowledge and creativity of adolescents on sexual and reproductive health related matters and does not intend to create discrimination to any particular group."
The competition, which closes at the end of August, has three main categories: gender confusion, sex, and sex and the internet.
The guidelines cited gay and lesbian people, transgender people and tomboys as examples of people who suffered from "gender confusion".
"The very fact that they lump LGBT people under a category called 'gender confusion' shows that the authorities are very much confused themselves," activist Pang Khee Teik told AFP news agency.
"It is mind-blowing that a government agency wants the whole country to be sucked into its confluence of confusion," he said.
"I was shocked. This is encouraging discrimination, hatred and even violence towards the minorities," transgender activist Nisha Ayub told Reuters.
Homosexual activity is illegal in Malaysia under both secular and religious laws.
It is punishable by a prison sentence or corporal punishment.
In March, the release of Disney's Beauty and the Beast was postponed in the country, because it contained a "gay moment".
Despite demands from the Malaysian censorship board, Disney refused to remove the scene, featuring LeFou, their first openly gay character.
The Malaysian board eventually conceded and it was shown uncut.
A prominent Malaysian opposition leader, Anwar Ibrahim, was jailed for sodomy in 2014, after a court overturned his earlier acquittal. In elections in 2013, he led a three-party alliance which posed the strongest-ever challenge to the coalition which has governed Malaysia for more than half a century.