This is actually a good point. The Mexican government made a really bad move allowing white settlers into their territories. The Whites didn't assimilate into Mexican culture and government and eventually demanded sovereignty. This caused Mexico to lose a huge chunk of her land.
The same will happen to the United States if we allow large numbers of immigrants and especially immigrants further and further away from western european values and genetics.