I've heard that the Roman use of "barbarian" was a borrowing from Greek, who used it to describe anyone who wasn't Greek, under the logic that foreign languages just sounded like "bar bar bar" all the time anyway. (Classical Greeks could be pretty snobby when they felt like it.) In any case it was definitely used to refer to, well, everyone who wasn't Roman or maybe one of their client states - Gaulish tribes, Gothic tribes, Celts, Berbers, what have you.
Besides, if the (Western) Roman Empire fell to the Berbers, why were the first provinces to go the ones north of Rome, while the last was Mauritania, of all places?
Also, you'd think after 400 years of Arabic occupation, where the adults are brutally suppressed and the children taken as concubines, that there would be much less of a distinction in phenotype between Europeans and Arabians. Even if their depredations were limited to certain classes (why bother maiming peasants when they're busy producing your food anyway?), there ought to be social, genetic, and linguistic traces of that sort of thing, the way that many peculiarities of English culture can be traced to Norman occupation. And there are... in certain parts of Spain, which were held under Moorish occupation for a long time (although I have no reliable figures on maiming during that time).
If the Arabs - well, Moors, but why split hairs? - occupied all of Europe, why don't we see that pattern of cultural changes reflected anywhere but Spain, in either historical or contemporary sources?
@The Reptilian Jew: The rise and fall of the Ottoman Empire is particularly interesting in light of the Crusades, because the Turks would never have captured Constantinople at the height of its power. One of the big factors in its decline was that it was sacked... by Crusaders. (The Fourth, if I'm remembering correctly, but don't quote me on that.)
