You've got an excellent point there, one that truly deserves examination. A great deal of the stereotypical characterization of Jews comes from the fact they were limited by law in Europe in how they could make a living. Often they were outright banned from competing with local craftsmen, traders, and labourers which as you pointed out are what is generally characterized as "honest" work. One notable and very explicit exception to this in many areas was usury lending, an already stigmatized and gradually criminalized profession that generates considerable hatred that is easy to scapegoat despite it's necessity which motivated a lot of pogroms. Incidentally this was also deeply insulting and considered sinful to shared teachings of Jewish and early Christian doctrine. Even today the very term "usury" lending has shifted from being a term for any loan that charges interest to specifically unreasonably high interest and predatory practices to make this historical trend sound more sinister.
A pretty diabolical scheme when you look at it: Making survival contingent on playing a role that while easily vilified and labelled as driven by greed and exploitation is a means to finance the local economy while placing the bulk of risk on the "villain" lender who could frankly be murdered at a whim erasing debts after successful businesses were established using loaned money that peasants with mercantile ambitions otherwise could not have dreamed of getting their hands on. The uneducated (i.e. virtually everybody outside politically invested religious and aristocratic circles) would likely know little and care less for the laws restricting Jewish business and see only what they perceive as immoral and skimming off the work of others similarly ignorant or willfully blind to how much their own economy relied on these investments.
If a venture failed a Jew ate the loss and being considered outsiders this is seen as no loss to the rest of the community. Those who generated a substantial enough immediate profit to quickly pay off their loan pushed the Jew right out of a newly strengthened local economy and made it appear an achievement of the community keeping the financier an outsider not worth mentioning or easily painted a jealous leech who demanded a cut of an "honest" man's superior business. At the mid-point? Too many businesses with stable but not extraordinary enough incomes to fully outpace the interest on ongoing debts to a handful of people who don't appear to have other more "honest" sources of income? Well then it's time to break the "insidious" Jewish stranglehold and take it all back from their (in practice nonexistent) control. No debts to dead men and a self written local story of the hero underdogs vs an oppressive menace that wormed their way into every household bleeding them dry justifying their prejudice and leaving them free to enjoy their rightful bounty sweeping their need for the loans to begin with under the rug along with the bones and ashes.
A byproduct of those practices could still be seen centuries later when it was less restriction by hard law and more by community shunning/exclusion in that successful Jewish fortunes were not typically tied to local economies, so that when an area hit hard financial times wealthy or middle-class Jewish families (or simply comfortably above the poverty line) suffered less visible impact which in turn generated resentment and suspicion making them once again easy targets of ire by a design already meant to stymie them in the first place.