Ignoring the issues around benefit payments for their children, the migrant workers tend to spend little, and then often in 'foreign' run shops, such as the Bangladeshi run Skleps, which takes it out of the local economy, before leaving taking the money with them, to be replaced by others, where the cycle continues.
It leaves a reduced number of worker in their homelands, meaning they struggle economically and need support.
Paying people less drives down other wages, which leads to less money being available to spend.
There's a billionaire doing the talk circuits that emphasises the fact that lower wages (or zero wages) are not good for the local economy, but can make a few individuals wealthy he quotes the example of I think Seattle and Washington, as City's that increased their minimum wage, and then become the two fastest growing economies in the US.
He also makes the point that what limits entrepreneurs isn't a lack of ability, but a lack of customers able to buy the product. Hence people being paid nothing, such as the slaves, hindered their local economy on a very big scale.