But look at what you just wrote-- "IN THEIR OWN COUNTRY." What makes it "your country?" That you are a citizen. And who is "we?" Not humanity at at large, but your fellow citizens.
This is "your country" so you shouldn't have to deal with certain conditions. And if that means that other people have to go back to "their country" and live under even worse conditions, then too bad. I mean, not that you don't feel bad for them, but basically if it's you vs them, you're picking "you."
If this weren't the case, you could simply decide that NO ONE should have to work under certain conditions (in the UK or anywhere else) and put your effort into supporting the UN Human Rights Council or Amnesty International or something like that. Because withdrawing from the EU ostensibly to help British workers certainly isn't doing much for the migrants. It's just helping British people (supposedly). So it's totally about immigrants. It can be about immigrants without being xenophobic or racist, but it's still about immigrants.
Also, you're not going to get those jobs back by withdrawing from the EU. TBF, you wouldn't get them back if you demanded living wages, either. They're just gone. Until the whole world decides on rules that apply to everyone. Until then, manufacturing businesses will always use the cheapest labor. If it isn't in England, they'll just go somewhere else. They don't care who they pay or where they pay them, as long as it's cheap.