Aren't Poles white too? (and Romanians, Bulgarians, Lithuanians, Czechs... etc)
Is there a surfeit of British people doing minimum wage jobs, undercutting the local economy, or claiming unemployment benefits where you are? Or are they RICH white folk? Usually, British "ex-pats" are pensioners living on their British pensions, or adequately paid multi-national employees.
We have a lot of foreign ex-pats living here in the same fashion and nobody minds. Where people get a bit excited is when the locals are expected to work for a pittance, which means they would be worse off than being on the dole(including associated benefits), and are called lazy when compared with young Eastern Europeans, who are prepared to put up with lower standards of living in order to work.
Seems a retrograde step to me, which seeks to undo generations of improving British workers' living and working standards.