Birmingham's population is just over 800,000. Net population increases have been running at over 300,000 pa, so actually, I have understated the point.
I'll give you credit for being honest about your circumstances. You live in a lovely property in France with its own fishing, drive a Porsche and you own several properties in the UK, presumably some for use for yourself and your family, and some for investment purposes. You work in England and hate Brexiters, partly because it's inconvenient for your own personal circumstances and those of your family.
So let's take a fictitious example of a Brexiter based on actual facts on what people in the UK have been facing. We'll call this individual Elsie, she's 65 years old with a family, working class and has lived in her Lincolnshire village all her life, as did her parents and grandparents, indeed all her ancestors of which she is aware. She knows all her neighbours, indeed most people in the village and will stop to have a chat with anyone. She is not a racist. When the first European immigrants arrive in her village, she welcomes them. They are perfectly decent people and add colour to the village. Elsie has always been a Labour voter, but in the early 2000's she notes with concern that unlike other EU countries,Tony Blair decides to place no restrictions on immigration numbers when the Eastern European countries join the EU.
In a very short space of time, Elsie's village is accommodating very, very large numbers of (mostly young) Polish immigrants. The local store is bought by a Pole and stocks only Polish products. The local school is oversubscribed due to large immigration numbers and her grandson has to travel to the nearest school 7 miles away. The main language in his school is Polish and these difficulties hold back his education somewhat. There's a shortage of council houses due to immigration and Elsie's daughter and her partner can't get a house and move in with her. Elsie can't get an appointment at the local surgery for weeks. When she does, the waiting room is filled with young Polish mums with their children. Elsie's husband is a good electrician but is now undercut by good young Polish electricians who are happy to live 10 to a 3 bedroom house and think the minimum wage is a fortune. When Elsie walks into her village, she only hears Polish spoken now.
There's nothing wrong with the Polish people. Elsie doesn't blame them but the quality of her life and that of her family has taken a dive. Gordon Brown is now prime minister and announces that anyone concerned about the scale of immigration is a bigot. Elsie does something she never thought she would do. She votes Conservative. She has great hope that when the new PM Cameron goes to the EU to try to restrict freedom of movement, he will have some success, so in her village, all the newcomers have time to integrate. But with typical German inflexibility, Mrs Merkel says "Nein" and Cameron comes back and announces a referendum on EU membership.
Elsie votes to leave the EU in the referendum. Therefore you hate her. There really are a lot of people like Elsie, and from your beautiful home in France, you hate her. From your perspective, that's not a good look, is it? There's a saying, walk a mile in another man's shoes. Perhaps you should walk a mile in Elsie's and understand why some people voted out.