No, we dont call Leavers racists or fascists, because not all leavers are racists or fascists. All racists and fascists are Leavers, however, and so if we did say that, we would be right at least some of the time.
Brexiters is a new one on me, and Gammon is something I've seen some Leavers embrace. Even if it could be insulting, it's nowhere near as insulting as being called traitors, or told we are betraying the country our relatives fought and died to protect.
THAT is insulting, if you want a measure to go by.