A little tale to illustrate where it starts to go wrong. Both of my lads played football at a decent level back in the early nineties. One got a no-more-playing injury and took up reffing. In one of his first games he was subjected to sustained verbal abuse from supporters of one team, culminating with a drunken father coming on to the pitch to confront him. I'm glad to say I wasn't there, but I was made aware of the incident by the coach of the opposition team who rang me to tell me about it and offer his full support when the local FA held an enquiry. This coach was a former professional player and a man of very high status in the community - both his brothers played in the old First Division and his nephew was capped at every level by England, so you get the picture.
Guess what? There was no enquiry. When the coach rang the local secretary he was told that 'the referee will need to toughen up'. My lad immediately hung up his whistle.