I've always had a lot more passion for Liverpool than England. For comparison, I don't follow athletics, fencing, judo etc. etc but have maintained the same level of interest in the Olympics. I don't know why my interest in the England football team waned.
Would just like to add though that it's a different kettle of fish being selected to represent your country and showing minimum passion to being a spectator and showing little enthusiasm. It should be a requirement for the former and an option for the latter.