You're quite correct, barn is ridiculous. Barm, on the other hand, is a perfectly reasonable usage, as barm was originally used in the recipe. The etymology is not dubious, vague or ambiguous like cob or bun etc.
Hot cross buns are traditionally eaten on Good Friday which is a day of abstinence. Get rid of the bacon and you're good to go.
The word originally referred to bread, before sugar was even available. It also meant anything with a flattish rounded shape - as in a cake of soap. There's no specific requirement for it to be confectionery. Pancake, fish-cake, potato cake.
Don't really care what they're called if they look like them two above and are sitting on the table in front of me.