Not really, you can’t just get rid of a leader, someone has to challenge, and more than two candidates can be involved in the final, preferential vote, ballot.
It’s up to the parties how they choose their leader. What rankles (as it would if it was another party in power) is that the party is in effect choosing the Prime Minister. Given that the role of PM is increasingly ‘presidential’ and the role of party leader hugely determines the policy platform and style of the party - I don’t think it can be denied that Johnson’s popularity combined with Corbyn’s unpopularity rather than abstract Tory v Labour debate gave the Tories their current huge majority - the old ‘you voted for your local MP, not the party leader’ line might be technically true but doesn’t really hold water. With the nature of modern political leadership a change of PM should probably require a general election (in which I won’t vote, like you I think) because the existing majority was Johnson’s majority (as he has pointed out) and based on a policy platform (don’t blink you’ll miss it) which the two current candidates want to replace with what they call their own ‘radical’ agendas, which no one but party members have had a chance to vote on.
I seem to remember that this was always the dream of the old hard left and their infiltration of Labour - some unwitting Trojan horse leader gets made PM and is then kicked out by radicals with a completely different programme.