I've been the victim of card fraud in the past where they drained my current account and all my direct debits for the mortgage, council tax etc., were due out. The bank gave me a temporary increase on my free overdraft so my bills were paid. It took a couple of weeks to sort out getting my money back.
I now don't use the debit card on my main current account. That's still used for wages going in and bills coming out on direct debits.
I've got a second current account without an overdraft that I call my spends account. I just top it up with small balance from my main current account and use that debit card for spending with. If that card was lost or stolen, they'd only be able to spend a small amount and I'd still have access to the money in my main current account.
It takes seconds to transfer money into on on my banking app.