Stevo is correct on this one - the referee should just play on.
Any player other than the goal kick taker may play the ball once the ball has left the penalty area. The fact that the keeper didn't take the goal kick and was in his own box when the ball returned entitles him to handle it as normal.