In rugby league it is the referee who decides when to use the video ref. The suggestion for football is the teams manager gets 1 appeal a half. The ref has no say as to when it is used and will run the game as he currently does.
The problem is once you start something which is initially goal line technology, now they are discussing what you mention, where will it go next? I personally would prefer it to just stay at the goal line technology, as that is clear cut. If you allow appeals, will managers start to use it like a time out in Basketball, if they want to interrupt the flow of the at any particular time, by appealing a decision the referee makes?
I don't see the need for appeals, if done correctly the video ref could be sat there with all the different angles linked to a single playback at once and could give the correct call in about 5 seconds. It would just need limiting to big decisions, we obviously wouldn't want every little foul in the middle of the pitch to be scrutinised.