That's because they are both in the Schengen area. Luckily we're not. If we leave would that become a condition of any trade agreement? Something else we don't know.
You're right, it's another "we don't know" area.
Switzerland have, apparently, made noises about wanting to change things, with regards to freedom of movement, only to be told that other agreements would need to be looked at, as a direct result of changes made.
I think the experts believe this to be tariffs on trade agreements, which might be an indication as to what we might expect if we go down the "stop the freedom of movement" route.
