The Doctor just ignores the Outsider. It wasn't worth getting into the finer details of how time travel works - and that it's not connected to 'magic' - and he assumes the other wouldn't understand anyway. Especially since the Doctor was fairly certain it was science that brought them to this time.

The Doctor does look over at Corvo. "If you can throw it so it passes through the bars, then that skill might be put to better use, don't you think?"

It wasn't a challenge so much as an observation. They all had different talents. Maybe they could use them to get out. Question was, how to coordinate it so that the guards didn't know.