The Doctor reacts emotionally and that's what Cole reacts to. He draws back, looking uncertain, a far stronger reaction than the mild outward action should provoke.

"I don't want to hurt. I can't not see. The shard... blinds, but there's still shadows between the shafts, shifting shapes of..." He trails off, looking somewhere else. "This is... inside voice. Don't hear the inside voice, hear the outside voice. I don't know how not to hear."