"I never thought I'd say this to Alexander the Great, but you're wrong. It does matter what people think of me. People thinking highly of me can actually put them in danger. I once went across the whole of Time and Space deleting every reference to me because people were dying because they spoke highly of me."

To be fair, it was a bit more complex than that, but the results would have been the same. Drove River his wife nuts when she couldn't do research on him any more.

"Simple answer? Because in the end, they all leave. Because they should. Or they find someone else. Or they die. So many of them die. And some of them forget me. Or something happens to me and I forget them, or I have to leave... I suppose, it's because, after all the saving is done, they still manage break my hearts."

Knowing people in the manner that Iskander suggests is a double-edged sword and at this point in his life, the Doctor can't suffer any additional heartbreak.