Well, it wasn't a familiar face out right yelling in his face or out right attacking him, but it also wasn't a friendly greeting either, which leaves him to believe that this might not be a friendly welcome. So of course when he's being demanded to come along, Sam doesn't move right away, considering if following was a good idea - he was believed to be dead once after all...

After a moment though he slowly starts to follow, minding to hang back a bit just in case. "Where are we going?"