You are right. In fact, everything escalates the situation because technically you cannot even reach these people. I think what you did was right, no eye contact and walking away. But oh man, I would have been shaking just wondering if I was being followed.