I thought that the show was even MORE clear about this, because you can recognize Yoren by sight. IIRC, in book one, Ned gets his dome lopped off from Arya's point of view, and she doesn't recognize Yoren, and it's not like the description of him is so detailed that you go: "Oh, that's Yoren from the wall!"
Unless I'm misremembering it. I just remember finishing that chapter and being all, "What the hell!?" (for a number of reasons obviously).