> People directing traffic use only a handful of signals.

Sometimes. Other times they confusingly gesticulate or just shout out things, or even give conflicting signals. Humans can interpret these without much effort but it's a hard AI problem.