You have to be thinking about speed from the beginning.

You don't actually have to make it fast, but you have to be designing w/ an eye towards eventual speed. Otherwise you end up w/ a mess that can be made faster, but never fast.

At least that's been my experience.