He's thinking on the level of "replacing IPs", not "replacing domain names". Such addresses would be like .onion domains - only becoming marginally human-meaningful with a significant computational expense. But since IPs aren't very human-meaningful anyway it's a step forward.

You would still have the problem of name resolution. However since the address would be the public key, once you had resolved the address the identity of the other party would be assured. Assuming that an attacker cannot feasibly generate an equivalent public key, you remove key exchange+authentication as an attack surface.

Key management could be a downside. If you want to update your key you have changed your address, which would look like a name-resolution poisoning attack. There are feasible ways around this but none are ideal (particularly if your key was compromised, mechanisms like signed forwarding records would become extremely hazardous). It would probably have to rely on name-resolution mechanisms similar to those used by current IP addresses.