If I want to protect against the NSA I'm worried about them using Intel ME, AMD PSP and other black boxes to hack me. I don't worry that much about them sneaking into my data center or house and physically changing my hardware.

The security you lose from having a black box in your CPU is much greater than the security you win by virtue of being (theoretically) protected against unsigned bootloaders and rogue hardware.