To be honest, you could say the same about SSL/TLS, browsers, and (probably) virtualization.
Secure enclaves are new. It takes time to develop new technologies and work out all the issues. I probably wouldn't trust my data to SGX today, but I'm not opposed to it as an idea. In 5-10 years it may be in a reasonable state.
If we had a heartbleed every month, then I'd be rethinking whether I want to trust anything to TLS, yes. Browsers are awful, and tolerable only because they provide so much value (and even then, I overwhelmingly only run a browser with extra mitigations in place). And no, virtualization probably shouldn't be trusted against hostile code.
As long as we rely on languages like C for our foundations, we will have them, just not always on TLS, that is why it is so relevant to push for better foundations, even when most of us aren't doing systems programming.
Secure enclaves are new. It takes time to develop new technologies and work out all the issues. I probably wouldn't trust my data to SGX today, but I'm not opposed to it as an idea. In 5-10 years it may be in a reasonable state.