Performance is one thing, but i think a bigger one is that likely a number of the changes pushed by grsec would break existing kernel behavior in relation to userspace. And that is one big red line you do not cross with Torvalds.