Inviting an introvert to lunch with you is likely fine and would be appreciated.

Inviting an introvert to a group lunch with six other people would likely cause angst.

And yes, the introvert probably didn't notice. They probably don't often think about you either.