I would argue that the fall of the republic was the first nail in the coffin, and like you said, was caused by slave concentration in few patrician's hands (according to the Gracci brothers at least). The slave concentration around rome, and the slave owner concentration at the end didn't help the empire either.

And please don't draw parallels, you shouldn't, the society now is very different.

[edit] for clarification, the last sentence is a general 'you', I'm not talking about my parent who made an overall good point.