In fact, you can flag comments regardless if you reply to it or not. Once you pass the karma threshold - I think it's 50 - then you can flag any comment.
If you want to flag a comment, you will find beside the comment headline: link
Click on the link link , and you will be moved to another page that you can see the flag link.
Thanks for the tip (no sarcasm intended). That's how I've been flagging the two or three comments I've found inappropriate but it wasn't clear to me at first. On the other hand I can see how keeping the flag link off the main comments page may be desired to prevent mass random flagging.
If you want to flag a comment, you will find beside the comment headline: link Click on the link link , and you will be moved to another page that you can see the flag link.
I hope this is useful for you.