(The author has donned a flameproof asbestos bodysuit.)
I think many objections to the word ‘privilege‘ aren’t about the idea itself – that some types of people, all else being equal, are treated better by society – but about it being applied selectively.
For instance, it’s been very well documented that tall people earn more money and have other social advantages. The same is true for the physically attractive, older siblings, the left handed (and the right handed), Ivy League graduates, married people, and even women with blonde hair; one could go on and on. In fact, there are so many possible types of privilege that everyone is privileged, in some sense of the word.
Thus, I think we can say:
1. Certain types of privilege, like race and gender, are talked about endlessly precisely because they are hot-button flashpoints. This leads to other types of privilege being unfairly ignored, and it also discredits the word by causing it to invariably provoke flamewars.
2. Everyone has some type of privilege, but that doesn’t mean their lives are great, or even that they don’t totally suck. For example, tall privilege is very real, but being a tall subsistence farmer in Ethiopia still totally sucks.