The idea is based off of the society and the structures and thoughts it built over time.
You don't directly benefit from it. You don't have any implication in it. You're just born into it, and it's not your fault at all.
To go over it further, after the release of slaves after a century of free labor - they generally had nothing to start with, in fact many actually stayed in the plantations they previously worked at and worked for nigh un-existant wages. The stretch of social inequality allowed worker exploitation to continue for a long while.
Generally that kind of exploitation is long gone. However different acts have occured in the past. Society is generally segregated rich and poor, and a strong percentage of poor are black. Housing areas are reflected on this to, there were 'red zones' for development, and so forth. Those acts are also gone, but the impact they have created still lingers.
Pretty much just general social ladder stuff. I don't know how I feel about it either, since I don't want to be blamed for whatever much some dirt hats did a few centuries ago. Ideally we'd just lift everyone out of poverty, but the world isn't nice like that.