Yes, that is true... wait, no it isn't!
I honestly don't believe that to be the case. For the most part, such humour is based on the forbidden - which is certainly my angle. Sure, some use it as a joke cos they think it's true and funny. It's up to you to get what the person's intentions are. Or you could just point and shout racist every time, that does sound more fun.