I’ve asked myself that question when white people make racist comments. As you point out, it’s being white, or male in your case, that’s all they need to go on. I do speak up, and have lost “friends,” but we can’t act like this stuff is ok.