If somebody was, for example, to find pale skin especially physically unattractive to them, then they probably won't find white people physically attractive.
A preference isn't an absolute, though (in terms of attractiveness). I prefer darker men, but if a very pretty ginger man took an interest, I would happily adapt . Theoretically of course - I'm long and solidly married!
As an overall, this is true, and for many people it is also true. But that doesn't mean that there are not people for who the opposite is true. And there are people out there who are find all those of a certain "race" unattractive.
I feel bad for them, if so. What a lot of beauty they're missing out on! Not to mention that it has a touch of 'fetish' about it .. if you can only respond romantically/sexually to a very specific look.
I wouldn't consider a preferences fetish. Sounds like you're saying you're having sexual woman has a fetish for penises if she prefers men. Everybody has a different idea on what attractive is that seems to work out.