Not sure if it makes a difference or not, but do you have it mounted on the yakima bars or with mighty mounts? On my last vehicle I had the bars on and they whistled like crazy. I put a fairing on and that helped, but with the Xterra all ready have an air dam, I think adding a fairing would look a ridiculous.
I think the only way to avoid that whistle is just leave the ski rack off until you want to use it. The added air resistance doesn't do your gas mileage any favors, but maybe that doesn't bother you.