Well, better late than never.
Go to the dealer, any dealer, and drive another Xterra. If that feels different, you may have something. I would suggest flushing the power steering. Valves may be sticking causing it to be a little stiff at slow speeds.