Personally I love having the 4x4 for the long treacherous trips up north here in BC. The highways get slippery, and any more traction the better. Heading down to Whistler Mountain in confidence is also a plus.
To me, getting a 4x2 suv is like getting a mercedes benz without the leather interior...it's not right. Now if you just like the look of the xterra, by all means, no one will fault you for that, its an awesome looking truck...or 'pseudo' minivan if that's your flavour.