This is probably biased opinion, but I was in the same position as you when I bought my 2002. Should I get an S/C or non-S/C? I knew I wanted a SE no matter what, but I couldn't decide if I wanted if I should get the supercharger or not. Still, to this day, I'm happy with my decision to not get an S/C. With gas prices as high as they are, I couldn't afford putting premium in my truck all the time. Also, many S/C members will tell you that they are getting horrible gas mileage. I only had that problem around my 30K checkup. Once I did that, it was back to normal again. I'm sure the same thing will happen at 60,000 (i.e. spark plugs and stuff). Finally, I'm sad to say that I'm selling my Xterra due to a new baby. I thought you might want to check it out because you'd be hard to find an Xterra from that year in better shape than mine. I'm not just saying that because it's for sale, I'm saying that because I know how much work I've put into keeping it clean. Anyway, here's the XOC link if you're interested:

http://www.xterraownersclub.com/ubb/ultimatebb.php?ubb=get_topic;f=25;t=004690

Whatever Xterra you get, welcome to the club. This place is amazing and will be all you need to get info on your truck.

EDIT: I should note that my X is for sale on eBay and it's in the last hours (ends tonight).