get some good tires and join up with an xterra club in your area. i'm sure there are many 4x2's that go with clubs.
you may not be able to do everything a 4x4 can, but you can still have fun. there are times where i'll take my xterra out of 4x4 while offroading to see if there's a difference. yes of course, it makes a difference between making over particular areas, but it doesn't mean you have to stay on the street . just remember that you have a 2wd and be realistic about certain obstacles.
and get rock sliders and underbody armor. with or without 4wd you will have instances where you'll need them.
oh yeah, and welcome to the club.