It depends what kind of terrain you are going to tackle...you can still do alot of "moderate" off-roading with a 4x2, what alot of people don't know about the X's is that, ALL XTERRA'S ARE BUILT FOR OFF-ROAD....but if you going to tackle "4x4" type of terrain (mudd, deep sand, big rock trails), I would take along a buddy with a 4x4, and plenty of rescue equipment (just in case)....I take my X to the construction sites and uncharted land for surveying purposes (cause of my job)...with no problem...my advise is just do it and use your own judgement of the types of terrain you are going to, and take a 4x4 buddy along...
P.S. - maybe get some all-terrain tires for off-roading....