Well, seeing as I currently own both, let me throw my .02 cents.

Stock - Hill Climbing Cherokee all the way. Modified - Cherokee all the way.

Stock - Mud - toss up, more driver than anything.
Modified - toss up again in my book

Stock - Rock Crawl - Cherokee all the way
Modified - Rock Crawl - Cherokee all the way. That solid axle is tough to beat, plus you can mod the hell out of a cherokee for half the price of the X.

Stock - Getting babes - X all the way
Modified - Cherokee - A modified Cherokee looks beastly and is a chick magnet

Ride/reliability/comfort/ - Stock or Modified Xterra hands down. I love the Cherokee, but it feels like an amusement park ride on regular pavement compared to the Xterra. Also, I add an extra hour on my work time whenever I work on the Cherokee just for broken bolts. I swear DC makes parts out of tinfoil and peanut brittle.

The things the Cherokee has over the X is

1. Much lower gearing
2. More horsepower
3. Solid Axle (this is debatable I guess)
4. Lighter
5. Cheaper, and easier to fix

I think that if you were strictly going for an off road vehicle, a Cherokee is a very good choice. I am not a wrangler fan, and from what I here the longer wheelbase of a Cherokee makes for a better climber anyway.