I drive an '04 4X4 and my wife drives a '99 4Runner. We haven't had a problem with either one. Depending on what we're doing, we'll use one or the other. The 4Runner definitely has more horse power, faster pickup/acceleration, and better gas milage. The rear window can be moved up and down (Xterra's don't) which comes in handy for long stuff that needs to hang out the window a bit. We use the X for off-roading, getting to trailheads for backpacking, pulling our Sea-Doo,taking the kids to the beach, driving up to the mountains, etc. If you are choosing between a 4Runner and an X with about the same mileage, I think the Toyota will cost more. As far as style, the X has, by far, more personality (minus the '05's, which is an "Ele-Scape-ish" boxy-blob that blends in with every other car on the road--purely my opinion). When I have a choice between the two, I always drive the X!
Good Luck!