For serious off road use it is pretty much a must to remove them. For moderate/light off road use it's probably no big deal. But I think removing the bare makes a huge difference in handling, personally. I had a Frontier 4x4 for a year before trading it in for an X and I noticed the tighter feel the X had in the rear immediately. The back end on the Frontier feels very numb when when cornering during city/highway driving. I actually like the feel of the sway bars connected than removed, but this is probably subjective to the driver. Most of my off-roading is through mud, sloppy trails, fields, etc so I leave them connected.

------------------
01 SE 4x4
Silver Ice