It is not that hard to adjust, just turn it in a little (as Ian had mentioned), and if it starts to get tight as you adjust it in stop! And back it off...as it should be adjusted to just before it starts to get friction.

BTW my X wanders too...and it has been somewhat abused off-road.

Tightening the screw = clockwise (its standard thread)