Bolt or ball joint taper ?

If it's the ball joint taper, then it should not turn when loosening the castle nut.
Getting the castle nut tightened is tougher though, and you need some force on the ball joint so it can get a bit of grip. This is where an impact wrench does wonders.

Please post the video of the action, it will be a good learning video.