At 5000lbs, trailer brakes are not a suggestion they are NEEDED. If you try and pull 5000lbs without them, you are putting you and other people in harms way.

Upgraded brakes on the X, also wouldn't be a bad idea. Although many people don't like them because they can crack, cross drilled rotors will help keep the temp down and help reduce brake fade.

Moving is only part of the battle when towing...stopping is the other...and IMO, more important.

Also, if you removed it, put the swaybar back on in the rear and maybe even run Polly bushings. You need the back end to not sway when/if you need to make a sudden move.