2004 XE with stock none RF system.
What if any speakers can i just add on without a problem, and is it even worth it?

I changed my entire system in my 04. went with Alpine speakers all around, Pioneer headunit, JL sub, and Eclipse amp mounted on the back of the rear pass seat. Sounds great

Go to Crutchfield website select your vehicle and they will tell you which speakers are easy fit aka (no modification required) fit.

I would say no, the old HU might not be able to power the speakers enough for it to make much of a difference, I got a full alpine set up off ebay for a really good price and did the instillation myself which is saying a lot since I know nothing about electronics.