Is it at all possible to switch the manual heater control to that of the XLE with the climate control?

I know that pretty much all of the climate control units come with a wood fascia, but I'd look into covering that.

 

With enough time and money most likely yes, however.

For 2003 it appears the XLE only offered this. Need to check if the SE wiring harness has provisions for it which speculating is no as this car uses different components requiring different wiring.

The auto system is integrated into the car and has more components, it is not just swapping a few parts. If the wiring is not there, then this will be a bigger job. Plus require working under the dash which is access limited.

You may need to find an XLE wiring harness and all components (from a XLE parts car to save money), then blend this with the SE harness.

It has been done on Toyota, but not without a LOT of work. If you do go this route, my opinion is the job will be very complicated, requiring a lot of work. The reason being the SE and XLE were built with different systems. Trying to convert one to the other is beyond just swapping parts, plus given the car is built there will time consuming access issues.