Weight. Have I mentioned weight? We don't have a huge HP or torque budget to work with, and the power available at idle or just off idle is pathetic. It's very easy to put a 1,000 pounds of crap on a TJ in an effort to look cool and keep up with the Jones's. If you want to do that, you don't want the four banger. Be very conscious of how much weight you add to your TJ SE either through the stuff you bolt on or the stuff you throw in the back. Ten pounds here and ten pounds there adds up rather quickly. Loaded for the trail, my SE crosses the scale at 3,200 - 3,300 pounds with a full tank of gas and my fat ass out of the Jeep. I work diligently to keep the weight there, or even less if I can swing it. Think like a backpacker, not like an RV'er. Ditch the hard top if you can get away with it. Take your MRE's out of the box and saw the handle off of your toothbrush. You get the idea.
A note on aluminum parts. Just because a part is made out of aluminum, don't assume that it is lighter weight than the same part in steel. That is very often NOT the case. Be a pest and check with the manufacturer. If they can't or won't tell you what a part weighs, even approximately, look elsewhere.