Yep, that's the problem. The "well while I'm in there I might as well...". Eventually you get to the point where you've dumped so much money into it that you've passed the point of no return. But in my mind, once you get past 35s, you're really not dealing with a street vehicle anymore. 37s aren't going to fit without some stronger (and wider axles). At that point you may as well put on coilovers since it makes more sense. But you're also going to want to stretch the wheelbase... Where does it end?
Like you, I often think about just buying a second TJ or LJ and getting it for cheap enough that it's sole intention would be just an off-road vehicle, period. I think that it's kind of hard to have a vehicle that's great on-road, but also great off-road.
I remember when I used to ride motorcycles, all these guys would be looking for the perfect Enduro bike. One that had amazing off-road capability, but was also a joy to ride on longer highway trips.
What it always came down to was that there was no such thing, it's just not possible. You could have something that was a happy medium, but it's as the saying always goes, "jack of all trades, master of none".
And for that reason, I think the idea of a dedicated off-road trail rig makes a lot of sense. Which is ultimately where I think I'll get to eventually if I were to purchase a second one.