Interesting topic, and i think it will be different for everyone.
Lets take 31.5" tires and two gear sets, 3.07 and 4.56
The one with 3.07 gears is coming from the perspective of struggle, and every -50lb is a noticeable difference.
In contrary, one with 4.56 is considered sightly overgeared, you can add +100lb to their Jeep and they will swear on their life that there is no difference.
That breaking point will be different for everyone.
Gross weight of my Jeep with soft top is 4120 lb. Other than 60lb tire carrier, it is stripped off all "creature comfort".
When i was on 3.07, i switched from 33" MT tires that came with Jeep, to 31.5" AT, the difference was unarguably there. And only way to make it better was to go down to even smaller tires.
I went up and down lift heights 3 times, tried lighter oils, had days without passenger seat.. Everything was a moot point, a sort of placebo effect. Yes, unarguably every weight reduction is a gain in performance, but i never noticed that 1% or 3% difference in driving characteristics, or on the gas receipt.
I was burning more gas in 1 hour trip against the wind, than i did in a month of driving with all seats in.
The engine was struggling with those gears, period. Less struggle, more struggle - don't matter, struggle is struggle.
Eventually i have regeared... Bruhhhh.....
Stop to go - effortless.
Highway 70mph 3000rpm 6th gear, head on wind, rain, hills, trunk full of tools and camping gear - effortless.
Right now i am at the point where other than mounting larger and heavier tires, or hooking up a heavy trailer, not that much will have an effect. I am not at that breaking point where + or - 200lb in my cargo will be noticeable.
I saw rooftop tent mentioned somewhere, that is different, it is a combination of weight and resistance that only get stronger with added velocity.