You are not crazy I have this exact issue with my 2004 4.0 wrangler. Warms up then starts to make noise at what I would say are High Vacuum times. Just after a shift or when I'm slowing down in gear. Touch the throttle and it goes away. Play with the throttle and you can get it to change pitch...