I have been debating taking mine off also because the padding covers are faded and look like crap. In my case Chrysler\Jeep did not paint the entire rollbar.
My last TJ had faded covers also, I took them off and soaked and cleaned them really well, I then painted them with fabric paint. They came out nice and held up great.