There is absolutely no reason to loosen the park brake cables when adjusting the drum brakes. Unless of course you previously over tightened them thinking that was your problem.
Actually, as the brake shoes wear it creates a larger gap between the shoe and the drum, this accounts for the added park brake lever movement. Backing up with the park brake on does not stretch the cable it just wears the brake lining down no more or less than riding the brakes.
Cable stretch does happen over time, it's the nature of the beast but brake shoe wear is the most common cause of long brake lever travel.