Personally, I wouldn't do it, but that's because I know that, at some point, someone in our household would leave diesel in that gas can (me, because I'm the only one who handles diesel), and either forget about it (me), or not know about it (her), and put diesel in the lawnmower (more likely her because she wouldn't be as in-tune to the differences, but if I'm in a hurry or daydreaming, I could do it, too). To me, it's cheap insurance to have true yellow diesel cans for the tractor.
Edited to add: I know you just asked about residue, but I commented on this because I've made this mistake (forgetting what's in a container and assuming incorrectly the next time it was dispensed) with other chemicals before. I wasn't implying that you're as absent-minded as I am, so no offense was intended.
