I don't recall any major structural failures or peeling with plastic wheel caps in the past, but I can't remember if there were cracks since it's been so long.
In case you are interested in ever skipping caps, as a reference, I've attached photos of the wheels that my TJ came that didn't have caps.
Edit: The system inserted them rather than leaving them as an attachment as I intended.
View attachment 710567
View attachment 710570
View attachment 710571
View attachment 710572