How do the tires look when they are installed? It might be possible the drum got a little distorted from being clamped down on the shafts before. But if the axle flange to drum flange to wheel flange are all true you shouldn't have any problems.
Also the drum castings can be deceivingly imperfect from the outside. What matters is that where the drum mounts to the axle is true to the machined surface the shoes ride on.