Before you get carried away, do a test on a straight road without much camber. Try to drive it straight with the steering wheel on center. Pay attention to it not wanting to be centered as in as soon as it gets to center, it pops over ever so slightly to either side requiring constant correction because you can't drive it with the steering wheel centered.
I'm going to re-check that every thing is tight, then start with the straight road test. I'll have an annual VA State inspection due in June, so when that is done, assuming it passes, my next step will be a professional alignment. I really don't want to put any parts on this until I clearly identify the issue.
There is some slight wandering. I'm beginning to wonder about the ball joints and will confirm if they are good our bad asap.The other way to think about "memory" steer is the breakaway torque is way too high on the pins in the balljoint body when they are loaded. What happens is the torque or force to move them rotationally is way too high but drops very low once they move. That leads to very wonky steering in that as soon as they move, you've steered too far and have to pull it back. The problem is when it comes back, it stops, the force goes back up to move it again, apply high force, it drops off when it moves, so you rinse and repeat with a very exhausting to drive steering set up.
Loose ball joints only cause steering issues if they induce DW, the rest of the time, all they create is a very slight vagueness and maybe some very slight wandering.