Before coming to any conclusions this will have to be fixed
It should handle better with the swaybar connected. It will not handle better with an AR.
Reading the new information you provided, it sounds like you need to get several things fixed, including alignment, before giving up on it. Try to set your caster around 6. 6-7 is better.
I just had the alignment done, i.e why I knew it got knocked really far out, out of curiosity, why would the caster affect the ride quality?