Small frame long stem vs Large frame short stem

I currently ride a 56cm frame with a 110mm stem. Here's a question for those of you that have had an opportunity to ride the same frame in multiple sizes: Assuming the same seat height, how different would a 54cm frame with a 130mm stem, or a 58cm frame with a 90mm stem feel? Is the primary difference in the steering speed (or stability/twitchiness if you want to put it on that spectrum) or are there other factors to consider? Weight?

I'm always on the prowl for Craigslist deals, but I don't want to go up or down a frame size if it's going to feel significantly different than the setup I've spent so much time dialing in over the winter.

Thanks,
Skye