Any other vms running on the host? We have a dedicated host box for our SW environment. I would imagine it all depends on how much data is flowing into it as well that could cause additional overhead. Sorry to hear your having issues with it. Ours has been pretty stable since the upgrade to 5.6 with the exception of having to restart the manager every once in a while.
FYI I just saw RC 5.7.0 out.