A reliable way of accomplishing without switching the frontend for routing messages (which is a good way of going about it) would be defining your own policy in addition to a custom action to facilitate the communication. Give the policy the highest priority in your configuration and let it predict the action when a certain slot is triggered like handoff_active.
After you are done talking with the human agent you can unset the slot to finish the story and optionally predict some follow-up actions like feedback and whatnot.