@tyd man thanks for the feedback. I had my suspicions about the limits to an automated workflow, but it was good to have them confirmed.
Also, big props for the E2E evaluation link you posted, I had no idea that there was a rasa test command. In addition, that page provides a whole lot to digest in regards to evaluating models programatically.