I’ve used the response selector for handling my FAQs. There are a couple of issues with that, one is about not being able to trigger the Fallback classifier or policy for the response selector and also there is no way to consider any threshold for the retrieval intent class to trigger any fallback.
The intent classifier finds FAQ with quite very high confidence even for distinctive queries.
@aLoneStrider First, I believe your ResponseSelector overfits if it misclassifies w/ high confidence. Therefore, you should probably tweak such hyperparameters as drop_rate, connection_density, and maybe couple of others to control model complexity. If it doesn’t help, it makes sense to double-check your NLU data for mislabeling and/or intent confusion. Also, you might want to save model checkpoints and evaluate response accuracy on validation set to further select the best model via checkpoint_model, evaluate_every_number_of_epochs, evaluate_on_number_of_examples, and tensorboard_log_directory.
As for fallback for retrieval intents, if reimplementing ResponseSelector is not an option, one (but possibly not idiomatic) way to monitor responses is to implement a custom action for FAQ and replace utter_faq w/ this action in the corresponding rule. This action should retrieve response metadata from user events and follow up a fallback action if fallback conditions are met.
P.S. Maybe a custom meta-learner can help as well? 
The response selector has limitations, including the inability to trigger the Fallback classifier and the lack of a threshold for retrieval intent class confidence. These limitations can lead to high-confidence matches for queries. Consider exploring alternatives and reaching out to platform support for possible solutions.