This helped for me, thanks for the tip @rctatman
Maybe this is incorrect but I noticed I have to downgrade the pip version in the respective virtual environment as well. So make sure the virtual environment with rasa is active and:
pip install --upgrade pip==20.2- run
pip -Vto make sure the right version is installed - run
pip install -U rasa-x --extra-index-url https://pypi.rasa.com/simple
Installation time was now quick and under a minute. Without performing these steps I had to wait over an hour and still no result.