Okay I found the reason is because I was not familiar with Docker. I am using these two as my base docker images:
https://hub.docker.com/r/rasa/rasa/dockerfile
https://hub.docker.com/r/rasa/rasa-sdk/dockerfile
So if you take a look you will find it pre-defined the port. To change the port, just override the CMD. For rasa/rasa, it is run -p 5006, for rasa-sdk, it is start --actions actions.actions -p 5056