You are right I copied the wrong override example. This is the correct setup:
services:
rasa-worker:
volumes:
- ./directory/my_custom_component.py:/app/my_custom_component.py
Preferably you don’t edit the docker-compose.yml file directly because it will get overwritten when it is changed in a rasa-x update. Just place a docker-compose.override.yml file in the same root directory and docker will sort out the overriden settings.