Can you paste in the related errors from the mongo logs? It sounds like you are skipping the link step in the Rocket.Chat container if it can't connect at all. From the instructions:
LINKS: your mongoDB container > alias: db
Also check the port is set to
3000:3000for the Rocket.Chat container. Maybe posting all the env vars from both containers can help us troubleshoot?