Other containers beside it will autostart, like apps downloaded within the umbrel webui, but for whatever reason the umbrel container itself didn't.
I fixed this by tweaking the compose command using --restart=always instead of --rm , then manually adding restart: always to the .yaml config files once it was deployed.