I found the same thing until I started strictly controlling the resources each container could consume, and also changing to a much beefier machine. Running a single project with a few images were fine, but more than that and the WSL connection would randomly crash or become unresponsive.
Databases in particular you need to watch: left unchecked they will absolutely hog RAM.
This was one of the reasons we switched to docker in the first place. Our Devs with M series processors spent weeks detangling issues with libraries that weren’t compatible.
Just started using Docker and all of those issues went away