Set up GTNH 2.8.4 using Prism Launcher, then join our server.
GTNH docs commonly recommend ~5–7 GB for singleplayer and 4GB+ for connecting to a dedicated server.
Double-check Prism is using the right Java, and that you’ve actually allocated enough Max RAM.