So are you talking about Stanislaw Lem and his novel Solaris?

Where a world spanning alien ocean of goo, in a distant solar system, creates artificial humans to mess with people researching it.

https://en.wikipedia.org/wiki/Solaris_(novel)