I'm trying to create basically realistic mermaids, and working on developing body language and vocalizations. I have most of the vocalizations I can think of (suggestions are still appreciated), but definitely need a lot more body language. Ideas?

Edit: They are a highly social colony species, so it should be somewhat complex, and interactions between colony members would be especially helpful. I'm also trying to avoid overly human interactions such as kissing or holding hands.

They could use clicks and whistles like a dolphin, or they could also communicate through bioluminescence like a bobtail squid. They could also cummunicate through whale song.

The reason we rely on our hands and faces so much when communicating is because they are very good for it. The hands and face have lots of muscles allowing for very precise control. This also means our brains have very good ability to control hands and face. And ability to understand movements of hands and face by others. The two are linked due to mirror neurons.

So unless you want them to be very different from normal humans when it comes to hands and face, which you probably would have mentioned as it is kind of critical, their body language communication would heavily rely on hands and face and work fairly similarly to human body language of hands and face.

Some differences.

If you are floating in water instead of standing on a solid surface large movements will have unintended consequences so body language will avoid large movements and focus on just hands and face. Same applies to physical contact such as holding hands or kissing. I think such would be preserved for private moments when you are entangled and not moving.

Underwater sound carries better and light worse than in air. So there should be clearer distinction between private and close range with hand gestures and facial expressions and with public and extended range without those components.

I think those might be a solid starting point for you. Having, from our point of view, much stronger distinctions between public, private, and intimate modes of communication, possibly to the point where even the vocal language used is noticeably different is good flavour that should be visible in pretty much everything and be fairly simple to highlight without needing boring exposition.

Sonar

They might have a specialized organ much like porpoises do. It could have bandwidth enough to communicate complex states of mind, to a point where it could be considered a limited telepathy (something of the kind happens with the Tines in Vernor Vinge's A Fire upon the Deep).

Human-like body language has the problem of interfering with swimming. You could render sonar interactions as you wished, though - being to all intents and purposes a sixth sense, mermaids might "hear" it as a voice plus anything else.

Dolphins are social animals and they love to "play" in a variety of ways - some of those appear in David Brin's Uplift cycle, suitably adapted to the uplifted dolphins' increased intelligence.

Since their environment of motion is four dimensional, you could consider a complex dance as a means of communication. The bees do in the hive to indicate pertinents about nectar flowers.

These Merfolk are intelligent and presumably sentient people, so the relatively simple dances of their ancestors -- warning of dangers, locations of food sources, approach of other Merfolk -- will have evolved into a means of telling stories, describing the loot to be found in a shipwreck, courtship and any of ten thousand other things they might to dance about.