How Do You Get Access To Avatar Phantom Gem???

So I was told recently I should casually just go round, beat all the primes, and start doing Leviathan. But I can't, because I got all the six whispers and was unable to get Avatar Phantom Gem.

So I read on bgwiki you're supposed to "turn them in", so I asked a Summoner where I can do that, and they told me to talk to the taru on top of the Rhinostery. After two cutscenes I got a Moon bauble and lost my whispers.

I went back to the NPC and still don't have the option. Now I have no whispers and a moon bauble. How does one get access to this gem if you never did anything on Summoner? My SMN is level 4.

You turn the whispers in to the taru you got the avatar tuning fork ki from.

That sounds like it. So I've lost all my whispers now for a moon bauble so I have to go beat the primes again to hand them all back in to each individual taru?

Yes. What you did got you access to Fenrir fight.

I'm going to change the information on BGWiki because it's not clear enough at all. "Turn them in" can mean turn them in to Leppe, or those tarus. I know other people who have had the same problem getting access to Avatar Phantom Gem, and I googled it and found nothing. So thanks for pointing us in the right direction!

Specifically, you have to have cleared the 6 "Trial by ____" quests. If they're under your Completed Quests list, then it's a clear. This might help you keep track if you're unsure. It's only an actual clear when you return to the proper taru to receive your reward.