Male harem

Is there a word that equals harem when attached to men as opposed to women?

I have just been chatting to Thomas formally known as my knight. we were discussing the men in my life. I described them as the male equivalent to a harem. In the past I have talked of my stable of men.

I had thought that sometimes men runaway from me. they enjoy being with me but suddenly bolt like frightened cats. This I put down to them not really liking me as much as they could. But I am beginning to realise now that I was wrong. They run because they do like me. It isn't me that frightens them but their own feelings towards me. Thomas has just promised that he wont do it again. He hadn't realised that I had noticed that was what he had done.

which best describes what I have harem or stable or is there a better way to collectively describe my men.