That is just a fantasy. Harem is, as willowfall did write the word for the part of the house where women lived. Romans did not have any harems.
Romans married someone of apropriate family to produce offspring, divorces were common and not frowned about. Each household thet could afford it (wich was almost all) owned slaves. They were kept primarily kept for economic, not sexual purposes. They were workers, servants, sectretaries, teachers. Often old people, who stayed at the same household for all their life. The slaves of a houshold were called 'familia'. Business men, such as building contractors owned larger numbers of trained and untrained slaves as a work force. So did the big farms (latifundiae) in the countryside or production sites for anything you can think of.
Now, naturally in a rich household, owning - amongst others - some attractive women, these might be used for the pleasure of the owners, whenever they felt like it.
This was something 'normal' but nothing you spoke of. In the time of the Republic and Early Empire it would have been considered very bad taste to do so in public, just as naturally you visited a lupanar (brothel) from time to time, but this was nothing to be discussed.
During the late Republic and Early Empire some equestri or even freed-men came to considerable wealth. Those 'nouveaux-riches' were frowned upon and considered to be of bad taste and worse behaviour. As such they were a matter of jokes and comedies. Using slaves for sexual purposes and even offering them to visitor was one of the topics of these comedies to show how primitive and uncultivated these people were, underlining how uncommon and scandalous such behaviour was considered to be.
In real slavery sexual abuse always (maybe with the exception to modern days trafficking of women) always was a resulting peripheral issue, while the main purpose was economic.