What would victimizing be in your view ?
As a rule of thumb, it would be anything done to someone else that's against the law & yes, I know, that opens up another question about what ought to be against the law; in that case, the question - at least when it comes to having sex (with or without a relationship) - should be whether or not it's consenting adults.
As someone who's not religious, I don't think government ought to be in the marriage business or laws pertaining to activities within a marriage & I think constitutional separation of church and state (1st clause of the 1st Amendment) prohibits that anyways.
If someone has sex with their spouse against their will, that's not consent & that individual is a victim.
As a libertarian, I am also opposed to laws that make it a crime for things like oral sex, or having sex with someone else of the same sex; when two individuals doing such things willingly, they're not victimizing each other. Such laws are for religious control over individuals.
Originally marriages did not limit the number of spouses to just one individual, but some contemporary religious policies prohibit polygamy & laws prohibiting polygamy are also for religious control over individuals. Some cultures or religions are ok with polygamy, such as Mormonism & when the government tells them that they're not allowed to practice polygamy, they're infringing on their constitutional freedom from religion (1st clause of 1st Amendment) as well as their constitutional freedom of religion (2nd clause of 1st Amendment). Unless someone is being forced into a polygamous marriage against their will, there is no victim.