If two people are in a monogamous relationship where neither person has sex with anyone outside of it, but one person is interested in a religion that engages in ritual sexuality,
Before jumping to the second point, this issue needs to be opened up with the partner. How the partner views the issue, and how the issue is explained also will naturally effect point 2:
how would this affect a relationship?
There is always a strong chance it may make the foundations of the relationship deteriorate. Unless of course, the couple discussed an open relationship before, and also if both people are generally opened to alternative religion, and sexual experimentation.
What if a religion dictates that a person's relationship is not recognized by the religion when it demands sex acts with a priest or priestess?
Not sure I follow you there. Do you mean to say, for example if one partner is Protestant, and their religion frowns (to put it mildly) on the other partner's sexual rituals?
Or you mean to say that the religion of one partner demands that they should have sex with a priest or priestess?
In the latter case I would advice the person that there is a likelihood that they might be taken advantage of, and they should consider the entire spectrum of circumstances better. Especially if it means endangering a solid and productive relationship. Is the partner more important? Or a demand to serve as a sex object.