That probably depends on the scale of it. If it's just a few people coming over for a quiet evening, that'd probably fall under some reasonable definition of normal residential use. However, I remember a case that happened in Toronto a year or two ago where hundreds of people were visiting a house every weekend for worship services in a large multipurpose room in the house. The neighbourhood would be overwhelmed with parked cars, traffic and noise every week. I don't think that this person's neighbours should've had to put up with that... and in the end, By-Law Enforcement decided that they shouldn't.
As an example between those two extremes, there was a sort of shrine two doors down from my house while I was growing up. It was very popular with local Catholics, especially the Philipino community, to the point that it was often difficult for our guests to find a place to park on the street on a Saturday night and on a few occasions, the street was actually blocked by tour buses (yes, tour buses... Greyhound-sized Prevost buses on a small residential crescent). We never complained to the City, but what went on was certainly disruptive and inconvenient for us.
I don't think it is analogous, because the main reason I gave the garage sale example wasn't because it's infrequent; it's because a garage sale is ancillary to a residential use: at a legitimate garage sale, you aren't trying to make a living, and you aren't selling off inventory you specifically bought to sell; you're just trying to offset some of your cost of living by getting rid of unwanted items.
I think a one-day religious service is a lot more like a one-day trade show or seminar, which very well could be done as a business, even if it's infrequent.