I use the term holiday quite broadly, as I think some do. Maybe it's an American thing. Even though government offices are not closed, as they are for Christmas and Thanksgiving, things that are widely recognized as shared celebration, commemoration, etc seem to be labeled as holidays, like Valentine's Day and St. Patrick's. There are non-christian religious events/celebrations that are not official days off that would be referred to as a holiday. Non-religious as well, like Chinese New Year in cities with large ethnic Chinese populations.
Interestingly, I don't think Mothers Day and Fathers Day are considered a holiday in the same way. What would be the word for these types of shared celebration and does that word better describe Halloween?
As for Halloween being just for kids I would say no, although much of the celebration revolves around entertaining children, much like Christmas in the US.