Of course, Theravadins say that Theravada is older, purer, more authentic, etc., but it's not actually any more of those things than the Chan tradition is, nor is the Chan tradition (despite the flower sermon origin myth) any less orthodox when it comes to scriptural grounding etc. The phrase "a special transmission outside the scriptures" doesn't mean what people think it means; the Chan school has what might be the largest scriptural canon of any religious tradition (basically anything that was ever canonized at any time in Chinese Buddhism), and the actual masters and their authorized Dharma teachers know it well. It's not Biblicist in its treatment of scriptures in the way that Protestant Christianity is, but the idea that the Chan tradition doesn't bother with scriptures is patently and demonstrably false. In fact there are a number of scriptures unique to the tradition.
What's really going on here is that a lot of Westerners received Japanese Zen, which is a descendant of the Chan school, in a particular set of circumstances back in the mid-20th century that led to a grossly inaccurate understanding of what the tradition is all about. For example, people sold it as a counterculture challenge to the orthodoxy, when in the Chinese/Korean/Vietnamese tradition Chan is the orthodoxy. If you want the real thing, I'd suggest seeking out a Chinese Chan group such as Dharma Drum or else the lineage of Thich Nhat Hanh, which is the Vietnamese branch of Chan. Both of those are socially engaged, active in charity work, and promote a holistic approach to Buddhadharma, rather than trying to rip one aspect of it out of its context in the name of helping Western intellectuals feel superior. I'm sure there are genuine Japanese lineages too, but the "pop Zen" stuff is entirely Japanese in origin, so by avoiding them you pretty much avoid the problem entirely. And the fact is that Japanese Buddhism has some internal problems of its own and has had for a very long time--suffice it to say they didn't come out of the Meiji-era state persecution unscathed, and they still haven't really recovered.
For what it's worth, I completely understand your frustration with your aforementioned Zen group, which seems like a good example of everything that's wrong with Western dilettante Buddhism. The real thing is a completely different world. Also be aware that you get the exact same problem with Theravada groups in the US, many of which attract the same kinds of people, who are keen to turn Buddhism into a secular philosophy plus meditation, without the ethics or social engagement. And there are people who dabble in the Tibetan tradition that get lost in the esotericism and never extract their heads from their backsides or get around to manifesting all that legendary compassion they go on about. The key is to find a group that's down-to-earth and practice-focused, whatever the school--and that they realize that ethics and interpersonal relations are practice. In fact, the ability to live in this world and make it better for others is the entire point.