One thing that comes to mind while reading this thread is that, very often, men can be wary of other men as well. Men are not a singular collective entity or all part of the same boys' club. A smaller man might have similar misgivings about larger men, similar to what women might have. All men also have memories of being small children and having to stay out of the way of the bigger boys and even the bigger girls and adult women. That's what many learn early in life and may condition them to violence as they grow into adults.
I recall one time I was with my mom. I was around 7 or 8, kind of small and skinny. (I didn't start to sprout until my teens, when I grew to be 6'4", stronger than average, and with reasonable fighting skill. Not Chuck Norris, but I probably could have made short work out of Johnny Lawrence of the Cobra Kai.) However, when I was a little kid and walking with my mom, who was in her mid thirties at the time and attractive to males, there were two young men in a parking lot ogling her, catcalling, and stalking us. My mom acted as if she didn't even see or hear them, perhaps as her way of dealing with the situation. I was somewhat fearful and kept looking back. They stopped pursuing at some point, and my mom and I safely entered the store we were going to. But I was worried. I knew that I could not protect her from two larger men, and she could not protect me either. We were in a vulnerable, scary situation. I realized that I needed to be bigger, stronger, faster, and tougher to be less vulnerable.