Few =/= zero. That's why I said overestimated and underestimated. The actual difference between brains of adult male and female humans is very small. Though not as small as in children, which may be because of gendered parenting, epigenetics or a myriad of other reasons.
However, the few things that have been shown to be different also shows that difference in transgender individuals matching with their gender identity rather than their assigned sex.
Transgender brains are more like their desired gender from an early age
Is it likely more complicated than this? Hell yeah. You can't look at an autistic brain and determine with any consistency which parts were altered to make them autistic, or where in the brain autism is. Why wouldn't something as complicated as sex and gender be at least as complicated?