There has been, since the later 19th century, a technical term coined specifically to refer to anti-Jewish feeling. That term is "Anti-Semitic." It was not established in the generic. Semite, in the intent of the word's creator, referred specifically to Jews.
anti-Semitism | History, Facts, & Examples
If you would like to go back and time, and correct him, feel free. Until you do, the term will be used to point to a feeling against Jews.
Language is dynamic, words change usage and meaning per how people use it.