A character wielding a sword does not disclude it from being an SF story - Snow Crash and The Book of the New Sun are indisputably SF, yet the character in Snow Crash (Hiro Protagonist, what an awesome name) wields a pair of katanas, while in "The Book of the New Sun" Severian wields Terminus Est, a large claymore used for beheadings. And psychic powers are part of SF too. Not the sort of SF I am interested in, certainly, but SF nonetheless.
Regarding your main argument, a lot of SF stories are not steeped in science. In fact, some of my favourite SF stories have next to no science in them whatsoever; they use SF as a vehicle to examine moral and philosophical ideas that could not be examined in any other genre. Philip K Dick, for example, is unquestionably an SF author, but his writing contains very little science at all.