That is correct. What leaves a person without excuse is rejection of
comprehended light given of God by someone sent by
him. The prophet Joseph Smith explained this well:
"The idea that some men form of the justice, judgment, and mercy of God, is too foolish for an intelligent man to think of: for instance, it is common for many of our orthodox preachers to suppose that if a man is not what they call converted, if he dies in that state he must remain eternally in hell without any hope. Infinite years in torment must he spend, and never, never, never have an end; and yet this eternal misery is made frequently to rest upon the merest casualty. The breaking of a shoe-string, the tearing of a coat of those officiating, or the peculiar location in which a person lives, may be the means, indirectly of his damnation, or the cause of his not being saved.
"I will suppose a case which is not extraordinary: Two men, who have been equally wicked, who have neglected religion, are both of them taken sick at the same time; one of them has the good fortune to be visited by a praying man, and he gets converted a few minutes before he dies; the other sends for three different praying men, a tailor, a shoemaker, and a tinman; the tinman has a handle to solder to a can, the tailor has a buttonhole to work on some coat that he needed in a hurry, and the shoemaker has a patch to put on somebody's boot; they none of them can go in time, the man dies, and goes to hell: one of these is exalted to Abraham's bosom, he sits down in the presence of God and enjoys eternal, uninterrupted happiness, while the other, equally as good as he, sinks to eternal damnation, irretrievable misery and hopeless despair, because a man had a boot mend, the button-hole of a coat to work, or a handle to solder on to a saucepan.
"The plans of Jehovah are not so unjust, the statements of holy writ so visionary, nor the plan of salvation for the human family so incompatible with common sense; at such proceedings God would frown with indignance, angels would hide their heads in shame, and every virtuous, intelligent man would recoil.
"If human laws award to each man his deserts, and punish all delinquents according to their several crimes, surely the Lord will not be more cruel than man, for He is a wise legislator, and His laws are more equitable, His enactment more just, and His decisions more perfect than those of man; and as man judges his fellow man by law, and punishes him according to the penalty of the law, so does God of heaven judge "according to the deed done in the body."
"To say that the heathens would be damned because they did not believe the gospel would be preposterous, and to say that the Jews would all be damned that do not believe in Jesus would be equally absurd; for "how can they believe on him of whom they have not heard, and how can they hear without a preacher, and how can he preach except he be sent;" consequently neither Jew nor heathen can be culpable for rejecting the conflicting opinions of sectarianism, nor for rejecting any testimony but that which is sent of God, for as the preacher cannot preach except he be sent, so the hearer cannot believe without he hear a "sent" preacher, and cannot be condemned for what he has not heard, and being without law, will have to be judged without law." (Joseph Smith, Teachings of the Prophet Joseph Smith, pp. 220-221)
So yes, one should not fear and tremble because he rejects preachers "not sent" by God.
God will ensure that
everyone hears a sent preacher, whether they hear that preacher in mortality or after. No one will be left without excuse.
There is another truth (the Light of Christ) that affects the soundness of your assertion, but that's a different discussion.