The question that arises in my head is this: is the division between those two groups one of choice, or one of immutable law, or one of inherent necessity.
I think a bit of all three.
I think that those who find themselves in Outer Darkness, the ultimate and most radical example of Spiritual Death, have no recourse but to remain in that awful place where there is "weeping and wailing and gnashing of teeth" (Mosiah 16:12). Of COURSE they cannot leave in any sense:
1) MENTAL - they have trained themselves to be this way by the choices made in their lives.
2) EMOTIONAL - per Joseph Smith, they could not emotionally abide the fiery glory of the kingdom of God.
3) PHYSICAL - they simply do not have the power to walk "up the mountain" as it were. They are physically restrained to the actual, geographical LOCATION to which they are damned. They can no more leave their place of final judgment than can a lobster in a trap.
But what, then, of the righteous? Are they RESTRAINED from going to Outer Darkness? After all, those in the highest level of the Celestial Kingdom are joint heirs with Christ (Romans 8:17). This means they either ARE as God is (as we know that Abraham and several other prophets have attained to this status), or are training to become such. God, we know, is all-powerful: if it is possible, God can do it.* So would it not stand to reason that those in the Celestial Kingdom could, if they wished, travel to Outer Darkness? After all, we know that those in the terrestrial kingdom receive the personal ministrations of Jesus Christ, and those in the telestial kingdom are ministered unto by the Holy Ghost, so we clearly see that there MUST be SOME kind of "vertical" mobility - a movement between the different classes of glory and ignominy.
However, in that answer, we have the answer to our greater question. We have already seen that the Sons of Perdition are confined without recourse - they have no power to leave their assigned state or location.
And the righteous of the Celestial Kingdom? They most certainly have the POWER: God organized the kingdoms, and having done so, He certainly must have the power to visit. But having the power to do so does not mean it will ever happen.
The damned cannot visit the upper kingdoms of Grace, for they have decided not to, and sealed their fates by the choices made in their lives.
And the Glorious, the Saved, those who have tasted of Christ's Atonement and received His Grace... they have the power to travel where they will. So to say that they do not go to the planes of perdition is not to say that they canNOT do it.
It is simply saying that, like all righteous, they are anxiously engaged in good works. There are no good works in Outer Darkness, and so the Saved do not travel to Hell, not because the canNOT, but simply because there are always better things for them to do.
* There is a statement that is commonly batted about in Christendom that with God, nothing is impossible. This brings up a whole host of OTHER discussions, and will be the subject of another post.