The toll-houses don't bother me, the only thing about them that bothers me is the concept that Angels and Demons will be the ones to judge me. The fact that God seems so distant in this concept is frightening to me, and it seems like the concept of Angels/Demons judging me undermines the mercy of God. If demons and angels want to tempt me, take me to all these places and accuse me and all this crap, I don't mind, but is it so much to ask for that God judges me and not them?
That is the thing, the demons and angels are more like lawyers, not judges.
This is a very important point. I find that critics of the toll-house teaching approach this subject with a literalist Protestant fundamentalist mentality which typically doesn't characterize their approach to other subjects within Orthodox theology. For instance, Protestants are so afraid to detract from the glory due to God that they completely ignore, and even protest against, the role that God has given to the saints and angels in the economy of our salvation. Saints and angels are not to be worshiped as God, but God Himself has given them a role to play. Our guardian angel has been given to us by God Himself to help and protect us, and the saints and angels may intercede for us before God. The fact that they have such roles, and the fact that we acknowledge their roles, does not mean that we therefore put all of our hope in the saints and angels and forget about God. It also does not mean that the saints and angels save us rather than God.
Similarly, with regard to the so-called toll-house teaching, the Lord himself speaks of angels carrying the soul away at the time of death. In the story of the rich man and Lazarus, the Lord says that "So it was that the beggar died, and was carried by the angels to Abraham's bosom (Luke 16:22)." In the story of the rich fool who stored up treasure and then died, the Lord said, "But God said to him, 'Fool! This night your soul will be required of you; then whose will those things be which you have provided (Luke 12:20)?'" In this text, "required" could also be translated "demanded". The Fathers have pointed out that this "demanded" refers to the fact that while angels carry the souls of those pleasing to God to Paradise, demons carry away the souls of those who are not fit for the kingdom of heaven. This is just to point out that the angels and demons have a role to play in carrying out the particular judgment after the separation of the soul and body. To recognize that they have this role affirms rather than denies the Scriptures, and affirms the way in which God has chosen for our judgment occur rather than denying God's role in judging us. As with much of Orthodox teaching, it is not a matter of "either/or", but "both". The Particular Judgment is not the Final Judgment, neither is God absent from the Particular Judgment just because the angels and demons have a role to play.
It is true that the Scriptures do not spell out with great clarity what the soul encounters after its separation from the body. There is actually a great deal that the Scriptures do not spell out in great detail, but which the Fathers explain more completely later. To expect the Scriptures to spell out everything in great detail is to have a Protestant rather than an Orthodox understanding of the Scriptures, their purpose, their function, and their role in the life of the Church. Any attempt to invalidate Tradition by using one's own view of what the Scriptures say, or to consign to "theolgoumena" a teaching that is found throughout the patristic literature, hagiography, hymnography, etc. just because someone doesn't like it or understand how this teaching relates to Scripture - this approach is not an Orthodox but a Protestant one.
The Scriptures themselves testify that "the natural man does not receive the things of the Spirit of God, for they are foolishness to him; nor can he know them, because they are spiritually discerned (1 Cor 2:14)." To rightly understand the Scriptures one must first become a dwelling place of the same Spirit that inspired the Scriptures. In other words, the proper interpretation is known only to the deified. It is from the deified saints, not simply by their written words, but from their recorded experiences and from what God revealed to them, that the teaching regarding the toll-houses has come down to us. The Church has used this imagery in its hymnography, and the Church throughout the world has testified to this teaching by singing and chanting this hymnography with one accord.
The problem with this teaching in Orthodoxy in America has mostly to do with certain argumentative people who wanted to start a fight about this at a time when much of the hymnography, hagiography, and patristic writings were not available in English. This was also a time when certain persons in a certain jurisdiction were trying to make a name for themselves, trying to set themselves up as teachers, and trying to lead others into their own factions. They attempted to do so by denouncing this teaching as "gnostic heresy", so as to exalt themselves as "pillars of Orthodoxy". These early critics have all came to a shameful end, but sadly it is true that flies are still attracted to manure. Not wanting to come also to a shameful end, it is best not to imitate their shameful beginning, but to trust in the teachings of the Fathers that have come down to us on this subject.