I am not sure if what I have to say will help or not but, still..
Your spiritual father can only help you if your relationship is open and although we are all tempted to maintain an illusion persona about ourselves before others, it is very important that we form an honest and genuine relationship with our father confessors. he probably has heard every sin under the sun, and he is there to help you spiritually with all your struggles of life. I can assure you , he will not be surprised with what you might have to tell him. in fact he will be greatly pleased if you were to take that chance and open up to him and let him see the real you, i.e. the YOU that loves God enough to open herself and expose her weaknesses before Him and his priest. if your confessor knows your heart its easier to give you advise that will be beneficial specifically to you.
you said he has a lot of commitment that makes it difficult to get a hold of him, then in that case talk to him about making an appointment at a time when it is convenient for both of you. if he is unable to find time to accommodate you then find another less busy priest and ask your present father confessor for blessing to go to the new priest.
I am hopeful that your father confessor will find time for you. Most importantly PRAY about the situation you are struggling with, and for your father confessor so the Lord may work according to His will for both of you.
that's all I have dear, May the prayers of the Saints be with you!