Well, thankfully things have gotten so much better! It took about 7 days for him to settle into sleeping without his swaddle. After I posted my original post, he quickly learned how to roll from stomach to back. Once he got the hang of that , he had no trouble and didn't get frustrated. He now prefers to sleep on his stomach or on his side, which originally freaked me out but now I am ok with it. It still sometimes takes him 20 minutes or so to fall asleep at night. (actually right now he has been in bed for 40 minutes and he's still awake, not upset just playing with his feet). I am not going to lie, it was a tough week. I did actually just have to let him work it out on his own sometimes. I didn't want to but I felt like I didn't know what else to do. Sometimes I would just stay in the room and stand over his crib without making eye contact with him until he got sleepy or rolled over, then I left. The biggest thing I can say is to not end up doing something that will be hard to break later like rocking to sleep or feeding to sleep. I stuck to my guns and made sure not to create another sleep association to help him through that phase.
The best news about him not being swaddled is that he now is not dependent on his pacifier anymore. Plus if it falls out, he has his hands free to find it and he can put it back in his mouth on his own.
Another bonus is that if he wakes up, he spends about 15 minutes just rolling and playing before he starts "caliing" for me. It was a long journey to get from there to here but I am so happy we are out of the woods. Good luck to you when you come to this point. I'm sorry I don't have better advice with specific steps on how we did it. It was just trial and error mostly.