It can sometimes be very intimidating to talk to someone in their native language, when you are no master of the language in question. But you still need to practice the oral in order to improve and breach the gap.
One way to do so is to join groups with like minded enthusiasts, who are also looking for someone to speak with. Even better are groups where a fluent French speaker is willing to share the secret of the language with you all.
One way to find such groups is through Meetup, it is a great website find local groups that spans various interests, including languages. I encourage you to join one of those groups if you want to progress. Not only you’ll be able to listen and speak fluently, you will also make new friends.