Note to reader: This blog post was originally written in 2018. Some settings have moved and changed names. I have done what I can to update this post with the interfaces that were present in November 2022, but some things might not be entirely accurate anymore.
Facebook seems to be on a security breach, hole plug mode, with new updates and new features that are constantly coming online. (And some settings are disappearing.) I am updating this post as I discover more settings. If you desire, you can just skip to the updates and the new settings that I've found.
Unless you have been hiding in a hole somewhere, and NOT on Facebook, then you will know about the blowup that is going on about how the Facebook app seems to be accessing information from outside its app — in particular contact details. It's not surprising that people are all up in arms over this. This sort of information should be private, and Facebook should only be accessing information that we've given them permission to access.
Cue the evil doer music, because... Guess what... You did give permission for Facebook to access that information when you installed the Facebook or Facebook Messenger app on your phone. And permission wasn't given by installing the app. Permission was given by a hidden setting that syncs your contact details on your phone with Facebook. Here's the kicker... The setting is set to ON by default — sort-of.
There is actually a dialog that asks if you want to sync your contacts, but the only options it gives is Learn More and OK. How is the average user supposed to know that buried at the bottom of the Learn More option is a tiny Not Now link? So most people hit OK, and presto... You just gave Facebook permission to access your contacts list.
But not all hope is lost. Here's how you can change it, and remove your contracts list from Facebook, without uninstalling the apps — not that uninstalling the apps would remove your contacts list from Facebook anyway.