Considering you are on a small island surrounded by the Mediterranean, swimming in Pantelleria requires more planning than you would imagine. On many parts of the coast it is impossible to access the sea.
My absolute favorite swimming spot was Balata dei Turchi, the southest tip of the island. It can be reached by car. Just walk down from the parking lot until you reach a large rock that extends to the water.
We went in June, when the island was still waking up after winter months. It was peaceful and quiet everywhere, and even though we were not swimming alone, it was easy to find a secluded spot. We laid our towels on the rock, basked in the sun, and took frequent dips in the water.
As nice it is to read and relax on the rocks, there is whole other world underwater, abundant with flora and fauna. Balata dei Turchi is a great place for diving but you can discover the lush landscapes by simply snorkeling.