Not sure how to to toggle the button on or off but I was able to remove that button in our directory

removing: <?php the_contact_listing_owner_button(); ?>
on line 76 in single-listing.php...