Good for you guys - it might be a public street. But, it's up to you to choose (when you can) who you ride with.
I recall one time telling a guy he couldn't ride with us. We were getting ready for a ride when a random guy came through the gas station. ZERO gear. I told him we only ride with people that are properly geared up for a ride.
He got all pissy, telling me I wasn't his mother yada yada yada. I had to stop him and explain that I didn't care about HIM - it was all for me. I didn't deserve to see his head cracked open. Not a mental image I wanted to see...
He didn't like me