Ezra 4

Sometimes unbelievers want to be a part of God's work, for their own reasons. When they're (rightly) rejected, they often take it personally, causing trouble and hindering the work instead of helping it.

That's still no excuse for not saying no.