"It was never apart and will never be apart of any HOA."
There's a big difference between the meaning of 'apart' and 'a part', which is what I think you meant.
Hope this isn't protracted for you. As others have said, if the HOA has no rights to the land/property, then they are overstepping. It is likely someone there realized a property within view of theirs could not be controlled by their by-laws and thus could ruin the look of their sterile, homogeneous subdivision.
If you have evidence of one of their members specifically going out of their way to enact extracurricular activities on your property, have at it.
I've not dealt with this level of HOA fuckery but, having volunteered for the rules committee at my last subdivision, there are some real pieces of work out there. Some of the rules people wanted were:
- No pets. Mind you, the subdivision was 90+% full at the time and people had pets. They thought it would prevent someone from not picking up after their oet but failed to account for the other 2 subdivisions we were surrounded by.
- Kids can't play outside. The Earth would have imploded if a kid left a toy outside.