If your system is exempt from needing to register you won't need to contact them, they would write to you if they receive a complaint.
ICO wouldn't get involved in site visits etc, that would be police or council.
I don't believe you have done anything wrong, I just feel from what you have said your method of application has possibly provoked the situation.
Also considering that there are always two sides to these things...