The offence of "harassment" is contrary to the Protection from Harassment Act 1997 which makes it an offence to undertake a course of conduct which, on two or more occasions, causes alarm or distress to another.
There is a defence which amounts to an honest held belief that the course of conduct was reasonable and not causing such harassment, but it is fraught with difficulty.
By your own admission this would be unlikely to apply. It would be best to desist from further contact.
If you think it, don't say it.....If you say it, don't write it
If you write it, don't sign it......If you sign it......don't be surprised