Can leaseholder build structures without freeholders' permission?

I own a freehold to a converted house and our neighbour erected a
fence at the front of the building. They knew that we were totally
against it as it causes us inconvenience. They waited and built the
fence while we were on holiday. Our solicitor went through the lease
and said they needed our permission to erect any structures around the
property. He sent them a letter giving them one month to remove it. If
they choose to ignore it we can have it removed and pass them the
bill. We tried to talk to them and explain why the fence cannot be
there but they literally told me to f*** off. I'm sure they will
ignore all the solicitors' letters and I'm afraid that if I have it
removed they can take me to court or something. I would really
appreciate a second opinion in case our solicitor is wrong. Can we
really have the fence that had been built illegally removed? According
to our lease we have all rights to do it.