A former Cabinet minister has said it is “an insult” to Jewish people for the Brighton Grand Hotel bomber to be invited to speak at a Holocaust Memorial Day event in Suffolk.

Pat Magee, who planted a bomb in the hotel in 1984, is set to speak at the event on Monday in Ipswich which has been organised by Islamic charity JIMAS.

Speaking yesterday from his home in Bury St Edmunds, Lord Tebbit, whose wife Margaret was left permanently disabled after the blast, said: “All I would say is none of my Jewish friends would wish Holocaust Memorial Day to be muddled up with Sinn Féin, IRA terrorists.

“It is an insult to Jewish people for a low-grade terrorist and convicted murderer to try to associate himself with them.”

Lord Tebbit and his wife were staying at the Grand Hotel during the Conservative Party Conference when the bomb went off. Five people were killed in the explosion – including two high-ranking members of the party – and 31 people were injured.

Yesterday, despite attempts to contact him, Muhammad Manwar Ali of JIMAS was not available to address Lord Tebbit’s concerns.

However, earlier Mr Ali said the event would be a “solemn occasion” to mark the Holocaust.

He said the talk was to get those involved to make a “clear-cut commitment against anti-Semitism and any kind of hatred and genocide”.

He added: “We’ve all been on a journey and we want to give a powerful message from ex-radicals to say they’re rejecting it and say ‘never again’.” Organisers have also faced criticism for inviting Tommy Robinson, the former leader of the right-wing English Defence League, as a guest speaker.

The event was originally planned to take place in conjunction with the Education Quarter Multi Faith Chaplaincy at Suffolk New College (SNC) and University Campus Suffolk (UCS). However the colleges have since disavowed themselves entirely from the talk and JIMAS has decided to proceed independently.

Unite Against Fascism (UAF) wrote to the college urging them to cancel the talk. Weyman Bennett, joint national secretary of UAF, said: “This is not a gathering to pay respect to the millions of Jewish people murdered by the Nazis in the Holocaust.

“We call on the organisers to withdraw their invitation to the former EDL members.”

Olivia Marks-Woldman, chief executive of the Holocaust Memorial Day Trust, said: “Local Holocaust Memorial Day events are organised and led by local activity organisers, for local communities.

“The Holocaust Memorial Day Trust provides resources, support and guidance to help plan successful activity. We do not ‘approve’ events and do not tell local activity organisers what to do. We would not support anything that could detract from the central messages of Holocaust Memorial Day.”

Other speakers will include Holocaust survivor Frank Bright and Umme Thara, a former member of Al-Muhajiroun. It is not yet known where the event will take place.

Mr Ali is a fellow of Suffolk New College and a member of the chaplaincy for SNC and UCS.