MEPs from Orban's nationalist Fidesz party have been unaffiliated with any of the major groupings since their split from the traditional right-of-centre European People's Party.
The parliament leaders also urged the European Commission and the European Council to condemn Orban's statement "in the strongest terms", with the commission so far declining to comment.
Only its vice president Frans Timmermans has reacted, calling racism a "poisonous political invention" with no place in Europe in a tweet on Wednesday that did not mention Orban.
The Hungarian leader's remarks sparked widespread backlash, including from the International Auschwitz Committee, while the United States called them "inexcusable" and reminiscent of the Nazi era.
Orban's government is already under scrutiny for a recent law seen as harming LGBTQ rights in Hungary and for flouting democratic standards over corruption, the independence of the judiciary and the media and public procurement.
Brussels has blocked the release of post-Covid economic recovery funds for Hungary over its failure to respect the rule of law.