Sayed Fazil Agha, the head of the mosque, said someone they believed was a suicide bomber joined them in the ceremony and detonated explosives.
"Black smoke rose and spread everywhere, dead bodies were everywhere," he told Reuters, adding that his nephews were among the dead. "I myself survived, but lost my beloved ones."
Resident Mohammad Sabir said he had seen wounded people being loaded into ambulances.
"The blast was very loud, I thought my eardrums were cracked," he said.
A health source said hospitals had received 66 dead bodies and 78 wounded people so far.
The United States and the United Nations' mission to Afghanistan condemned the attack, with the latter saying it was part of an uptick in violence in recent weeks targeting minorities and adding that at least two UN staff members and their families were in the mosque at the time of the attack.
"No words are strong enough to condemn this despicable act," said Mette Knudsen, the UN secretary general's deputy special representative for Afghanistan.
Emergency Hospital in downtown Kabul said it was treating 21 patients and two were dead on arrival. A worker at another hospital treating attack patients said it had received 49 patients and around five bodies. Ten of the patients were in critical condition, the source added, and almost 20 had been admitted to the burns unit.
A spokesman for the ruling Taliban, Zabihullah Mujahid, released a statement condemning the blast and saying the perpetrators would be found and punished.
It was not immediately clear who was responsible.
Scores of Afghan civilians have been killed in recent weeks in blasts, some of which have been claimed by Islamic State.
Emergency Hospital said it had treated more than 100 patients wounded in attacks in Kabul in April alone. The latest attack came on the last Friday in the month of Ramadan in which most Muslims fast, and before the religious holiday of Eid next week.
The Taliban say they have secured the country since taking power in August and largely eliminated Islamic State's local offshoot, but international officials and analysts say the risk of a resurgence in militancy remains.
Many of the attacks have targeted the Shi'ite minority, however Sunni mosques have also been attacked.
Bombs exploded aboard two passenger vans carrying Shi'ite Muslims in the northern city of Mazar-e-Sharif on Thursday, killing at least nine people. Last Friday, a blast tore through a Sunni mosque during Friday prayers in the city of Kunduz, killing 33.