He breathed his last on Friday afternoon at Sylhet MAG Osmani Medical College Hospital.
Earlier he suddenly fell sick in the morning and was rushed to the hospital from the jail, reported our correspondent.
He was sentenced to lifetime imprisonment after allegations of war crimes were proved against him.
Confirming the death of Mujibur Rahman, senior jail superintendent of Sylhet Central Jail Muhammad Manjur Hossain said, "He died of old age health complicacies."
Mujibur was arrested and sent to Dhaka Central Jail on 12 February 2012.
The International Criminal Tribunal sentenced him to a lifetime in jail on 1 June 2016.
He was serving his terms at Habiganj District Jail before being transferred to Sylhet Central Jail in May this year.
According to jail sources, he was sent to Sylhet MAG Osmani Hospital for better treatment after he fell ill with various ailments including chest pain. After being taken there, the doctor on duty declared Mujibur dead.