The charred interior of the fourth floor, which housed a clinic that provided mental health services and general medical care, was visible through broken and blackened windows.
An Osaka police spokesman told AFP that officers were investigating the cause of the fire.
But he could not confirm Japanese media reports citing police sources saying arson was a possibility, with some media outlets saying a man had allegedly dispersed a liquid to start the blaze.
A young woman who witnessed the fire told public broadcaster NHK she had seen a woman trapped on the fourth floor.
“She leaned out (from a window) and was saying things like ‘Please help’... She seemed very weak. Maybe she inhaled lots of smoke,” the woman said.
The Osaka fire department told AFP that 27 of the 28 people injured in the blaze showed no signs of life. In Japan, only a doctor can officially certify someone dead.
Local media said in the afternoon that five people had been confirmed dead.
“The fire was detected at 10:18 am (01:18 GMT) on the fourth floor” of the eight-storey structure, the fire department official said.
“As of noon, 70 fire engines are at the scene.”
The blaze, in a busy business area near Kitashinchi train station in the city in western Japan, had been put out after half an hour, the official added.
Most of the building’s exterior remained intact after the fire, with around 20 square metres (215 square feet) reportedly burned in the blaze.
“There was lots of dark smoke... there was a very strong smell, too,” a middle-aged woman told NHK at the scene.
Osaka, a major economic hub, is Japan’s second-biggest metropolis after the greater Tokyo region.
Deadly fires are unusual in Japan, which has strict building standards, and violent crime is also rare.
One year ago, a man was charged with murder over a 2019 arson attack on a Kyoto animation studio that killed 36 people, the country’s deadliest violent crime in decades.
The attack sent shockwaves through the anime industry and its fans in Japan and around the world.
A 2008 arson attack on a video shop in Osaka killed 16 people. The attacker is now on death row.