The city's fire department said its officers arrived at the scene in the Fairmount area at 6:40 am (1140 GMT) "and found heavy fire coming from the second floor of a three-story rowhouse."
"PFD responded to a fatal fire this morning on the 800 block of N. 23rd St," it said in a tweet.
"It took 50 minutes to place the fire under control."
Philadelphia mayor Jim Kenney said the rowhouse was owned by the Philadelphia Housing Authority.
"This is without a doubt one of the most tragic days in our city's history, the loss of so many people in such a tragic way," he told reporters.
The Philadelphia Inquirer quoted police as saying that the three-storey home had been converted into two apartments.
Neighbors reached by local media said they had been shaken by the fire.
Bill Richards, who lives nearby, told the Inquirer that shortly before 7:00 am he had heard a woman scream "Oh my god, oh my god."
"It's totally devastating and upsetting," he said.