Hundreds milled about in the grounds of the colonial-era whitewashed residence, with few security personnel in sight.
Rajapaksa had left the official residence on Friday as a safety precaution ahead of the planned weekend demonstration, two defence ministry sources said. Reuters could not immediately confirm his whereabouts.
Protesters also broke into Wickremesinghe's private residence and set it on fire, the prime minister's office said.
The prime minister had also been moved to a secure location, a government source told Reuters.
At least 39 people, including two police officers, were injured and hospitalised during the protests, hospital sources told Reuters.
Wickremesinghe held talks with several political party leaders to decide what steps to take following the unrest.
"Wickremesinghe has told the party leaders that he is willing to resign as Prime Minister and make way for an all-party government to take over," his office said in a statement.
Parliament speaker Mahinda Yapa Abeywardena said in a letter to Rajapaksa that several decisions had been made at the meeting of leaders from parties -- including the president and the prime minister resigning as soon as possible and parliament being called within seven days to select an acting president.
"Under the acting president the present parliament can appoint a new prime minister and an interim government," said the letter released by the Speaker's office.
"Afterwards under a set time an election can be held for the people to elect a new parliament," it added.
Leaders of several opposition parties have also called for Rajapaksa to resign.
"The president and the prime minister must resign immediately. If that does not happen political instability will worsen," said Sri Lanka Freedom Party leader and former president Maithripala Sirisena, speaking before Wickremesinghe had offered his resignation.
Early in the day protesters forced their way through heavy metal gates into the Finance Ministry and the president's seafront offices and did not vacate the campus until late evening.
A local news channel said four journalists outside the prime minister's private home were assaulted during the protests. The prime minister's office, in a statement, expressed "grave regret" over the assault on journalists by security personnel.
The Indian Ocean island of 22 million people is struggling under a severe foreign exchange shortage that has limited essential imports of fuel, food and medicine, plunging it into the worst economic crisis since independence in 1948.
Soaring inflation, which reached a record 54.6 per cent in June and is expected to hit 70 per cent in the coming months, has heaped hardship on the population.
"This is a dicey situation. If a clear transition is not put in place the president and prime minister's resignation will create a power vacuum that could be dangerous. The Speaker can appoint a new all-party government but whether they will be accepted by the protesters remains to be seen," said political analyst Kusal Perera.
Political instability could undermine Sri Lanka's talks with the International Monetary Fund as it seeks a $3 billion bailout, the restructuring of some foreign debt and fund-raising from multilateral and bilateral sources to ease the dollar drought.
'We will not stop'
The crisis comes after the Covid-19 pandemic hammered the tourism-reliant economy and slashed remittances from overseas workers.
It has been compounded by the build-up of hefty government debt, rising oil prices and a ban on the import of chemical fertilisers last year that devastated agriculture. The fertiliser ban was reversed in November.
However, many blame the country's decline on economic mismanagement by Rajapaksa and there have been months of largely peaceful protests demanding his resignation.
Before breaking into the government buildings on Saturday, the protesters dismantled several police barricades in Colombo's government district.
Police fired shots in the air but were unable to stop the crowd from surrounding the presidential residence, a witness said.
Despite a severe shortage of fuel that has stalled transport services, demonstrators packed into buses, trains and trucks from several parts of the country to reach Colombo for the weekend demonstration.
Discontent has worsened in recent weeks as the cash-strapped country stopped receiving fuel shipments, forcing school closures and rationing of petrol and diesel for essential services.
Sampath Perera, a 37-year-old fisherman, took an overcrowded bus from the seaside town of Negombo 45 km (30 miles) north of Colombo, to join the protest.
"We have told Gota (Rajapaksa) over and over again to go home but he is still clinging on to power. We will not stop until he listens to us," Perera said.