Efforts to negotiate a settlement appear to have broken down, with each side blaming the other for the failure of talks as the Taliban prepared to announce a government.
Taliban spokesman Zabihullah Mujahid said the group's fighters had entered Panjshir and taken control of some territory.
"We started operations after negotiation with the local armed group failed," he said. "They suffered heavy losses."
However a spokesman for the National Resistance Front of Afghanistan, a grouping of rebels, said it had full control of all passes and entrances and had driven back efforts to take Shotul district at the entrance to the valley.
"The enemy made multiple attempts to enter Shotul from Jabul-Saraj, and failed each time," he said, referring to a town in the neighbouring Parwan province.
The spokesman said NRFA forces had also killed large numbers of Taliban fighters on two fronts since clashes first broke out earlier in the week.
"It has been proven to the other side that they cannot resolve this issue through war," the spokesman said in reference to the Taliban's losses.
Both sides provided varying figures for the other's casualties, without offering evidence. It was not possible to verify the numbers of fighters on either side killed.
The Taliban says the Panjshir valley is surrounded on all four sides and a rebel victory is impossible. The rebels say they will refuse to surrender.