"He just played a perfect first set," Djokovic said. "I could do nothing. I was still finding my footing on the court.
"I must say it wasn't a great start. Jenson was pumped. He had a clear game plan. He was executing his shots tremendously. I was on my back foot. He was reading the play well for a set and a half."
Djokovic broke in the second game of the second set, but was broken in an electrifying 20-minute fifth game, netting a backhand on Brooksby's sixth break chance.
"Definitely I had confidence I could beat him," Brooksby said.
But Djokovic broke back in the sixth game, ripping a cross-court forehand winner for a 4-2 lead, and held from there to take the 68-minute set.
"After that, I started hitting more cleanly and through the court," said Djokovic. "Whenever I needed a serve I found my spot well. It was physical, a lot of exhausting rallies."
In all, Djokovic won 15 of the last 20 games, rolling to victory in the stamina test after two hours and 59 minutes.
Berrettini advanced by defeating 144th-ranked German qualifier Oscar Otte 6-4, 3-6, 6-3, 6-2.
Olympic champs advance
Tokyo Olympic champions Alexander Zverev and Belinda Bencic advanced with straight-set triumphs, as did British 18-year-old qualifier Emma Raducanu.
German fourth seed Zverev stretched his win streak to 15 matches by beating Italian 13th seed Jannik Sinner 6-4, 6-4, 7-6 (9/7).
Swiss 11th seed Bencic eliminated Polish seventh seed Iga Swiatek, last year's French Open champion, 7-6 (14/12), 6-3.
And 150th-ranked Raducanu ripped 43rd-ranked American Shelby Rogers 6-2, 6-1 in 66 minutes.
Raducanu is only the third qualifier to reach the US Open women's quarter-finals after Estonia's Kaia Kanepi in 2017 and American Barbara Gerken in 1981.
Zverev, last year's US Open runner-up, reached his seventh career Slam quarter-final and fourth in the past five Slams.
He saved two set points on his serve in the 12th game of the third set and three more in the tie-breaker before Sinner netted a forehand to end matters after two hours and 25 minutes.
"That's the last few months for me," Zverev said. "Players with confidence know what to do in these situations. I'm happy to get through in straight sets."
He will next face 46th-ranked Lloyd Harris of South Africa, who eliminated US 22nd seed Reilly Opelka 6-7 (6/8), 6-4, 6-1, 6-3.
Swiatek beat Bencic in their only prior meeting in February's Adelaide final, but the Swiss hasn't dropped a set.
"I've improved my game a lot," Bencic said. "I'm glad to turn it around and get through."
Bencic next plays Raducanu, a US Open debutant who hasn't dropped a set in four matches.
"Belinda is a great player who is in great form," Raducanu said. "I'm going to just have to bring it."
Czech fourth seed Karolina Pliskova, this year's Wimbledon runner-up, beat Russian 14th seed Anastasia Pavlyuchenkova 7-5, 6-4.
She next faces Greek 17th seed Maria Sakkari, who ousted 2019 US Open winner Bianca Andreescu 6-7 (2/7), 7-6 (8/6), 6-3
The Canadian sixth seed was limping with a left thigh injury in the final games before suffering her first US Open loss after three hours and 30 minutes.