Rain delayed the start of the contest by 90 minutes.
Malan got going with opening partner Aiden Markram to put on 43 runs before medium-pace bowler Chamika Karunaratne struck with the first wicket. Markram was caught behind for 21.
Malan and Hendricks, who was one of the three changes in South Africa's team for this match, mixed the right dose of caution and aggression and hit seven boundaries between them.
Hendricks raised his third ODI fifty before being bowled by off-spinner Dhananjaya de Silva.
Malan reached his third ODI hundred to raise his bat as he put on an attacking partnership of 86 off 54 balls with Heinrich Klaasen to take the attack to the opposition bowlers.
Pace bowler Dushmantha Chameera broke the stand to send back Klaasen, for 43 off 27 balls, trudging back to the pavilion and got Malan lbw on the next delivery.
South Africa got 97 runs from their last 10 overs to set Sri Lanka, who lead the three-match series 1-0, 284 for victory.