More from GlobalPost: Finding the exit in Afghanistan
Rahmani was a minister during the Taliban rule, and is described as a "key negotiator" in the council, established to spearhead reconciliation efforts with the Taliban.
Family members and security officials say he was shot dead by a gunman in a passing car, while he was on his way to work in the capital’s west, the BBC reported.
Witnesses told the BBC a single bullet was fired from a gun using a silencer.
Rahmani’s death comes after High Peace Council chief Burhannudin Rabbani was assassinated in a suicide attack in September 2011.
In April his son Salahuddin Rabbani, who is Afghanistan’s ambassador to Turkey, was appointed to replace him.
More from GlobalPost: Yandex, Russia's Google, takes off
The High Peace Council was set up by Karzai two years ago to negotiate and end to the decade long war in Afghanistan