The report, published the New York Times on Thursday, also said however that flotilla organizers had been reckless and Israel's naval blockade of Gaza is legal.
Eight Turkish nationals and an American of Turkish descent died in the May 31, 2010 raid when Israeli special forces in speed boats and helicopters raided the six vessels in international waters.
"Israel's decision to board the vessels with such substantial force at a great distance from the blockade zone and with no final warning immediately prior to the boarding was excessive and unreasonable," said the inquiry, led by former New Zealand prime minister Geoffrey Palmer, AFP reports.
The 105-page report is expected to be released on Friday and says the Israeli military’s treatment of passengers afterwards was abusive, New York Times reports.
The inquiry said forensic evidence showed that "most of the deceased were shot multiple times, including in the back, or at close range" and this has "not been adequately accounted for" by Israel, AFP reports.
It added, however, that the flotilla "acted recklessly in attempting to breach the naval blockade" set up around the Hamas-controlled Palestinian territory by Israel.
The inquiry called for Israel to make "an appropriate statement of regret" for the raid and pay compensation to the families of the dead as well as to injured victims.
Turkey and Israel should resume full diplomatic relations "repairing their relationship in the interests of stability in the Middle East," it added.
Israel has rejected Turkey's demands for an apology. Israeli government officials have said compensation is possible, however.
Neither the Turkish nor Israeli governments reacted immediately to the publication of the report by the New York Times.
The official release of the report has been delayed because the former allies have been unable to agree on a final version. A U.N. spokesman said, however, that the report was expected to be handed to UN Secretary General Ban Ki-moon.
The Mavi Marmara ferry led the flotilla of six ships in which about 700 people from 40 countries took part. All the dead were on the ferry.
According to the report, a Turkish commission said "there was indiscriminate shooting, including from the helicopters."
The UN inquiry concluded, however, that Israeli forces "faced significant, organized and violent resistance from a group of passengers when they boarded the Mavi Marmara requiring them to use force for their own protection."
But it added that the loss of life inflicted by Israeli forces on the Mavi Marmara was "unacceptable." "No satisfactory explanation has been provided to the panel by Israel for any of the nine deaths", AFP reports.
Relations between the two countries deteriorated after the raid and Turkey wants an apology and insists the blockade is illegal.
The report did back Israel over its naval blockade. "We have made it clear that we consider that Israel was entitled to impose the naval blockade. It follows that Israel was also entitled to enforce it."
It added: "The manner of its enforcement, however, raises serious issues of concern".
The Hamas movement called the inquiry report "unjust and unbalanced".
The New York Times says Turkey has not yet commented on the report:
Turkey is particularly upset by the conclusion that Israel’s naval blockade is in keeping with international law and that its forces have the right to stop Gaza-bound ships in international waters, which is what happened in the 2010 episode.
That conclusion oversteps the mandate of the four-member panel appointed by the United Nations secretary general and is at odds with other United Nations decisions, Turkey argued.
The report noted that the panel did not have the power to compel testimony or demand documents, but instead had to rely on information provided by Israel and Turkey.
Therefore, its conclusions cannot be considered definitive in either fact or law.
The Israeli and Turkish members of the four-man panel that wrote the report said they disagreed with key findings. The Turkish panelist dissociated himself from some conclusions.