The report by Policy Exchange alleged that books condoning violent jihad and encouraging hatred of Christians, Jews and gays were being sold in a quarter of the 100 mosques visited.
But BBC2's Newsnight said examination of receipts provided by the researchers to verify their purchases showed some had been written by the same person - even though they purported to come from different mosques.
Several receipts also misspelled the names or addresses of the mosques where the books were supposedly sold.
The report, the Hijacking of British Islam, was based on the work of four teams of two researchers each who visited 100 mosques. They claimed to have found the controversial material in bookshops attached to 25 mosques, including one at Regent's Park, London, and others in Manchester, Birmingham, Edinburgh, Oxford and High Wycombe.
Published on the eve of a state visit by Saudi Arabia's King Abdullah, the report prompted front page news stories. Tory leader David Cameron pledged to raise the revelations with King Abdullah, because much of the literature was said to have been sourced from Saudi Arabia.
According to the report, one book, which said that there can be "no brotherhood" between Muslims and non-Muslims, was bought at the Leyton mosque in east London.
But the address on a receipt provided by the researchers was found to be that of an unrelated bookshop next door.
A spokesman for the mosque, Dr Usama Hasan, said: "It has nothing to do with us. It is totally inaccurate and misleading information. It is completely false. In fact, we are considering taking legal action over this because it has the potential to damage the good name of our mosque."
Inayat Bunglawala, assistant secretary general of the Muslim Council of Britain, said: "Policy Exchange produced a report that was given a lot of publicity, and Newsnight deserve credit for exposing the incredibly shoddy and dubious methodology that Policy Exchange have resorted to. It would seem that Policy Exchange had already decided what they wanted to say about mosques and just went out to find or should I say invent the evidence to justify their prejudices."
Policy Exchange's research director, Dean Godson, insisted it stood by the report "100%". He said the thinktank had checked its evidence thoroughly and the allegations did not challenge the substance of the study - that such extremist literature was being widely sold.
"We are standing by our report and the Muslim researchers that helped compile it," he added.
The researchers were unavailable for comment because they were all on a religious retreat in Mauritania, Policy Exchange told Newsnight.
Source: Guardian (English) h/t Islamophobia Watch (English)
Newsnight available here
See also: UK: Study of extremist literature in British mosques