40% of peered reviewed scientific articles can't be reproduced

[sayak83]

Should we take the results on faith?

Most scientists 'can't replicate studies'

Mostly social science and psychology...which aren't umm ...science...yet.
‘Replication crisis’ spurs reforms in how science studies are done

"That’s the conclusion of a research team, led by Caltech economist Colin Camerer, that examined 21 social science papers published in two major scientific journals, Natureand Science, from 2010 to 2015. Five replication teams directed by coauthors of the new study successfully reproduced effects reported for 13 of those investigations, the researchers report online August 27 in Nature Human Behavior. Results reported in eight papers could not be replicated.

The new study is an improvement over a previous attempt to replicate psychology findings (SN: 4/2/16, p. 8). But the latest results underscore the need to view any single study with caution, a lesson that many researchers and journal gatekeepers have taken to heart over the past few years, Camerer’s team says. An opportunity now exists to create a scientific culture of replication that provides a check on what ends up getting published and publicized, the researchers contend.

On the plus side, the new report appears as such practices are changing. “The social and behavioral sciences are in the midst of a reformation in scientific practices,” Nosek says."

Things that could not be replicated are usually absurd things such as this:-
"For instance, one new replication study that Camerer and colleagues examined did not support a 2012 Science report that viewing pictures of Auguste Rodin’s famous statue The Thinker reduces volunteers’ self-reported religious belief. This finding was part of a project examining how mental reflection affects religious belief."


A "science" that does not have a set of fully worked out quantitative theories that provides precise explanations and predictions for testing and observation will necessarily flounder and get lots of false signals.

 

[Stevicus]

It could be that the scientific peers are up to no good:

 

[sayak83]

And there is a question of the embellishment of data for repeatability being so elusive 2 times out of 3

But there is science and there is science. Fraud is not science and assumptions of materialism not inherent to the scientific method

you are correct in the 2 statements above.

However biology, particularly evolutionary biology is part of the hard sciences and does not suffer from these sorts of problems. They have a highly sophisticated set of mathematical theories that are widely applicable not only in the natural world of life systems, but also in the field of AI and computer science as well and fully integrated with the physical theories of themodynamics, information theory, complexity sciences etc.
Evolutionary approaches to big-data problems
Evolutionary algorithm outperforms deep-learning machines at video games

Thus within the context of evolutionary science, your objections do not have relevance.

 

[Altfish]

And there is a question of the embellishment of data for repeatability being so elusive 2 times out of 3

But there is science and there is science. Fraud is not science and assumptions of materialism not inherent to the scientific method

There is certainly problems in commercial terms with medicines and environmental issues. But these are rarely peer reviewed papers.
Read 'Bad Science' or 'Bad Pharma' by Ben Goldacre.
But again it is a scientist exposing the bad science

 

[Sky Rivers]

Should we take the results on faith?

Most scientists 'can't replicate studies'

Thank you for giving me another good reason to question these studies.

 

[Father Heathen]

Scientists sometimes error, therefore we should abandon science (and thus medicine and technology) and fall back upon superstition and sorcery.

 

[Jumi]

Nice to see you are learning about how science works. A research team makes a study that has interesting results, so other teams (around the world with different ideologies and beliefs, it doesn't matter because it's science) try to reproduce it. If they can't reproduce it, it gets rejected. All interesting research gets attempted by others. So it's very much a working system.

 

[Jumi]

Scientists don't claim something is truth just because one article is peer reviewed. That's where the fun part begins, because science is not based on belief, others can try to reproduce it and you betcha there will be scientists trying to disprove your research if it's of any consequence.

 

[IndigoChild5559]

Should we take the results on faith?

Most scientists 'can't replicate studies'

VIrtually all of the research documenting evolution is not in this 40% category. The evolutionary research has been around for nearly two centuries and not only are the same things coming up over and over, but new evidence is coming up that bolsters the old evidence even more. For example, we have the new science of Genetics, and all of its studies underscore evolution very consistently. Plate tectonics links fossils of the same species in the same strata of rocks across continents. You get the idea.

It is not necessary to take the results on faith. Take the results on the reliability of the results.

 

[Subduction Zone]

Thank you for giving me another good reason to question these studies.

Have you not seen yet how this was misrepresented? These inabilities occur more in medicine than any other science. Even with this that forty percent represents an increase in knowledge forty percent of the time versus zero percent of the time for creationism. It is not a valid reason for ignoring reality.

 

[ChristineM]

Should we take the results on faith?

Most scientists 'can't replicate studies'

I am sure this wasn't your intention but what you are pointing out here is just how rigourous the scientific method is.

 

[ChristineM]

when if can't be reproduced is 2 to 1 against what can... it suggests problems

And when god magic cannot be reproduced at all,is that a problem?

 

[SalixIncendium]

Should we take the results on faith?

 

[ChristineM]

More than 70% of researchers have tried and failed to reproduce another scientist's experiments, and more than half have failed to reproduce their own experiments.
1,500 scientists lift the lid on reproducibility

perhaps there should be more caution and less rush to publish?

applicable in the sciences and also in the media making rush stories

So we have 40%, then 50% 2:1 and now 70% come on, be fair settle on a figure

 

[ChristineM]

when if can't be reproduced is 2 to 1 against what can... it suggests problems

Consider this non scientific comparison.

Some doubter, in the same vain as you do here, said of Winston Churchill "he is useless, has a thousand ideas a day but only one is any good"

The reply was "i wish i could have one good idea a day"

 

[SalixIncendium]

Thank you for giving me another good reason to question these studies.

Sound like a sound plan to me. Science questions everything.

As should a good critical thinker, in my opinion.

 

[Augustus]

That's a problem??? No, it's a solution and the reason why science is so powerful.
That is the whole point of peer review; new science has to be repeatable, if not it is modified or rejected.
It is why Theories are so powerful, they have been reviewed modified, re reviewed and found to be solid and repeatable.

That's some impressive spin there. It is a huge problem that large quantities of published scientific research is wrong, despite passing the peer-review process.

"What we see in the published literature is a highly curated version of what's actually happened," he says.

"The trouble is that gives you a rose-tinted view of the evidence because the results that get published tend to be the most interesting, the most exciting, novel, eye-catching, unexpected results.

"What I think of as high-risk, high-return results."

"The way the system is set up encourages less than optimal outcomes."

One thing to remember is that in the sciences knowledge is cumulative. So even with only 40% of ideas being right they still add new knowledge when they are right and do not harm the sciences when they are wrong. The wrong ideas are eliminated and right ideas are passed on. Hmm... sounds a bit like a biological process.

This is the problem with treating science as a normative concept rather than as a positive real world phenomenon. People engage in apologetics for what an idealised form of the concept should be, at the expense of critical thought regarding the reality.

If false information does not not harm 'the sciences' (which is highly debatable), it is only as these are abstract concepts. It is undeniable that false information causes real world harms though, as this information is used as a basis for decision making. It is also far easier to introduce a new idea, than it is to remove an idea from society, so bad information endures long after it has been refuted.

It is good that the sciences may self-correct over time, but that doesn't negate the fact that until they self-correct they are often causing actual harms. The harms of Scientific Racialism were not undone by its later refutation, the harms of poor dietary and medical advice can't be undone. Social policies based on erroneous research may impact millions before they are quietly swept under the carpet.

False research findings also impact future generations of scientists and researcher, divert funding and priorities, etc.

In addition, the damage public confidence in the validity of all scientific research. If a media outlet ran 30-50% fake news, it would be seen as very problematic, yet if a scientific journal publishes errors at that rate many people revert to normative platitudes.

The sciences, for all their worth, are still one of the biggest sources for false information (certain disciplines far more than others which are very reliable). Moreover, many people place a high trust value on scientific findings. One thing far more dangerous than being wrong is being wrong yet highly confident that you are right.

The replication crisis in some scientific fields should be seen for what it is, a major problem that needs to be fixed.

Science doesn't fear change

Many scientists do though as their careers and livelihoods may well depend on it.

 

[Subduction Zone]

That's some impressive spin there. It is a huge problem that large quantities of published scientific research is wrong, despite passing the peer-review process.

"What we see in the published literature is a highly curated version of what's actually happened," he says.

"The trouble is that gives you a rose-tinted view of the evidence because the results that get published tend to be the most interesting, the most exciting, novel, eye-catching, unexpected results.

"What I think of as high-risk, high-return results."

"The way the system is set up encourages less than optimal outcomes."



This is the problem with treating science as a normative concept rather than as a positive real world phenomenon. People engage in apologetics for what an idealised form of the concept should be, at the expense of critical thought regarding the reality.

If false information does not not harm 'the sciences' (which is highly debatable), it is only as these are abstract concepts. It is undeniable that false information causes real world harms though, as this information is used as a basis for decision making. It is also far easier to introduce a new idea, than it is to remove an idea from society, so bad information endures long after it has been refuted.

It is good that the sciences may self-correct over time, but that doesn't negate the fact that until they self-correct they are often causing actual harms. The harms of Scientific Racialism were not undone by its later refutation, the harms of poor dietary and medical advice can't be undone. Social policies based on erroneous research may impact millions before they are quietly swept under the carpet.

False research findings also impact future generations of scientists and researcher, divert funding and priorities, etc.

In addition, the damage public confidence in the validity of all scientific research. If a media outlet ran 30-50% fake news, it would be seen as very problematic, yet if a scientific journal publishes errors at that rate many people revert to normative platitudes.

The sciences, for all their worth, are still one of the biggest sources for false information (certain disciplines far more than others which are very reliable). Moreover, many people place a high trust value on scientific findings. One thing far more dangerous than being wrong is being wrong yet highly confident that you are right.

The replication crisis in some scientific fields should be seen for what it is, a major problem that needs to be fixed.



Many scientists do though as their careers and livelihoods may well depend on it.

No one has stated that the peer review process is perfect. It is only the first step in new ideas being accepted. And there may be some abuse of it. Most of the problems come from the fields of medicine,which is highly driven by economic pressures, i.e. money, and you may have a point there. In the harder sciences it still exists, but not to such a degree. One is supposed to take initial peer reviewed findings with a grain of salt. New concepts are generally not well accepted without confirmation. A place is needed where new ideas can be presented and further tested upon. That is the purpose of peer review. If too many people treat peer review as the end all that is their problem.

The OP that presented this is very anti-evolution. He appears to be looking for any excuse to denigrate peer review. But then the ideas he believes in cannot pass peer review since they have been refuted a thousand times. It is likely that he sees an attack on peer review as a way to elevate creationism. But even if all peer review is right only forty percent of the time it still beats a topic that never passes peer review.

 

[Twilight Hue]

Should we take the results on faith?

Most scientists 'can't replicate studies'

That's what peer review is for. To find stuff like that out.

 

[Altfish]

That's some impressive spin there. It is a huge problem that large quantities of published scientific research is wrong, despite passing the peer-review process.

No, you misunderstand me. It may be published but it is the peer review that will then find it out, so they do NOT pass peer review.
You can smell a dodgy paper when it is funded by an oil company, a tobacco company or the like of Proctor & Gamble.