How much faith do you have in science?

[youknowme]

Some scientific methods have demonstrated themselves to be by far the best methods we've ever found for determining how some parts of the universe works.

There is only ONE scientific method. Some scientists may employ it better than others, but there remains just the one scientific method.

When certain scientific disciplines (neuroscience, psychology, etc.) have 30-60% error rates in publication that is often passing peer-review, blaming the problems on individual scientists and the media seems a bit misguided.

Who do you think does the peer review, if not individual scientists? It is individual scientists who are passing these studies that you claim have large error rates, not the scientific method. And how would you know that there is a 30-60% error rate, unless other scientist who properly used the scientific method hadn't evaluated the evidence in order to reach that conclusion?

Only one true scientific method? Is that something like the one true religion?

 

[QuestioningMind]

you greatly oversimplify what was going on in climate research at the time, and since.

Are you suggesting that in the 70's there was scientific consensus that a new ice age was on the way?

 

[beenherebeforeagain]

Some scientific methods have demonstrated themselves to be by far the best methods we've ever found for determining how some parts of the universe works.

There is only ONE scientific method. Some scientists may employ it better than others, but there remains just the one scientific method.

When certain scientific disciplines (neuroscience, psychology, etc.) have 30-60% error rates in publication that is often passing peer-review, blaming the problems on individual scientists and the media seems a bit misguided.

Who do you think does the peer review, if not individual scientists? It is individual scientists who are passing these studies that you claim have large error rates, not the scientific method. And how would you know that there is a 30-60% error rate, unless other scientist who properly used the scientific method hadn't evaluated the evidence in order to reach that conclusion?

It's an estimate of 30-60 percent, based on a targeted study of a sample of published articles...why is that margin so uncertain? Oh, yeah...it's a study of the social sciences, which are inherently inexact...

Do social scientists need to do a better job designing, executing and reporting their results? Yes.

Is the peer review process flawed? Yes, to some extent, but some journals are much better at it than others, and some fields are better than others...

Is the funding process and the publish-or-perish system flawed? Yes, indeed.

The use of the p value is only one small aspect of the problem. A finding of a p value significance at the 0.0001 level is wonderful, but if it only explains 5 percent (what's revealed by the r-square or similar statistic) of the variance, then it's pretty useless.

 

[QuestioningMind]

Only one true scientific method? Is that something like the one true religion?

Not in any way shape or form. It's a method that has validated itself time and again... the fact that we can harness electrons so that we are communicating via the Internet is but one example of the method being validated.

 

[Dan From Smithville]

Only one true scientific method? Is that something like the one true religion?

My understanding of the scientific method is that it is a framework around which scientists in a diverse number of disciplines can conduct scientific work that is customized to the discipline and the type of experiment. For instance, you would use the scientific method to study particle physics, insecticide resistance, or human behavior, but you would not use the same exact methodology for each.

 

[youknowme]

My understanding of the scientific method is that it is a framework around which scientists in a diverse number of disciplines can conduct scientific work that is customized to the discipline and the type of experiment. For instance, you would use the scientific method to study particle physics, insecticide resistance, or human behavior, but you would not use the same exact methodology for each.

Because they each approach a different set of problems that require different methods.

 

[youknowme]

Not in any way shape or form. It's a method that has validated itself time and again... the fact that we can harness electrons so that we are communicating via the Internet is but one example of the method being validated.

There is no such thing as "the" scientific method, if anything it should be scientific methods.

 

[Dan From Smithville]

Because they each approach a different set of problems that require different methods.

Yeah. It is all under the scientific method. Just adjusted to the different problems of research in different subjects.

 

[Dan From Smithville]

There is no such thing as "the" scientific method, if anything it should be scientific methods.

I think we may be saying the same thing in a different way, but there is no one size fits all scientific method.

 

[youknowme]

I think we may be saying the same thing in a different way, but there is no one size fits all scientific method.

Personally, I think the whole thing is a myth. Methodology should be weighted on its own merits and not by some arbitrary labeling. People put faith in science by virtue of being called science, they don't understand the how's or the why's they just want scientist to put it in a nice little package for them and then tell them what to think about it.

 

[Dan From Smithville]

Personally, I think the whole thing is a myth. Methodology should be weighted on its own merits and not by some arbitrary labeling. People put faith in science by virtue of being called science, they don't understand the how's or the why's they just want scientist to put it in a nice little package for them and then tell them what to think about it.

I am not sure exactly what you are saying regarding a weighted methodology, but I see a growing trend away from trust in scientists and the science that is reported. Both sides in the issue bear responsibility, but there is definitely a trend away from trust based on belief

 

[youknowme]

I am not sure exactly what you are saying regarding a weighted methodology, but I see a growing trend away from trust in scientists and the science that is reported. Both sides in the issue bear responsibility, but there is definitely a trend away from trust based on belief

I just mean people should consider the value, limit and relevance of the methods being used. The problem with this, however, is that people need to understand the methods being used.

 

[Dan From Smithville]

I just mean people should consider the value, limit and relevance of the methods being used. The problem with this, however, is that people need to understand the methods being used.

I agree with that. I know that I would have difficulty following some methods in other fields outside of biology. Even within biology, there are methods I am unfamiliar with other than in passing.

 

[MonkeyFire]

I think it is a bit reckless to think of science as some type of truth validation, personally, I view science as evidence.

Believing is the evidence, science is the means.

 

[youknowme]

Believing is the evidence, science is the means.

Not for me.

 

[QuestioningMind]

There is no such thing as "the" scientific method, if anything it should be scientific methods.

There are numerous fields of science, but they all employ the same scientific method. In a nutshell the method is coming up with a hypothesis and then coming up with a way to test that hypothesis that others can replicate to get the same results.

The hard sciences, like chemistry and astronomy are far more exact than the soft sciences, like psychology and sociology, because the soft sciences are often measuring far more subjective data, such as 'happiness' or 'quality of life'. But that doesn't change the reality than any legitimate psychological study employs the scientific method of coming up with a hypothesis and then finding a means of testing that hypothesis so that others can replicate it and get the same results.

 

[exchemist]

I think it is a bit reckless to think of science as some type of truth validation, personally, I view science as evidence.

Hear hear.

Science in general avoids claims of "truth", because in science all "truth" is provisional.

 

[exchemist]

It always amazes me that *any* scientist uses a p value as large as .05. Let's face it, a 1/20 change of the result being due to chance is way too high for any real confidence in the result.

Instead, what would happen if we started using, say, a maximum p value of .0005?

Now, instead of a 1 in 20 chance of the result being due to chance, we have a 1 in 2000 for it being due to chance. Still, given the thousands of tests being done, there will be some 'false positives' that get through, but many, many fewer than is now the case.

That would greatly increase the overall confidence in our result and, most likely, improve the overall image of the science among the lay public.

The main downside? Some scientists would have to work a lot harder to get publishable results. I bet that would be a good thing.

In particular, way too many results in the medical journals use a p<.05 standard. This, in my mind, is almost criminal. It is also a standard in many psychological journals.

I found myself instinctively agreeing with you here, but then had second thoughts.

It seems to me that if we were to apply a criterion as strict as you suggest, a lot of real phenomena would not be brought to light. This could quite seriously retard the advance of knowledge.

Is there not a case for two standards: a weak one, as a trawl net, to catch possible phenomena and effects, and then a strong one to turn "possible" into "probable"?

And is this not, in fact, what current research does, broadly speaking?

 

[sayak83]

One of the two types of studies you can do with statistics (the science) is to make cause and effect inferences. There are only two types of studies in statistics, observational and experiments. Then there are two types of models, predictive models (not looking for cause and effect relations on this one) and explanation models. In some cases we are very much modeling to make cause and effect inferences and the science has been designed to do just that (realizing these are still probabilistic inferences).

Now you can test predictive models in a more reliable fashion than explanation models; you just test how close it is to predicting actual values, and in some cases this will work with the other model as well, but not always. We need to consider statistics as just evidence, and not as facts, because that is what it actually is, and then we need to take that evidence and consider it alongside other evidence. We cannot rely on a one statistical study and we cannot rely on one approach, but to hit deadlines that is what is happening. So I agree with you in a sense, that we need to take our model and find some why to actually test it; however, your claim that we can't make cause and effects inference with statistics is just not correct, under the right conditions you can.

Scientific papers are not meant to be conclusive but only indicative of something that might prove promising and as a form of communication between research groups who are doing research in that field. Its only after hundreds and thousands of papers pointing in the same directions and building up on that idea, can one say that "yes" that idea has actual substance. One can get a far more accurate sense of where a certain topic is heading by looking at what are called review or progress papers that are written by long time researchers in the field and these papers encapsulate the trends from hundreds of various research groups over 10-20 year period.

A good example of such excellent review or progress journals are "Nature Reviews Genetics", "Annual Review of Astronomy and Astrophysics", "Chemical Reviews" etc.

It takes a research student doing PhD typically 2 years or more to get a hang of how to properly analyze scientific papers in the topic of her research. Its a difficult and vital skill, part of any successful PhD in science. Hard to expect media groups to understand what is going on regarding research "news". Science reporting is inaccurate and wildly hyperbolic...but better than nothing.

Regarding medical biology in particular, the current questioning about the utility of some reseacrh papers is probably due to lack of results in terms of drug discovery. Some are arguing that the standards need to be raised as to what can be communicated to other research groups as "promising" as there are too many blind alleys around. Unfortunately, a typical PhD lasts 4-5 years, and biological research is already becoming too complex for a new trainee research scholar to pick it up and deliver substantial results in 4-5 years of time. Maybe we need CERN or NASA like super-facilities for biomedical research, where hunderds and thousands do collaborative work outside of a university or research hospital setting?

Sociology and psychology is not yet a science. So I don't care.

 

[Augustus]

There is only ONE scientific method. Some scientists may employ it better than others, but there remains just the one scientific method.

Nobel winning physicist Steven Weinberg:

Not only does the fact that the standards of scientific success shift with time make the philosophy of science difficult; it also raises problems for the public understanding of science. We do not have a fixed scientific method to rally round and defend. I remember a conversation I had years ago with a high school teacher, who explained proudly that in her school teachers were trying to get away from teaching just scientific facts, and wanted instead to give their students an idea of what the scientific method was. I replied that I had no idea what the scientific method was, and I thought she ought to teach her students scientific facts. She thought I was just being surly. But it’s true; most scientists have very little idea of what the scientific method is, just as most bicyclists have very little idea of how bicycles stay erect. In both cases, if they think about it too much, they’re likely to fall off.

The problem of thinking of 'science' as a singular is that it really is diverse range of knowledge, techniques and practices with differing levels of reliability and value.

Due to cultural conflicts about the role of 'science' in society, people, rightly, want to defend the importance of scientific enquiry. Yet treating it as a singular often results in people feeling compelled to defend anything deemed science, even when this forces them to defend fields where a majority of published findings are false.

This is often done by referring to chemistry to defend social sciences, or Newtonian physics to defend complex predictive climate models simply because all are 'science' and 'science' is reliable.

Who do you think does the peer review, if not individual scientists? It is individual scientists who are passing these studies that you claim have large error rates, not the scientific method. And how would you know that there is a 30-60% error rate, unless other scientist who properly used the scientific method hadn't evaluated the evidence in order to reach that conclusion?

This is just another version of 'if only people were truly Christian there would be no more problems', or 'if only Lenin had practiced proper communism it wouldn't have been oppressive'.

Science isn't a normative textbook concept to be admired and praised, it is the total of knowledge and actions deemed scientific by a society.

You can't get 60% error rates based on malpractice by individual scientists, you get them because there is something wrong with the system these scientists are operating in.

This is the inevitable result of the way certain sciences operate at this point in time.