Critically Examining "Method"

[Iconoclast08]

I am an empiricist at heart as are many on this forum (with a Popperian-Lakatosian outlook). Recently, I read something in a chapter from the book Bergin and Garfield's Handbook of Psychotherapy and Behavior Change (5th Ed. by Slife) that gave me some pause:

"The doctrines of observability and replicability are not themselves observable and replicable; they are philosophical (or moral) assertions about what should be valued, and they cannot be supported by scientific evidence because, again, such values have to be assumed to gather such evidence." (p. 51)

While this doesn't bother me to the point of wanting to just to pack it up, go home, and play golf the rest of my life 'cause "hey, we'll just never know!" (ah, those annoying post-modernist nihilists!), it is an interesting point that deserves to be addressed/discussed seriously rather than simply dismissed or denigrated.

Of course, I still think that objectivism is the way to go and that the scientific method IS the best we have at the moment for systematically trying to control extraneous factors, increase precision of measurement, and check our own inevitable biases (e.g., experimenter effects) that creep into our investigations.

Nevertheless... it is fascinating to me how few researchers (in my field, at least) ever really bother to critically examine or be skeptical about the method itself, which is assumed by some hard-core scientists to be free of a priori values and devoid of any political or philosophical assumptions.

Paul Feyerabend had some interesting thoughts on these matters in Against Method, but he lost me with the whole "anarchic scientism" and provocative, outrageously silly quotes in essays where he posited that society should be defended from science.

Thoughts/reactions/comments?

[Limbo]

"Science is too important to be left to scientists"

[Dancing David]

Well whoever wrote that is silly, of course the methods of observation and replication are subject to scrutiny.

So far the universe appears to be isotropic in space and time, the methods of observation are subject to question and verification, the validity can be tested as well.

The author is quoting a fool or someone who is a fool.

[dudalb]

The thing is the scientific method is like all methods: It works on the GIGO Principle:
"Garbage In, Garbage Out".

[Jeff Corey]

..."The doctrines of observability and replicability are not themselves observable and replicable; they are philosophical (or moral) assertions about what should be valued, and they cannot be supported by scientific evidence because, again, such values have to be assumed to gather such evidence." (p. 51)...
Scientists generally agree that these are two crucial components of the scientific method and the support for the use of these tools is provided by the fruits of the scientific method. It works.
The same type of statements are made about the assumptions of determinism and orderliness. Our only choices are to either continue to look for lawful functional relationships or to abandon the whole enterprize.
I am not familiar with the book or the authors. Are they suggesting that the scientific method is of little worth in evaluating the effects of psychotherapy?

[Dr H]

I am an empiricist at heart as are many on this forum (with a Popperian-Lakatosian outlook). Recently, I read something in a chapter from the book Bergin and Garfield's Handbook of Psychotherapy and Behavior Change (5th Ed. by Slife) that gave me some pause:

"The doctrines of observability and replicability are not themselves observable and replicable; they are philosophical (or moral) assertions about what should be valued, and they cannot be supported by scientific evidence because, again, such values have to be assumed to gather such evidence." (p. 51)

While this doesn't bother me to the point of wanting to just to pack it up, go home, and play golf the rest of my life 'cause "hey, we'll just never know!" (ah, those annoying post-modernist nihilists!), it is an interesting point that deserves to be addressed/discussed seriously rather than simply dismissed or denigrated.

Why does it bother you at all? Observability and replicability are essentially axiomatic components of a particular formal epistemological system. Every system is built upon certain axioms, and by definition axioms are not provable within that system. Godel established that no formal system above a certain elementary level of complexity can be both complete and consistent.

If it becomes an issue it's always possible to design a metasystem in which the axioms of the original system are provable. Of course that metasystem will, in turn, be based upon other axioms which are not provable within that metasystem.

All that really says to me is that you don't get something for nothing, and there are limits to what you can explain through self-referents.

Of course, I still think that objectivism is the way to go

"Objectivism" as in the philosophy propounded by Ayn Rand?

and that the scientific method IS the best we have at the moment for systematically trying to control extraneous factors, increase precision of measurement, and check our own inevitable biases (e.g., experimenter effects) that creep into our investigations.

Agreed.

Nevertheless... it is fascinating to me how few researchers (in my field, at least) ever really bother to critically examine or be skeptical about the method itself, which is assumed by some hard-core scientists to be free of a priori values and devoid of any political or philosophical assumptions.

Paul Feyerabend had some interesting thoughts on these matters in Against Method, but he lost me with the whole "anarchic scientism" and provocative, outrageously silly quotes in essays where he posited that society should be defended from science.

Thoughts/reactions/comments?

I think it is a good thing to be aware of the limitations inherent in one's epistemology, as far as that is possible. To that end I've spent a lot of time looking at the history and philosophy of science. That said, and being of an engineering bent myself, for practical purposes one must at some point draw a line and decide "this is good enough, for now."

Of course that line can be in different places in different situations. But (without trying to be nihilistic) it really may be the case that we can't know everything.

For example, if we can demonstrate with a high level of probability that the universe as we know it stems from the Big Bang, there will still doubtless be sufficient mystery as to how all this "stuff" works together to keep human intellects occupied for countless generations to come.

Is that not sufficient? Why need we concern ourselves additionally with "first cause" questions like "what was there before the Big Bang" -- questions which not only may not have answers, but which it might actually be meaningless to ask?

[paximperium]

"The doctrines of observability and replicability are not themselves observable and replicable; they are philosophical (or moral) assertions about what should be valued, and they cannot be supported by scientific evidence because, again, such values have to be assumed to gather such evidence." (p. 51)

I can see the obvious conundrum here however I don't have an issue with just rejecting the premise.

Here is their argument, the Scientific Method is flawed because its premise is inherently circular. In order to state that the doctrines of observation and replication are true, you need to be able to observe and replicate the observation and replication itself BUT that leads to a circular argument where you are using an unproven doctrine to support an unproven doctrine.

1)Observation is hampered by your tools: All observations are inherently flawed. It is nothing more than a distillation of sensory input into information that can be passed onto others. Taking that flaw into account, as long as it is capable of being confirmed by others(people or tools), the reliability of that observation is "good enough" approximation of reality until better tools come along...unless you are a post-modernist or believe in solipsistic nonsense.

2) Reproducibility is due to causality: If you accept the tenet that "the universe is inherent causal", that one action leads to another action and that those actions follow certain natural laws )ie. uncaused or contracausal events do not apply) then you should have no problem with reproducibility if the universe is truly as stated. Confirmation of causality is evidence of causality itself. To prove that the universe is contracausal or uncaused, you just have to show a single event that completely impossible to explanation.

These "doctrines" are not provable within the system itself. So if you accept that observations are flawed and never perfect and that the universe is causal and can only study things within nature; then I see no problem with the Scientific Method at all.

[Iconoclast08]

Are they suggesting that the scientific method is of little worth in evaluating the effects of psychotherapy?

The point of the chapter was more of the "let's not let the assumptions and values that underlie our methods of inquiry go unexamined" variety. The way I took it wasn't so much an attack or refutation of our methods as it was a point of consideration, and many scientists may not necessarily grapple with these types of issues regularly.

They point to the importance of fusing qualitative research with the quantitative since quantitative has certain limitations that can be rectified by qualitative (and also very much so vice versa).

Plurality of methods. That type of thing.

[Iconoclast08]

Why does it bother you at all? Observability and replicability are essentially axiomatic components of a particular formal epistemological system. Every system is built upon certain axioms, and by definition axioms are not provable within that system. Godel established that no formal system above a certain elementary level of complexity can be both complete and consistent.

If it becomes an issue it's always possible to design a metasystem in which the axioms of the original system are provable. Of course that metasystem will, in turn, be based upon other axioms which are not provable within that metasystem...<snip>

Excellent points, thanks for this. I really like the Godel connection here.

I guess it didn't so much "bother" me as it did just give me pause since I hadn't really looked at it too deeply from this angle previously. If anything, it has just expanded my awareness of "method" and provided me with more food for thought.

[Dancing David]

Well, here is my issue as a former mental health worker and current mental health patient.

Ethics of efficacy.

Qualitative studies are nice, and are often used to demonstrate a case study style of interventions. They have their uses for delineating terms, theories, symptomoloy and interventions, they are very useful in the exploration of social network and support interventions.

However this is this huge problem in mental health of efficacy.

First off there is this patchwork of providers, it varies by state, who is licensed and who is not. How counseling can be done and by whom in what circumstances. As a second layer on this there are all sorts of agencies, private and public and individuals who provide various mental health services. Then there are all sorts of people who engage in counseling as just the routine course of their jobs (business mangers and supervisors). So in this first impactor you just have a huge array of people who provide services, many (even in the agencies) with little to no experience or supervision.

Second you have all the different models of dynamics and intervention: family theory, network theory, social model theory, cognitive theory, 'psychotherapy' and all sorts of interesting fringe elements in there (my favorites being FISH and 'nurtured heart') including all the business motivational models and then the whacked out ones. I haven't even begun to list all the major groups like psychosocial rehab, twelve step and others.

Third you have all the different approaches to intervention, counseling and therapy. And they are legion! I could not even begin to delineate the different styles and modes of intervention. Individual services, groups services, medication, inpatient, community based, intensive residential, intensive outpatient, case management. There are the actual intervention people use themselves: resources management, supportive listening, role playing, problem solving, play therapy, diagramming, journaling, guided imagery, religious based counseling, and sooooo many more.

And so here is the deal, there are a lot of people who provide these services and many of them are doing so in an informal manner with little to no supervision or professional support. Many are using pop psychology, alternative medicine and old fashioned models of intervention, many are just making it up as they go along.

So with that framework we get to the ethics of efficacy (without even addressing the boundaries and morals): are the services being provided that actually are effective and beneficial to the recipient?

1. Are clear goals and time frames set?
2. Are measures of attainment in place?
3. Is there actually a documented research base to the intervention being used?
4. Is there a supervisory check of some sort on the person providing the services?

But mainly is there any benefit at all to the recipient and is there a control to make sure no damage is done.

Take neo-Freudian psychotherapy (of which I despise existentialist the most), there are no clear goals, there are no measures, there is no research base, there is no supervisor of providers. So what happens, the person wastes the money they spend, they don't get better by any measure and they are often traumatized and exploited.

So, to the methods, you have to have measures (be they self report, tracking or observation), you have checks of inter-rater reliability, you have to have a research base to what you are doing.

Otherwise you are exploiting people, often damaging them and wasting their time.

[Iconoclast08]

... And so here is the deal, there are a lot of people who provide these services and many of them are doing so in an informal manner with little to no supervision or professional support. Many are using pop psychology, alternative medicine and old fashioned models of intervention, many are just making it up as they go along. ...

So with that framework we get to the ethics of efficacy (without even addressing the boundaries and morals): are the services being provided that actually are effective and beneficial to the recipient?

1. Are clear goals and time frames set?
2. Are measures of attainment in place?
3. Is there actually a documented research base to the intervention being used?
4. Is there a supervisory check of some sort on the person providing the services?

But mainly is there any benefit at all to the recipient and is there a control to make sure no damage is done. <snip>

Right on. In my view, this really cuts to the heart of the problem in contemporary clinical psychology, although I would add that a huge share of the fault lies with cash-cow, diploma-mill professional psychology training programs, not to mention a fair share of sub-par Ph.D. programs at some rather prestigious universities (New York schools in particular with the psychoanalytic woo still strong at places like Columbia and NYU). Training programs simply don't expose students to the rigors of scientific thinking, nor do they promote much in the way of skepticism and critical thinking when it comes to therapeutic interventions.

Luckily, we do have people like Scott O. Lilienfeld at Emory University who aggressively go after B.S. therapies and hocus pocus "theories", but we could use more of this type of non-fearful, confrontational spirit in an age when people are more than happy to elevate Oprah Winfrey's advice over research, science, and reason.

[Dancing David]

Right on. In my view, this really cuts to the heart of the problem in contemporary clinical psychology, although I would add that a huge share of the fault lies with cash-cow, diploma-mill professional psychology training programs, not to mention a fair share of sub-par Ph.D. programs at some rather prestigious universities (New York schools in particular with the psychoanalytic woo still strong at places like Columbia and NYU). Training programs simply don't expose students to the rigors of scientific thinking, nor do they promote much in the way of skepticism and critical thinking when it comes to therapeutic interventions.

Luckily, we do have people like Scott O. Lilienfeld at Emory University who aggressively go after B.S. therapies and hocus pocus "theories", but we could use more of this type of non-fearful, confrontational spirit in an age when people are more than happy to elevate Oprah Winfrey's advice over research, science, and reason.

Neo-Freudian psychotherapy is unethical and not demonstarted to have any benefit for the recipients of the treatment. There are behavioral methods which are slowly going to dominate psychology, eventualy. I think that any treatment that costs such high sums, damages the healthy functioning of the client and has no benefit is immoral and should be illegal.

Now I am not sure I would label the whole field of clinical psych with a the Freudian label, but I went to a university noted for behaviorism.

[Dr H]

Neo-Freudian psychotherapy is unethical and not demonstarted to have any benefit for the recipients of the treatment. There are behavioral methods which are slowly going to dominate psychology, eventualy. I think that any treatment that costs such high sums, damages the healthy functioning of the client and has no benefit is immoral and should be illegal.

Now I am not sure I would label the whole field of clinical psych with a the Freudian label, but I went to a university noted for behaviorism.

But where will you find gainful employment for all of those out of work Freudians?

As astrologers? :)

[Soapy Sam]

You can't "prove" inductive reasoning either.

But you ignore it at your peril.

[Dancing David]

But where will you find gainful employment for all of those out of work Freudians?

As astrologers? :)

Trash haulers!

[schlitt]

"Science is too important to be left to scientists"

How profound. :rolleyes:

Considering science is a methodology, and once one engages in it they become a "scientist", that quote is not only inane, but nonsensical.

[Personal Grudge]

I would posit that all of life, theory, axiom, what-have-you... is dealing with the best of what we have. Can we prove our methods? Can we prove the Scientific Method? No! Can we prove anything?

But, we can objectively say that what we currently use is working to a certain extent. If a new method were introduced that worked better, I would certainly (perhaps, optimistically?) hope that the community at large would be ready to abandon former methods in favor of the new, more productive method.

The same could perhaps be said about the very nature of skepticism. Currently, the idea of "not accepting any hypothesis without evidence" seems to work very well. Can the idea of skepticism be proven using a skeptical outlook? Perhaps not. But, until a better method is provided, we have to work with what we have.

[Iconoclast08]

...If a new method were introduced that worked better, I would certainly (perhaps, optimistically?) hope that the community at large would be ready to abandon former methods in favor of the new, more productive method.

I doubt that it would happen readily for a variety of reasons, some good (proper dose of skepticism) and some not-so-good, one <of the latter> of which pertains to the dethronement of human arrogance: many people become ego-involved with methods, some more so than others.

The same could perhaps be said about the very nature of skepticism. Currently, the idea of "not accepting any hypothesis without evidence" seems to work very well. Can the idea of skepticism be proven using a skeptical outlook? Perhaps not. But, until a better method is provided, we have to work with what we have.

My sentiments exactly.

[blobru]

I am an empiricist at heart as are many on this forum (with a Popperian-Lakatosian outlook). Recently, I read something in a chapter from the book Bergin and Garfield's Handbook of Psychotherapy and Behavior Change (5th Ed. by Slife) that gave me some pause:

"The doctrines of observability and replicability are not themselves observable and replicable; they are philosophical (or moral) assertions about what should be valued, and they cannot be supported by scientific evidence because, again, such values have to be assumed to gather such evidence." (p. 51)

... Thoughts/reactions/comments?


Nothing too serious.

"Observable" and "replicable" are philosophical assertions. So we support their value not with scientific, but with philosophical arguments.

Can we come up with a philosophical argument that observable and replicable are more valuable to a scientist than nonobservable and nonreplicable phenomena? Shouldn't be too hard. Imagine a science based on nonobservable and/or nonreplicable phenomena: "Our top scientists are convinced the man in the moon is lonely, and since cupid shoots arrows at lonely people, our space program will consist of a giant bow and arrow operated by a naked four-year-old... volunteers?" :scarper:

As others have noted, circularity is about the limitations of any one system and not getting caught up in them. A carpenter can build houses even if he can't build the hammer and nails; a metallurgist can build hammers and nails but not the foundry; neither limitation is serious, if we allow for interdependency. So science and philosophy are interdependent.

[Brian-M]

The doctrines of observability and replicability are not themselves observable and replicable;


I know I'm being pedantic here, but I don't agree with how this statement is written.

How do you observe a doctrine? If doctrines are fundamentally unobservable, then asserting that one specific doctrine is not observable is meaningless.

But if doctrines can be observed (in the statements or actions of the person who holds the doctrine? In written form?) then the doctrines of observability and replicability are observable, and the assertion is false.

Presumably, it is possible that someone unfamiliar with the doctrines of observability and replicability could come up with them independently, in which case the assertion that these doctrines are not replicable is also false.

[Iconoclast08]

I know I'm being pedantic here, but I don't agree with how this statement is written.

Pedantic can be fun. And you make a good point.

How do you observe a doctrine?

That's a tough one, and I don't think it can be done cleanly. What leaps to mind is a scenario in which person A observes person B observing and using the tools of their particular method and to try to somehow capture those variables as independent variables to be studied, but this gets tricky. It also makes me think of recursion or an infinite regress of sorts; who will then observe person B, what methods is person B using in observing person A (probably those same tools as person A since scientific methods would best enable person B to capture what person A is doing, but the problem here is readily apparent --> method capturing method).

I agree with previous posts stating that philosophy and science buttress each other, and along the lines of Godel's theorem, there is no wholly self-justified/proven system in the sense of justification of every single constituent axiomatic principle.

[KingMerv00]

The doctrines of observability and replicability are not themselves observable and replicable

Let's say I take this statement at face value and throw out observation and replicability altogether. How could you function in daily life? You couldn't speak because language is relies on replication. You couldn't drive your car because you wouldn't know how to start it. You couldn't do anything that required thought...at all...ever.

So basically, my answer is I have no choice but to believe in the scientific method. It is the foundation of everything I do and experience.

[Iconoclast08]

Let's say I take this statement at face value and throw out observation and replicability altogether. How could you function in daily life? You couldn't speak because language is relies on replication. You couldn't drive your car because you wouldn't know how to start it. You couldn't do anything that required thought...at all...ever.

So basically, my answer is I have no choice but to believe in the scientific method. It is the foundation of everything I do and experience.

Yes, it would be over the top to just "throw out observation and replicability altogether". This really wasn't the authors' point, though; they didn't suggest that we should just toss these things to the side simply because they weren't formally inherently justified (in the recursive sense I mentioned previously). It was more of a pointing out of Godel's theorem (unbeknownst to the authors) in the context of our own consistently applied methods in a meta-method sense-- that is, thinking about our own methods, e.g., what kinds of values, assumptions, and philosophical underpinnings are in place that we may not necessarily readily think about.

[Beausoleil]

"The doctrines of observability and replicability are not themselves observable and replicable; they are philosophical (or moral) assertions about what should be valued, and they cannot be supported by scientific evidence because, again, such values have to be assumed to gather such evidence." (p. 51)



I think we had a similar discussion here a while back...someone was arguing that the scientific method of evaluating evidence to find out how the universe works had been proved to be the way the universe works, and that this proof was obtained by evaluating evidence... here it is...

http://forums.randi.org/archive/index.php/t-107037.html

I think I decided you either saw the problem with that or you didn't, and attempting to point it out to someone who didn't see the flaw immediately was a waste of time.

The point was being made to argue that science isn't "faith based". I think the correct response is that the "faith" on which we base science is the same "faith" on which everybody relies - Christians have faith that next time they open the Bible to page 23 they will read the same thing they read today, and so on.

I'm not sure what you mean by the doctrine of observability - science deals with observables, it has a restricted domain of operation - we don't need a doctrine to deal with that? For repeatability, I think I stick to my previous response. We assume repeatability in our daily lives - otherwise one is doomed to solipsism - science is no more based on this faith than any part of daily life.

There are some things one has to accept even to partake in daily life - science accepts them. These are not matters of faith - faith refers to "optional" beliefs.

[not daSkeptic]

There are some things one has to accept even to partake in daily life - science accepts them. These are not matters of faith - faith refers to "optional" beliefs.

It has been argued to me (although I do not necessarily accept) that faith is not limited to optional beliefs. I am curious as to by what means one makes a claim either way.