Psychometrics question.

[TruthSeeker]

I am reviewing a manuscript which used a combination visual analog and semantic differential scale to measure depression and anxiety.

Specifically, the anchors for depression are "cheerful" and "depressed" and for anxiety are "calm" and "nervous". Instead of fixed intervals for choices (thus an ordinal scale), there is a solid 10 cm line for subjects to rate themselves.

Ordinarily, a visual analog scale is used for rating construct X from "no X" to "maximum X". THe length of line from "no X" to the subject's mark is the "amount" of X they report. Usually this is treated as ratio data (although that has been debated)

Given the combination of the two scale types, what sort of data are these? Marking the cheerful end may mean the person is cheerful but does that necessarily mean they are not depressed? Perhaps it would be more meaningful to measure distance from the centre of the line (although this is not marked "neutral")

Am I just being obsessive?

Any thoughts? Psychometrics was a very long time ago.

Thanks

[jj]

Originally posted by TruthSeeker
I am reviewing a manuscript which used a combination visual analog and semantic differential scale to measure depression and anxiety.

Specifically, the anchors for depression are "cheerful" and "depressed" and for anxiety are "calm" and "nervous". Instead of fixed intervals for choices (thus an ordinal scale), there is a solid 10 cm line for subjects to rate themselves.

Ordinarily, a visual analog scale is used for rating construct X from "no X" to "maximum X". THe length of line from "no X" to the subject's mark is the "amount" of X they report. Usually this is treated as ratio data (although that has been debated)

Given the combination of the two scale types, what sort of data are these? Marking the cheerful end may mean the person is cheerful but does that necessarily mean they are not depressed? Perhaps it would be more meaningful to measure distance from the centre of the line (although this is not marked "neutral")

Am I just being obsessive?

Any thoughts? Psychometrics was a very long time ago.

Thanks

There is a very serious implication built into this scale, I think.

It implies that "cheerful" and "depressed" are opposites.

I have been momentarily cheerful in the midst of a situational depressions. So has most anyone else, I suspect. I can feel depressed in the short term, although I'm not clinically depressed at all.

I don't think the two are opposite enough, at least as described. Does depression exclude all chance of cheerfulness? Without some way to address both short and long term trends, you've got two confusing aspects.

Oh, and that's just a start.

Calm and nervious might be a better set of opposites, but then again, the long vs. short-term aspects come in there as well.

[bpesta22]

What's the reliability of the survey?

If it's reliable, I wouldn't worry about whether the data are ordinal or interval.

I assume they used the survey to predict something significantly, or to show that two groups differed on it.

If so, they must be measuring something consistently.

I guess the only issue left is validity.

But, if they didn't report reliabilities, I'd reject it. Plus, why are they using their own made up survey to measure this stuff-- tons of validated depression scales already exist

[jj]

Originally posted by bpesta22
What's the reliability of the survey?

If it's reliable, I wouldn't worry about whether the data are ordinal or interval.

I assume they used the survey to predict something significantly, or to show that two groups differed on it.

If so, they must be measuring something consistently.

I guess the only issue left is validity.

But, if they didn't report reliabilities, I'd reject it. Plus, why are they using their own made up survey to measure this stuff-- tons of validated depression scales already exist

Well, though, if it IS reliable, what are they measuring? At least one of the scales seems ambiguous to start with, and likely prone to different interpretations by different subjects.

One could just as well be testing how subjects interpret a question here.

[bug_girl]

it seems to me that this is still ordinal data. there are a finite amount of choices on the line, (less than cheerful and more than depressed) and a limit to the amount of measurement precision you can get when quantifying their responses on the line.

[bpesta22]

I'd have no problem doing inferential stats on these data, as long as the scale is reliable, and the results are replicable.

I find using the nonparametrics leads to the same stinkin' conclusions anyways, unless of course it kills your power.

JMO

[bug_girl]

can you define reliable for me?
i think we are coming from two different statistical camps:)

[bpesta22]

Originally posted by bug_girl
can you define reliable for me?
i think we are coming from two different statistical camps:)

The correlation between true and observed scores-- that all the tests items measure the same unidimensional construct-- Cronbach's coefficient alpha

[TruthSeeker]

Thanks so much for the great input. I'm just sitting down to work on it again...

I'll briefly summarize the design with the hope of not revealing too much (confidentiality and all!)

The authors are trying to assess gender differences in daily fluctuations in depression and anxiety and how this relates to symptom burden in palliative care patients.

Patients completed the measures once per day for 14 days. That is all the info they provide about that (no mention of when during the day, how they insured compliance on a daily basis rather than subjects filling in all the days retrospectively on the last day.)

As for issues of power and reliability etc., the sample size is small and the cells unequal (33 women and 17 men). Statistically, these numbers are inadequate but in a palliative setting it isn't surprising. Response rates in minimally intrusive studies are only about 30%. Given the 14 days survival and cooperation required for this study, the sample size makes sense. Unfortunately, the authors do not report attrition rates. So, it is unclear whether they are reporting on the 50 patients who completed the entire study or the 50 who agreed with a diminishing n over time.

Like jj, I'm still struggling with the validity of the measure. Cheerfulness just doesn't feel like the opposite of depression. On the well-validated scales, low scores do not indicate happiness or cheerfulness but rather lack of depressive symptomatology.

[Mercutio]

Originally posted by bug_girl
it seems to me that this is still ordinal data. there are a finite amount of choices on the line, (less than cheerful and more than depressed) and a limit to the amount of measurement precision you can get when quantifying their responses on the line. agreed. Seems like a weasel-like way to pretend to have more precision than they do. It is, to my thinking, very obviously an ordinal measure.

[TruthSeeker]

Originally posted by Mercutio
agreed. Seems like a weasel-like way to pretend to have more precision than they do. It is, to my thinking, very obviously an ordinal measure.

This is pretty much what I wrote...including crediting Mercutio and bug_girl for statistical consultation! :D

[Mercutio]

Originally posted by TruthSeeker


This is pretty much what I wrote...including crediting Mercutio and bug_girl for statistical consultation! :D Wait a sec...*checks post times* 4 minutes between my post and yours...I think bug_girl gets full credit for this one!:)

[TruthSeeker]

Originally posted by Mercutio
Wait a sec...*checks post times* 4 minutes between my post and yours...I think bug_girl gets full credit for this one!:)

But I like the look of (bug_girl & Mercutio, personal communication, 2003) :D


When I was an eager beaver Post doc, I used to include reference lists with my reviews. Now, I write them at 11:00pm on my way to bed! I'm getting old.

[T'ai Chi]

So, just to clarify, are we dealing with a graph where the horizontal axis is Depression (ranging from "cheerful" to "depressed") and the axis is also Anxiety (ranging from "calm" to "nervous"), where Depression is measured using the Visual Analog scale (as you described, ratio level data) and Anxiety is measured using the Semantic Differential scale (7-point rating scale, ordinal level data), and you're asking what data level we'd consider both measurements, Depression and Anxiety, at the same time, and say a measurement for one person's X on the line would be the pair (Depression level, Anxiety level) ? And you're not too sure if it is ok if the VAS could be considered ordinal?

Please let me know if I'm on the right track with the above.

[bpesta22]

some points:

1) if so called ordinal data produce interpretable, meaningful results using "interval" statistics, wouldn't that imply the data might indeed have interval characteristics?

2) Can't you tell from the degrees of freedom which sample is included in any analyses?

3) I wonder what stats they did to test their hypothesis?

Did they use some type of difference score data on the fluctuations across the 14 days? Or, did they just compare the day 1 to day 14 difference by gender.

Seems they're interested in variability versus overall score. So, I think the only sensible way to do it would be to use the standard deviations for each subjects 14 test scores. The authors would predict significant SD differences across gender

Also, it seems like the across people variablity in depression would produce so much noise in the data, that showing within-preson change across time would be difficult.

Finally, assuming depression is a relatively stable trait-- we have a good name for the daily fluctuations they might observe within people. That'd be "error in the test," or unreliability.

[Wudang]

During the course of my psychology degree I had to give out a lot of psychometric tests. In one, pursuing my own interest in face validity, in a questionaire (whose primary purpose escapes me now but of the rate "someone on a scale of 1 - 10" variety), I slipped in an "intelligent versus likeable" question. And the general response I got was "Okay I think I see what you mean". I would have liked to have taken that further.

[TruthSeeker]

Originally posted by T'ai Chi
So, just to clarify, are we dealing with a graph where the horizontal axis is Depression (ranging from "cheerful" to "depressed") and the axis is also Anxiety (ranging from "calm" to "nervous"), where Depression is measured using the Visual Analog scale (as you described, ratio level data) and Anxiety is measured using the Semantic Differential scale (7-point rating scale, ordinal level data), and you're asking what data level we'd consider both measurements, Depression and Anxiety, at the same time, and say a measurement for one person's X on the line would be the pair (Depression level, Anxiety level) ? And you're not too sure if it is ok if the VAS could be considered ordinal?

Please let me know if I'm on the right track with the above.

I'm not sure if I follow you but I think I haven't been clear. If there were a graph, it would be day (1-14) on the X-axis and depression (or anxiety) on the Y-axis. Both anxiety and depression are measured using the same Visual Analog-Semantic differential type scale (a separate rating for each construct).

The ratings are considered separately for all the analyses.

HOpe that helps to clarify.

[TruthSeeker]

Originally posted by bpesta22
some points:

1) if so called ordinal data produce interpretable, meaningful results using "interval" statistics, wouldn't that imply the data might indeed have interval characteristics?

My response:
I guess, in a way, this is what I'm asking about.

2) Can't you tell from the degrees of freedom which sample is included in any analyses?

My response:
Degrees of freedom not reported! I really really hate that. This particular journal (highest impact factor in the field) will publish with only the p value. UUUUUUUUUUGGGGGGGGGGGHHHHHHH

3) I wonder what stats they did to test their hypothesis?
Did they use some type of difference score data on the fluctuations across the 14 days? Or, did they just compare the day 1 to day 14 difference by gender.

Seems they're interested in variability versus overall score. So, I think the only sensible way to do it would be to use the standard deviations for each subjects 14 test scores. The authors would predict significant SD differences across gender



My response:
Day by day differences by gender using t-tests! No repeated measures analysis. I've already commented on this in my review.




Also, it seems like the across people variablity in depression would produce so much noise in the data, that showing within-preson change across time would be difficult.

Finally, assuming depression is a relatively stable trait-- we have a good name for the daily fluctuations they might observe within people. That'd be "error in the test," or unreliability.

My response:
I agree. No mention made of this by the authors. I've asked them to reconsider naming the constucts "mood" which may fluctuate meaningfully rather than "depression"

I will be so glad to send off this review this afternoon.

Thanks for the input!