The quality framework

The Quality Framework gives a score to the scale requirements…

Psycho-metrics

“Does the scale measure what it claims to?” max score 8

Content Validity is the extent to which the domain in question is comprehensively sampled. It requires clear item selection by more than one expert and a target population to develop scale items. Scores for content validity

• Unclear description = 0

• Clear description by developers = 1

• Clear description involving experts = 2

Compare to benchmarks

Criterion Validity is the extent to which a scale relates to a gold standard. The gold standard must be a genuine and comparable measure with a correlation >=0.7

Convergent Validity is the extent to which a scale relates to another scale that measures something where there is a theoretical liklihood of an association. Scores should correlate 0.3-0.6

• No or inadequate comparisons = 0

• Single criterion or convergent comparison = 1

• Multiple comparisons = 2

Internal consistency is the extent to which items in a (sub)scale are intercorrelated, thus measuring the same construct

• No or inadequate analysis = 0

• Adequate factor analysis or Cronbach's alpha 0.7 = 1

• Factor structure confirmed in different populations = 2

Discriminant validity is the ability of a questionnaire to detect clinically important changes over time or to determine if a condition exists or not

• No or inadequate analysis = 0

• Area under ROC curve >=0.7 or shows clinically significant change or adequate statistic = 1

• Multiple discriminations = 2

“Can the scale easily be applied to routine practice?” max score 6

Number of Items and Completion Time are related indicators of user acceptability. The total assessment package should be completed in a reasonable period of time – say 10-20 minutes. It works well to have a scale fit on an A4 sheet. An assessment package is likely to be made up of 3-4 scales

• 21+ items = 0

• 13-20 items = 1

• 1-12 items = 2

Clinically Significant Change is the gold standard of psychological treatment outcome. The calculation requires a value for reliable change (the measurement error) and a value for a well functioning population completing the scale

• Neither value published = 0

• One value published = 1

• Both values published = 2

It is important to have the value for reliable change not just as a way of demonstrating that the scale measures change but how much the score needs to change

Ease of Use means scales are better if they can be i) administered and ii) scored with minimal training and without the need for complex scoring, and iii) easy to understand, but users should always have some tutoring as to the correct interpretation of measures

• Training and knowledge to interpret are both needed = 0

• Minimal training and meaning is plain = 1

• Minimal training and user guide available = 2

NOTE: it is often desirable to use a scale that can be applied to most situations - this may be a factor determining suitability

“Is it practical for an agency to use the scale?” max score 8

Independent Evaluation is important but many scales rely on only one publication by the original creators of the scale. Independent evaluation strengthens validation

• Authors’ publications only = 0

• One independent publication = 1

• Several independent publications = 2

Cross Cultural Evaluation strengthens validation and may be crucial to generalising a scale’s use depending on the target population

• None found = 0

• One or limited culturally diverse groups = 1

• Several culturally diverse groups = 2

Language Check is evidence of testing for plain English (or other language) or service user feedback on the wording of the scale items and instructions

• Not found = 0

• Evidence of some checks = 1

• Formal and User checks = 2

It is better if scales are in the Public Domain or have a Creative Commons licence so that the scientific community and clinicians can use them freely = 1

It is better if there are no copyright constraints or fees for using scales = 1