POPS Properties

POPS: Reliability and validity
A discussion of reliability and validity issues with respect to the POPS is provided in Brown et al. (2004). The major point made there is that although one can shed some light on reliability and validity using typical psychometric approaches, such methods are inherently flawed in assessing “reliability” and “validity” with respect to instruments such as the POPS. The challenge of adequately assessing POPS reliability and validity is complex because the instrument is complex, providing both objective descriptive data as well as subjective data, the latter of which presumably draws on an underlying construct (i.e., cross-domain positive-negative view of participation in household and community), while objective data presumably do not. In terms of reliability, the typical assumption of classical test theory that, for example, what is being measured will stay relatively “constant” from test to retest while most of the variance detected is equivalent to error, does not hold fully when what is being measured is “activity in the past week or month”. Proportionately larger real differences in activities from week to week will make “reliability” of reporting appear lower than it “truly” is. Also with respect to assessing reliability, because of the disparate nature of items included in the POPS, within the PO total score and PO subscale scores no high within-scale correlations were expected.

Test-retest reliability of the POPS was tested with repeated measures of the POPS one to three weeks apart on a subsample of 65 people with TBI. Intraclass correlation coefficients for the Time 1 and Time 2 subscale scores were in the range of 0.37-0.89, with the ICC of the total PO score reaching 0.75, while the ICC of the total PS score was 0.80. These values, given the caveats provided above, suggest acceptable levels of reliability. In support of this conclusion, we cite the fact that the 0.89 value was associated with the Domestic Life subscale, for which one would expect a low level of day-to-day true variability. Thus, the subscale with the least expected variability was associated with a high level of reliability of reporting, in people with TBI.

Validity of the POPS was not assessed comparing its pattern of scores to a “gold standard”, as we believe that no measure provides such a standard at present. (None of the existing measures have been shown to be valid (or invalid) for the variety of purposes for which they are used, e.g., documenting individual change, describing population trends, evaluating “success” in a rehabilitation program, etc.) Instead, we developed a series of expectations of how PO and PS scores should perform if they are validly reflecting the constructs targeted by each of the components

First, we expected to see some differences between the subsamples of people with mild versus moderate-severe TBI and a non-disabled comparison group. Second, we expected certain logical consequences of engaging in a low versus a high level of activity in a particular area for the person’s desire for change in that area. For example, those doing nothing or a low amount of the activity would be more likely than those very active to want to “do more” rather than “less” or “the same”. Third, we expected higher correlations of PS scores than PO scores with measures of negative-positive affect such as scales of depressed mood. Strong support was found in the data presented supporting most of the expectations enumerated in the article.

In sum, preliminary evidence is provided that suggests that the POPS can obtain self-report data of an individual’s participation in home and community activities and of his/her feelings about the importance of those activities and how satisfied he/she is with level of engagement. The data appear reliable and valid within the limits of the methods that were available to demonstrate these characteristics of the POPS.