A Computerized Asthma Outcomes Measure Is Feasible for Disease Management
The ASTHMA-CAT, an online, patient-based assessment measure, may facilitate asthma care management.
Patient-based assessments of asthma impact are needed because asthma is common,1-4 disproportionately affects disadvantaged populations,3,5 and presents considerable economic burden.6 In the United States alone, total expenditures associated with asthma were estimated at $19.7 billion in 2007 dollars, with $14.7 billion in direct costs and $5 billion in indirect costs.7 Personal burden also is substantial, as asthma impacts physical, emotional, and social functioning, especially when symptoms are not well controlled.8-12
The National Heart, Lung, and Blood Institute National Asthma Education and Prevention Program (NAEPP) published Expert Panel Report 3: Guidelines for the Diagnosis and Management of Asthma,13,14 which cites 4 key components for long-term asthma control: assessment and monitoring, pharmacologic therapy, control factors contributing to asthma severity, and patient education. Although asthma severity or level of control often is assessed through case history and lung function measurement, NAEPP also recommends the use of patient-reported surveys (eg, the Asthma Control Test [ACT],15-18 the Asthma Therapy Assessment Questionnaire,19 the Asthma Control Questionnaire20) to facilitate and standardize asthma control assessment. Additionally, NAEPP recommends periodic assessment of key areas of health-related quality of life (HRQOL), including decrease in physical or social activities, sleep disturbances, and missed school or work days.13,14
Patient perceptions of health and illness, treatment satisfaction, and self-management skills are clinically significant as they may influence treatment adherence and preventative health behaviors.21-27 Self-management of asthma has been consistently recommended in treatment guidelines,13,14,28,29 and evidence supports the effectiveness of self-management programs.30-37
BACKGROUND
Conceptual Model for Asthma Impact Assessment
eAppendix A (available at www.ajpblive.com) presents an adapted conceptual model underlying asthma impact assessment.38-40 Box 1 represents the most specific clinical measurements (eg, forced expiratory volume in the first second of expiration), followed by asthma-specific symptoms (box 2), specific measures of asthma impact (box 3), and generic measures (box 4) that are not specific to asthma. This research focuses on asthma-specific and generic HRQOL measures (boxes 3 and 4). Condition-specific measures are more sensitive and more responsive to condition-specific changes. Research indicates that the responsiveness of disease-impact assessments is greater than that of generic outcomes,41,42 and that the efficiency of both kinds of measures is increased using modern psychometric methods.43-45 The strength of generic assessments is that they provide a common “ruler” for comparing the burden of diseases as well as comorbid illness, and benefits of treatment across a range of clinical conditions.
Complementary information provided by specific and generic instruments suggests advantages to their combined use.41,46,47 For example, because asthma often coexists with depression or anxiety,48,49 including a generic measure with a mental health screener may be particularly useful for case finding. However, comprehensive measurement using traditional methods is not always practical for clinical application (eg, respondent burden), and short forms may be subject to ceiling and floor effects. New technologies may positively affect asthma management.50-54 Modern psychometric methods (eg, Item Response Theory) can improve HRQOL measures, and competing objectives of more practical and more precise tools can be achieved over a wide range of severity levels using computerized adaptive testing (CAT).55,56
Computerized Adaptive Testing
Adaptive tests choose questions on the basis of an individual’s functional level, similar to what an experienced clinician would do while assessing a patient. Computerized adaptive testing uses a simple form of artificial intelligence that selects questions tailored to the test taker, shortens or lengthens the test to achieve the desired precision, and scores everyone on a standard metric so that results can be compared.56 Each test administration is adapted to the unique level of impact for each respondent (eg, an adult who is able to “walk 50 feet” is not asked a question about “walking 10 feet”). The CAT software first asks a question in the middle of the impact range, estimates a score and confidence interval, then selects the most appropriate question to administer next based on this score. This process continues until predefined stopping rules are met. Computerized adaptive testing requires a large set of items that consistently scale along a dimension of low to high functional proficiency and rules guiding starting, stopping, and scoring procedures.
The Dynamic Health Assessment Asthma Impact Survey (DYNHA AIS) is a CAT that draws on an asthma impact item bank.57,58 Its development was modeled after the DYNHA Headache Impact Test59 and was part of a larger effort to develop item banks for disease-specific CATs.58,60
Study Objective
The research objective was to develop and test an online assessment referred to as the ASTHMA-CAT. Aims were to (1) program the ASTHMA-CAT to include a static full-length AIS item bank or DYNHA AIS (CAT), ACT, and SF-8 Health Survey; (2) pilot-test the ASTHMA-CAT in a disease management population; and for DYNHA AIS, to obtain estimates of item use, respondent burden, range, score accuracy, and patient acceptance compared with the static AIS; and (3) design and evaluate an aggregate (group-level) report.
METHODS
Pilot Study
Participants
Participants were drawn from the Kaiser Permanente (KP) Care Management Institute Central Outcomes Research and Evaluation database and included a random sample of 10,000 adult KP members with chronic disease, 1440 of whom met Healthcare Effectiveness Data and Information Set (HEDIS) criteria for asthma. Letters were sent to these members inviting them to participate in a health impact assessment study. Interested members (32.6%) opted to complete the health impact assessment survey through the Internet or interactive voice response. Our goal for this feasibility study was to sample the first 100 Internet participants to complete the ASTHMA-CAT as part of the larger health impact assessment study.
Instruments
SF-8 Health Survey. The SF-8 is a generic measure of HRQOL outcomes.61,62 One item is used to measure each of 8 domains of health, including physical functioning (PF), role-physical (RP), bodily pain (BP), general health (GH), vitality (VT), social functioning (SF), role-emotional (RE), and mental health (MH). The SF-8 yields an 8-scale profile of norm-based scores (1 for each of the 8 domains of health) as well as physical (PCS) and mental health (MCS) component summary scores. Higher scores indicate better functional health.
DYNHA AIS. DYNHA AIS is a CAT that measures the impact of asthma on HRQOL.57,58 Drawing on a bank of 37 items (which we refer to in this study as the static AIS) covering role, social, mental health, and fatigue domains, it selects and scores only those items required to calculate a precise score for the individual. Item Response Theory (theta) scores are norm based (mean = 50, SD = 10), with higher scores indicating more impact.
Asthma Control Test. The ACT15-18 is a 5-item patient-based assessment designed to measure dimensions of asthma control that underlie current asthma management guidelines,13,14 including asthma symptoms, utilization of rescue medications, and the impact of asthma on everyday functioning. Asthma Control Test scores range from 5 (poorly controlled) to 25 (well controlled). An ACT score of ≤19 indicates asthma control problems.
Background Information. Several items assess sociodemographics, missed work, work productivity (ie, accomplished less at work because of asthma), and asthma severity (5-point response to “How would you rate the severity of your asthma in the past 4 weeks ?”; mild = 1-2, moderate = 3, severe = 4-5).
User Acceptance. Five Likert-type items assess the ASTHMA-CAT’s relevance, ease of completion, usefulness, and length; and the participant’s willingness to complete it again in the future. An additional open-ended comment section is included.
Procedure
Internet study participants were directed to Quality-Metric Incorporated’s www.amihealthy.com Web site to complete the ASTHMA-CAT (see eAppendix B; available at www.ajpblive.com), including the SF-8, either the DYNHA AIS or the static AIS (randomized), the ACT, the background information, and the User Acceptance Survey. DYNHA AIS item selection was based on information value,55 and CAT stopping logic was set at a maximum of 5 items. The study was approved by the KP and New England institutional review boards.
Analyses
Use of DYNHA AIS items was evaluated through an examination of item frequencies. Average response times were calculated, and t tests were used to examine differences between groups completing the DYNHA AIS versus the static AIS. Measurement precision was evaluated by plotting the confidence intervals against AIS theta (Item Response Theory—based) scores. Analysis of variance was used to evaluate score differences by asthma severity for these groups. To compare ratings on the user acceptance items, t tests were used.
Report Development
The aggregate report, designed to display group-level summary statistics, was developed with input from an expert in disease management at KP. Three providers from KP evaluated the report’s content, format, relevance, and usefulness.
RESULTS
Pilot Study
Participant Characteristics
The sample (N = 114) was primarily comprised of white women age 18 to 55 years with mild to moderate asthma (Table 1). Of those 114 participants, 60 completed the static AIS and 54 completed the DYNHA AIS. No significant differences were observed for age, sex, race, ethnicity, education level, or severity in these study groups.
Item Use
DYNHA AIS participants responded to 5 items, whereas static AIS participants responded to all 37 items. DYNHA AIS first administered a “global” item to each participant. Of the remaining 36 items in the bank, 12 (33%) were selected and administered by the CAT. Abbreviated items and frequency of their administration are shown in Table 2.
Response Burden
Previous research59 suggests that only 5 items are needed for an accurate CAT assessment; thus, DYNHA AIS stopping logic was set at 5 items. With this setting, DYNHA AIS offers an 87% reduction in response burden and takes significantly less time to complete than the static AIS (t60 = 9.95, P = .00). Average completion time for the DYNHA AIS was 51 seconds (95% of participants completed it in 1.3 minutes) compared with 6.4 minutes for the static AIS (95% of participants completed it in 14.0 minutes). DYNHA AIS respondents were more likely than static AIS respondents to rate the survey length as appropriate (χ21 = 14.1, P = .00) and easy to complete (t85 = 2.08, P = .04).
Score Range, Accuracy, and Discriminant Validity
DYNHA AIS scores ranged from 35.31 to 68.92, whereas static AIS scores ranged from 31.66 to 59.01. At a group level, DYNHA AIS achieves roughly equivalent measurement precision in score estimation with substantially fewer items administered than the static AIS (Table 3). The Figure shows that for both measures, precision was greatest at the 45 to 70 score range, and worst in the lower impact range as indicated by the larger confidence intervals when scores are below 40 (indicating little impact).
Discriminant validity was evaluated by examining how well each measure distinguished between participants with varied levels of asthma severity. The sample was skewed (75% mild, 22% moderate, and 3% severe) and no participants in the static AIS group reported severe asthma.
DYNHA AIS and static AIS discriminated between asthma severity levels, and scores increased monotonically with severity (Table 4). Additionally, DYNHA and static AIS mean scores were roughly equivalent within severity groups.
User Acceptance
DYNHA AIS respondents were more likely than those in the static group to rate the survey as easy to complete (see eAppendix C; available at www.ajpblive.com). No significant differences in ratings were found between groups for survey relevance, usefulness, or willingness to complete the survey again. Most respondents did not rate the survey as particularly useful in helping them to understand the impact of asthma on their health. All DYNHA AIS respondents rated the survey length as appropriate, compared with 77% of those completing static AIS.
A similar number of respondents in each group provided comments regarding the survey (DYNHA AIS, n = 16; static AIS, n = 19). Many static AIS respondents indicated that the questions were “very repetitive”; none of the DYNHA AIS respondents provided this type of feedback. In contrast, DYNHA AIS was characterized as “very quick and easy.” Suggestions for improvement included changes to the DYNHA AIS interface (“Add a previous key to allow changes to prior responses”) and inclusion of more asthma-specific questions (“You should ask questions asking what types of activity affect one’s asthmatic problems, ie, working out, vacuuming, being around cats, dogs…”). Other comments highlighted the importance of direct feedback to patients (“It has made me want to know more about asthma and how it affects my life.”).
Report Development
An aggregate report was designed (eAppendices D and E; available at www.ajpblive.com). eAppendix D presents the sample demographic profile (number, sex, age, race, ethnicity); scores for the asthma-specific (AIS, ACT) and generic (SF-8) measures with brief interpretation guidelines; and comparisons of sample group scores with asthma and general US population norms. eAppendix E presents scores for the total sample and by subgroup (sex, age, ethnicity, race, education level, geographic region). In addition, the report includes a screen for problems with asthma control (based on ACT scores), a screen for depression (based on SF-8 scores), and an indicator of work productivity (percentage of participants who reported “accomplished less at work due to asthma”).
Pilot data were used to populate the report. Descriptive statistics for this sample show that asthma had minimal impact on functional health, and mean ACT scores were similar to average scores of other asthma patients.57 However, 38% of respondents screened positive for problems with asthma control, 22.8% screened positive for likelihood of depression, and 24.7% reported that they accomplished less at work because of their asthma.
Overall, KP providers evaluated the report positively, indicating that it was useful and relevant to their work and presented important information in a clear manner. They also expressed a willingness to use a report such as this in practice.
DISCUSSION
An ASTHMA-CAT assessment was developed and pilot tested. Results provided evidence of its administrative feasibility, psychometric performance, and patient and provider acceptance. The DYNHA AIS component was completed in one-eighth the time of the static AIS, yet provided equally precise asthma impact scores. Mean DYNHA and static AIS scores were equivalent within severity categories, and both forms discriminated across levels of asthma severity.
One limitation of the study was that the sample included a small number of people with severe asthma, which may have affected DYNHA AIS item use. For instance, the CAT system selects bank items based on responses to the first item, “In the past 4 weeks, how much did your asthma limit your usual activities or enjoyment of everyday life?” If most responded, “Moderately,” “A little,” or “Not at all,” the CAT would administer the next item from this functional range. Other items assessing more limited function would not be selected because they are less relevant for patients with mild or moderate asthma. Further research should examine the DYNHA AIS in a more representative sample.
The AIS scores were less precise in the lower impact range, which is not surprising given that the tool was originally developed to assess disease burden. However, it also is important to provide good coverage in the mild range, so that we can measure the positive effects of treatment.
Interestingly, 38% of our sample screened positive for problems with asthma control (ACT score ≤19). Given the AIS score distribution and self-reported severity ratings, we might not expect as many to screen positive for control problems. As expected, scores on AIS and ACT were negatively correlated (r = −.73), and respondents who screened positive for asthma control problems had higher asthma impact scores (mean = 50.57) than those who did not (mean = 38.18). Although these findings require further investigation, initial results may reinforce the value of a comprehensive asthma impact and control assessment like the ASTHMA-CAT. For instance, patients with mild or moderate asthma severity and impact who simultaneously screen positive for problems with asthma control may require treatment intervention for undetected problems (eg, heavy reliance on rescue medications). A combined asthma impact and control measure can help identify this subgroup and target efforts for improved asthma care management.
Respondents found the assessment to be relevant and indicated a willingness to complete it again in the future. DYNHA AIS respondents were more likely to indicate that the survey was an appropriate length and easy to complete. Many respondents did not rate the survey as particularly useful in helping them to understand the impact of asthma on their health. This rating may be improved by providing patient feedback reports, which can facilitate self-management, an important strategy in the overall management of the disease21,23-27 that may positively affect HRQOL.
Aggregate reports were considered relevant, easy to use, and applicable for care management, which can help organizations identify successes and areas for possible improvement. For example, providers could use the ASTHMA-CAT aggregate reports to screen subgroups for asthma control problems, decreases in work productivity, or likelihood of depression, and then tailor treatment interventions accordingly.
Future research will focus on improvements to the survey interface, expanded AIS item bank content coverage, and development of individual patient feedback reports to help guide clinical decision making.
