Dan-Andrei Waitman, MD, MPH

: The risk of clinically significant drug-related problems is a major concern, particularly among elderly, polypharmacy-treated patients. Previous efforts to develop computerized alert systems have shown limited effectiveness. The purpose of this study was to evaluate the efficacy of a novel technology, the Drug-Drug Interaction Plus (DDI+) decision support system, used in a real-world ambulatory health maintenance organization (HMO) setting and developed to overcome the limitations of currently available programs.

: We conducted a controlled trial of the DDI+ system in Leumit Health Services (LHS), an HMO in Israel that provides coverage to approximately 700,000 members with approximately 2000 physicians nationally. All LHS physicians were geographically allocated by region to a study or control arm. Physician access rates, alert resolution rates, and resource utilization of their patients were evaluated and compared between the intervention and control arms.

 = .001) fewer mean episodes of hospitalization, 1.5% (

= .01) fewer mean drugs dispensed, and 2.1% (

 = .055) fewer mean episodes of imaging in

the intervention population versus controls. Drug-related problems that were re-encountered were often rectified without

: Comparing the acceptance by physicians of the DDI+ system to reports of acceptance of similar systems, DDI+ was observed to be better accepted. This will possibly initiate improvements in resource utilization patterns. We surmise that improved physician willingness to access the system can be attributed to the novel graphic presentation, specificity, and integration of DDI+ into the work flow of routine clinical practice.

Drug prescribing is among the most powerful and ubiquitous tools employed by physicians for treating patients. Over the decades, thousands of safe and effective drugs have been added to the armamentarium at physicians’ disposal, empowering them to treat most of the acute and chronic conditions encountered in the community setting. 

These developments, however, while generally positive, have not emerged without adding concomitant risks of harm to the patient. Administration of multiple drugs, and of herbal preparations, vitamins, and food supplements, directly increases the risk of avoidable medication error—induced sequelae, which include adverse effects, pharmacokinetic interactions, decreased efficacy, morbidity, mortality, and unnecessary resource use.

Furthermore, the degree of risk is exacerbated when multiple physicians practicing various subspecialties independently prescribe medications for a patient in the absence of a case manager whose job would be to actively coordinate a cohesive care plan. 

The risk of clinically significant drug-related problems (CSDRPs) is of major concern, particularly among elderly, polypharmacy-treated patients in whom the incidence of major iatrogenic events is estimated at more than 25%.

 Overall, CSDRPs are responsible for approximately 2% to 5% of all hospital admissions.

 Unfortunately, the growing use of electronic health record (EHR)-based modules for addressing CSDRPs has not dramatically changed the occurrences of such outcomes, given that more than 90% of physicians override or do not use these modules.

Accordingly, the development and implementation of intervention strategies, such as the requirement for pharmacists to assess drug-related problems prior to dispensing and improved uptake of existing technologies to prevent CSDRPs, is a well-established practice in managed care systems. 

Although significant progress has been made in developing new technologies, the applications currently available provide suboptimal prevention of CSDRPs. Their many shortcomings include lack of systematic assessment of pharmacokinetic/pharmacogenetic data,

 long and difficult-to-comprehend textual outputs,

 lack of suggestions for alternative drug options, an inability to evaluate drug combinations involving more than 2 concomitantly prescribed drugs, and limitation of use to the prescribing phase of the visit.

Additionally, because these systems are seldom evaluated in robust controlled trials, few data are generally available to clinicians and medical managers to evaluate the potential value of these technologies in a real-life clinical setting. 

 A new effort to overcome many of these limitations is the Drug-Drug Interaction Plus (DDI+) decision support system (Mediseen eHealth Ltd, Teva Pharmaceutical Industries Ltd, Israel). The DDI+ output is based on a unique technology that synchronizes and integrates pharmacodynamic, pharmacokinetic, and pharmacogenetic data from leading international databases (First Data Bank and GeneMedRx/Genelex United States) along with patient-specific factors accessed from the patient’s EHR in real time.

The system is periodically updated with new relevant data; it incorporates innovations designed to overcome the limitations of other currently available systems to improve physician uptake and to harness the data available from EHRs to enhance the precision and specificity of CSDRP identification. 

DDI+ presents real-time output via a visual user interface that enables detection, prioritization, and alternative suggestions within a few seconds while presenting only essential output to physicians throughout the entire patient office visit. DDI+ is designed to flag most of the clinically significant interactions that may occur between a drug and any other factor that may affect the pharmacokinetics/pharmacodynamics of administered medications (allergy/sensitivities, renal functions and liver function, food, vitamins, herbals, genomics, smoking, age, gender).”

Additionally, in contrast to other programs limited to evaluating 2 drugs at a time, the design of DDI+ allows it to evaluate drug regimens composed of 2 or more agents to be administered concomitantly. Furthermore, DDI+ can incorporate pharmacokinetic data to prevent not only improper dosing, but also false-positive alerts that might otherwise occur when inadequately robust literature would indicate interactions. DDI+ can detect when drug-serum levels would not be affected by particular doses of 2 concomitantly prescribed drugs, despite less-specific information to the contrary in the literature. 

We hypothesized that incorporating these improvements would overcome the barriers that have impeded physician use of previous systems, thereby yielding measurable improvements in the use of the following resources: hospitalization, number of drugs purchased, and imaging. The purpose of this study was to prospectively evaluate the efficacy of the DDI+ system to improve the efficiency of care in a real-world ambulatory health maintenance organization (HMO) setting via a large, robust, controlled trial.

DDI+ is a Web-based system that interfaces and operates in the background of an EHR. Throughout a patient visit, the physician sees that the relevant icon on the toolbar of the EHR’s main page is either “red” or “green”: red signifies potential CSDRPs and green signifies their absence. When the icon is red, the physician can enter the DDI+ system either volitionally or mandatorily, depending on organizational policy.

The system immediately displays the detected CSDRPs and provides information necessary to choose alternative regimens or make drug changes. Upon the physician’s selection of a regimen modification or the drug to be changed, the system displays potential alternatives; it then calculates the outcomes of these (1 or more) new potential regimens, facilitating the prescription of safer alternatives.

Once a CSDRP is rectified, the icon automatically switches from red to green. This platform thereby is designed to reinforce, albeit not replace, sound clinical judgment (nor does it obviate the need for independent learning). 

The DDI+ system platform was installed to interface with the EHR program used by Leumit Health Services (LHS), an HMO in Israel that provides health coverage to approximately 700,000 members with approximately 2000 physicians practicing in LHS-operated clinics or in private practice nationally.

LHS first implemented its EHR system in 1999 and has achieved 100% uptake among all physicians treating LHS patients in the community setting. Complete computerization has facilitated the construction and maintenance of a data warehouse that includes comprehensive records of the following: drugs prescribed and dispensed within the system; laboratory tests ordered and their results; referrals to specialists; diagnostic imaging; hospitalizations, including scanned discharge letters; and ambulatory procedures performed at LHS clinics.

Drugs prescribed outside the LHS system are not initially captured, but if a prescription is filled at an LHS-affiliated pharmacy or if the drug is later re-prescribed by an LHS physician, the system will document the prescription. Furthermore, LHS has developed and implemented data-capture and analysis tools for the EHR data warehouse that enable both longitudinal and retrospective observation of resource use in its defined patient and physician populations.

All elderly patients (≥65 years) receiving polypharmacy (≥5 concomitant chronic medications) were included for healthcare use assessments. Access rates and the system’s educational capacities were evaluated only for primary care physicians who treated these elderly patients (n = 369 and n = 346 for the intervention and control arms, respectively) as they were the main target population of the DDI+ system from the LHS perspective. 

Study Design, Data Accrual, and Outcome Variables

The LHS management structure is subdivided into 4 geographic regions subordinate to a central management. Physicians were geographically allocated by region where they practice to a study arm that received access to the DDI+ platform (North and Central regions) or a control arm that was not (Jerusalem and South regions).

Because the North and South regions and the Central and Jerusalem regions are similar in patient and physician demographics, the 4 regions were accordingly allocated into 2 groups. The allocation of these 2 groups into study and control arms was arbitrary. We chose to implement the system on regions in their entirety and not to randomize individual physicians in order to prevent cross-group contamination among physicians in different arms of the study working in the same region.

Neither arm was exposed to other decision-support technologies. After a run-in period that included an individual 30-minute training program, the system was formally launched in the study arm of physicians on January 1, 2013, for a 6-month controlled study period. Use of the program was voluntary, and physician access patterns during the study were not monitored to promote use among physicians who declined to do so.

Data were accrued and compared for both groups. To evaluate the efficacy of the system to reduce resource use, we calculated and compared the average number of episodes of hospitalization and imaging and the number of drugs dispensed per patient in both arms during the study period with those observed in these patients during the 12-month preintervention period. We then calculated the relative changes observed between the pre- and postintervention periods.

The rationale for analyzing the use of imaging was that because a physician may not immediately identify a primary complaint as a CSDRP, these patients are frequently referred to diagnostic procedures, such as imaging, to rule out suspected pathologies included in the differential diagnosis.

The effectiveness of the system to rectify alerted problematic drug combinations was measured by calculating the percentage of red alerts that turned green among patients treated by physicians who accessed the system compared with the rate observed among patients treated by physicians in the intervention arm who did not access the system. 

Additionally, the change in number of “severe” alerts present among patients treated by DDI+ users was calculated longitudinally over the 6-month study period. Changes in drug utilization patterns after an alert was rectified were measured by calculating the average number of drugs dispensed to patients before and after the first time a red alert turned green.

Additionally, we evaluated physician acceptance and application of the system by measuring both crude physician use rates (rates of physicians accessing the system at least once) and the proportion of alerted episodes in which the physician accessed the system. Furthermore, because we postulated that the system might yield a residual educational effect, we identified and quantified episodes of drug switching without accessing the system following an episode involving the same clinical scenario in which the physician heeded the alert. 

Data capture was performed using IBM Cognos 10.1.1 BI Report Studio software (IBM, Armonk, New York). Results of queries were downloaded into Microsoft Excel (version 14) spreadsheets for statistical analysis. Student’s t test, χ2 test, and 1-way analysis of variance were used as appropriate.

All results are expressed as average ± standard deviation, percent with P value, or 95% confidence interval. A P value ≤.05 was considered statistically significant. The data were analyzed using SAS version 9.3 (SAS Institute, Cary, North Carolina).

The study was approved by LHS’s Institutional Review Board and was registered (ClinicalTrials.gov identifier: NCT 01765595). 

The characteristics of the intervention and control populations of patients and physicians appear in Table 1.

Differences in physician gender and average number of primary care physician visits during the year prior to the study period were found to be statistically but not clinically significant. Statistically significant variance between the 2 groups was not observed in the other variables measured, including the number of distinct drug products received during the year prior to the study period, a proxy variable for number of comorbidities and disease severity. 

The changes in resource use in the intervention and control arms (difference-in-differences methodology) appear in Table 2. Although a rise in utilization of resources was observed in both groups, there were 5.6% (P = .001) fewer mean episodes of hospitalization, 1.5% (P = .01) fewer mean drugs dispensed, and 2.1% (P = .055) fewer mean episodes of imaging in the intervention population versus controls.

A subset analysis of the primary physicians in the intervention arm who accessed the system was conducted. During the 6-month study period, 272 of the 369 primary physicians (78.6%) accessed the system at least once. At least 1 episode of a “red alert” was documented during 104,711 visits. Physician voluntary access rates during these visits was observed to be 12.9% (n = 13,534).

A red alert was observed to be converted to green during 5386 (39.8%) of these 13,534 visits, versus 0.6% (P <.0001) among physicians who did not access the system despite having been exposed to the red-alert icon (Table 3). Among the 272 physicians who accessed the system at least once, 232 (85.3%; 95% CI, 80.5%-89.3%) were observed to appropriately respond to later alerts without subsequently accessing the system, correctly rectifying 68.9% of the potential CSDRPs re-encountered.

At the conclusion of the 6-month study period, 42% of the initial “severe” alerts had been resolved by the physicians in the intervention arm who accessed DDI+ at least once (P = .023) (Figure). Additionally, the average number of drugs dispensed to a patient after a red alert was resolved decreased by 7.8%, from 9.77 ± 3.62 to 9.01 ± 2.93 medications per patient (P <.05).

The observed success of this program compared with other marketed systems can be attributed to the unique features incorporated into the system to overcome the shortcomings of previous efforts. First, and most significant, the DDI+ system is interactively linked to the HMO’s EHR system, which facilitates inclusion of all clinical data available to inform personalized drug choice decisions.

We understand that this feature generates confidence among physicians concerning the relevance of the alternatives, thereby allaying previously observed apprehensions caused by the low specificity (eg, frequent false-positive alerts) that characterized other systems, Second, in contrast to other programs, the DDI+ system processes the “net” outcome of drug combinations of more than 2 medications administered concomitantly.

This unique feature may have afforded an elevated perception of reliability among physicians equipped with the system. Third, significant efforts were made to provide the physician with a user-friendly visual interface that enables a favorable workflow, one that streamlines integration of the system into routine care without disrupting the dynamics of the office visit.

Accordingly, the system facilitated rapid detection, prioritization, and complex problem solving. Moreover, because the “red alert” was visible even without accessing the system, it is possible that this notification was sufficient to motivate physicians to resolve the warning without the aid of the system.

Fourth, in contrast to other systems that flag only potential severe interactions, the DDI+ system is designed to provide information to guide clinical decision processes for prescription of safer, alternative drug regimens. Additionally, although not measured, it is possible that presenting only essential output to physicians diminished alert fatigue, thereby improving motivation to access the system.

Articles

A controlled trial of a system for rectifying and preventing drug-related problems was conducted to evaluate improvement in the efficiency of resource utilization.


Large-Scale, Community-Based Trial of a Personalized Drug-Related Problem Rectification System

Physician input when designing interventions for the promulgation of clinical practice guidelines (CPGs) can be an important component of the formulation and implementation process. Focus groups, a form of group interview that capitalizes on communication and interaction between research participants to generate data and gather information, can be a useful method for acquiring this potentially vital feedback. During focus groups, participants are encouraged to converse with one another: asking questions, exchanging anecdotes, and commenting on each others’ experiences and points of view.

 Consequently, of primary importance when conducting a focus group is creation of a permissive, comfortable, and nonjudgmental environment that fosters self-disclosure and encourages participants to share perceptions and points of view.

 Development of techniques the moderator can use to stimulate conversation among the participants is essential for conducting a successful focus group. 

In this study we investigated the feasibility of using interactive electronic-polling technology and nominal group technique (NGT) to present nationwide prescribing patterns for the purpose of initiating group discussion among primary care physicians. (For descriptions of these techniques, see the Methods section.) Brezis and Cohen have described their experience with these systems in teaching both students and physicians and have concluded that “this modality improves the quality of clinical learning and deserves further evaluation.”

 Because this modality facilitates anonymity and self-disclosure, we hypothesized that it could be used by researchers to attain the qualitative research objectives of focus groups.

As drug utilization studies conducted in the Leumit Health Fund of Israel (Leumit) have consistently shown low rates of physician adherence to Health Maintenance Organization (HMO) guidelines for use of antibiotics in the treatment of acute cystitis,

 we chose implementation of guidelines for the choice of drug to treat uncomplicated cystitis in adult community-dwelling women as the test case for this novel modality. We hypothesized that presenting prescribing data to small groups of physicians would elicit group discussion that would be beneficial in furthering our understanding of the underlying factors behind the prescribing behavior observed.

The primary research objective of the focus groups was to gain insights from practicing physicians concerning the observed nonadherence to the HMO’s guidelines, to learn why past efforts to promulgate HMO policies have failed, and to learn what tools and methods might be more successful in the future. The major themes ascertained from the focus groups would then be used to formulate a questionnaire to survey a large sample of HMO primary care physicians.

All primary care physicians practicing in clinics operated by the Leumit Health Fund in 3 different regions of Israel were invited to attend focus groups to discuss the implementation of guidelines for the treatment of infectious diseases. The physicians were representative and all saw a substantial number of cystitis cases. Similarly, because all invited physicians were full-time, salaried employees of the HMO, their knowledge levels and attitudes were relevant to our research objective. The sessions were conducted in a quiet room with the participants seated in a semicircle facing the screen with the presentation. The sessions were partly conducted during regular working hours or during the afternoon break immediately after the clinics were closed. A buffet lunch was served. 

The nominal group technique is a semiquantitative evaluation method implemented in a group setting that combines both qualitative and quantitative elements of data collection. It encourages members of the group to contribute their individual thoughts about an issue of common interest, and during an NGT session all participants have equal say in generating and rank-ordering of ideas and suggestions.

 Furthermore, because NGT sessions are driven by the participants and not by the perceptions of the researcher, the influence of the researcher and the effect of group dynamics are minimized.

Interactive-voting systems are a technology that is implemented in a group setting to induce audience participation. As the lecturer poses multiple-choice questions to the audience via a PowerPoint presentation, each participant responds by pressing the number key on a remote control electronic keypad corresponding to their chosen answer. A computer-based device retrieves and processes the responses. When cued by the moderator, the system displays the distribution of answers as a histogram. This immediate feedback to the lecturer and audience can then be channeled to initiate group discussion concerning possible explanations for the results observed.

The questions for the sessions appear in the 

). First, the moderator (NRK) presented a clinical vignette describing a case of a young woman diagnosed with simple uncomplicated cystitis. The participants were then asked via a multiplechoice question which antibiotic they would prescribe for the empiric treatment of this patient and the distribution of their answers was presented as a histogram, as described above. Although a trained moderator is generally a good idea, the structure of these focus groups with the NGT worked well. Also, the moderator had studied qualitative research methods with one of the authors (DPC), who teaches the course in qualitative methods at the Hebrew University Hadassah School of Public Health. The moderator’s familiarity with the organization and the data used in the focus group discussions contributed to the success of the process.

Second, the moderator asked participants via multiple-choice questions what they believed to be the current prescribing and antibiotic resistance patterns nationally in the HMO; again the distribution of answers was presented as a histogram. Afterward, the moderator presented the results of drug utilization studies and uropathogen resistance data that report the current patterns for this HMO’s patient and physician population. This immediate feedback then generated group discussion conducted with an NGT process previously described by Glasper et al,

 modified by the research team to be applicable to our research objectives as follows. 

Steps in the Nominal Group Technique Process

Step 1: Silent Generation of Ideas in Writing. 

The first question opened for group discussion was why do you think Leumit physicians did not generally comply with the guidelines for the empiric treatment of uncomplicated urinary tract infection in adult women? The participants were requested to write on index cards what they believed to be the 5 major reasons that HMO physicians did not generally adhere to the guidelines. 

The participants were asked in turn to present the first point recorded on their index card. Participants were instructed to say “pass” if at their turn they no longer had any original ideas to present. The moderator recorded each original contribution on a paper flip chart. This was done until the participants had exhausted their ideas. During this stage conferring and discussion was not permitted.

After all ideas had been recorded on the flip chart, the statements presented were discussed and clarified until the moderator perceived that the participants adequately understood the ideas. Similar ideas were collapsed into a single item to avoid duplication and dilution of impact in the subsequent voting stage.

The participants were asked to choose and rank what they thought were the top 5 ideas recorded on the flip chart. On the reverse side of the index card used in step 1, the physicians anonymously recorded in descending order of importance the 5 ideas (out of the 5-8 remaining after collapse) appearing on the flip chart that they believed best addressed the issue under discussion. Upon completion of this step, the moderator collected the cards.

The ideas listed on the cards were assigned a numerical rank order of 1 to 5 according to descending value of importance. The rank allotted by each participant for each item listed on the flip chart was tabulated. Items not scored by the participant were given a value of 6. Sums of the ranks allotted for each item were calculated. The rank order of the items was evaluated from this aggregate data. 

To prevent sessions from exceeding 90 minutes in length, vote tabulation was not conducted with the participants. Because exhaustive discussions already had been conducted at a previous stage to evaluate and clarify the merits and disadvantages of each idea, we saw no need to implement an additional time-consuming discussion session to evaluate the group ranking of the ideas. The purpose of this study was to identify the major themes to be later evaluated quantitatively via a survey of a large sample of physicians, not to calculate their relative ordinal value at the time of the sessions.

This process was repeated during a second discussion stage in which the participants were asked the following question: What techniques do you think would be most successful in the future for implementing clinical practice guidelines in the Leumit Health Fund? During this NGT process, the flip chart page with the ideas generated in the previous round was left on display. Participants were instructed to generate ideas for implementation that in their opinion would be effective in overcoming the barriers identified in the previous round of discussion. 

The sessions were taped and listened to in full. Supplementary information derived from the discussions that might enhance our understanding of the data captured on the flip chart (eg, anecdotes from routine practice, discussion during the electronic voting phase) was documented and analyzed.

The major findings of this study have been reported previously.

 Three focus groups comprised of 6, 10, and 15 physicians were conducted, each lasting approximately 90 minutes.

The results from the first round of discussion concerning barriers to guideline implementation are shown in 

. The 3 sessions generated 8 different ideas explaining the lack of adherence to the guidelines as follows: 

Drug of Choice Periodically Out of Stock. 

Over the past 4 years the supply of nitrofurantoin in Israel often was sporadic, with supplies often being depleted for extended periods of time.5 This point was raised during all 3 focus groups as contributing to the lack of adherence even during periods when the drug was in stock.

Similar to the findings of previous research,

 the physicians in all 3 groups attributed the prescribing behavior to lack of knowledge of the content of the guidelines. Additionally, the participants noted that they themselves were unaware of the current uropathogen resistance patterns as presented in the first part of the session by the moderator. 

Disagreement with the recommendations of the guideline was suggested by 2 of the groups to explain the findings. The major points of contention raised during the open discussion were that unlike ofloxacin, nitrofurantoin often causes gastric side effects and that patient adherence with ofloxacin is superior because it is taken twice daily as opposed to nitrofurantoin, which is taken 3 times daily.

One group attributed the results to what they described as an organizational flaw whereby physicians do not receive feedback concerning their prescribing habits. This theme was later developed and expanded in the second round of discussion.

One group raised the issue of patients who were treated in the emergency rooms of hospitals during hours when HMO clinics are closed, received a 1-day supply of the drug prescribed, and then later visited a Leumit physician to get an HMO prescription for the rest of the course of therapy with HMO coverage. The physicians claimed that because of this scenario, in many cases their prescriptions do not reflect their own clinical judgment. One physician expressed his aggravation with this situation by proclaiming that he was “fed up with being reduced to being a rubber stamp.”

Similar to the findings of previous studies,

 physicians in 2 groups attributed the patterns observed to a general proclivity not to change prescribing habits. The periodic shortage of nitrofurantoin exacerbated this situation because it conditioned physicians to prescribe other dugs, particularly ofloxacin. 

Physicians in 1 group articulated a general disenchantment with guidelines and with the pretense that one can describe a “usual” case to be treated in the clinical setting.

The results from the second round of discussion concerning barriers to guideline implementation appear in 

. The 3 sessions generated 7 different ideas for promulgating and implementing guidelines in the future as follows:

An idea suggested during all 3 sessions was that during a patient visit, the electronic patient record program could display the drug of choice when the diagnostic code was entered. 

Periodic, retrospective auditing of individual adherence to guidelines was suggested during all 3 sessions. Suggestions for the content of this feedback included current prescribing patterns and uropathogen resistance patterns. The participants did not believe that information regarding the costs of the different drugs would contribute to changing prescribing habits.

The technique of individual educational visits similar to those of drug representatives was suggested by the moderator and was considered to be a viable option in all 3 groups.

It was consistently suggested that small-group discussions similar to the NGT sessions themselves would be effective in conveying to physicians the rationale behind the guideline and would therefore be successful in changing prescribing practices.

One group surmised that because considerable portions of the nonadherence were due to shortages of the drug, ensuring a steady supply of nitrofurantoin would be sufficient to improve prescribing without the need for remedial interventions.

During 1 session the idea of inviting physicians to an academic lecture presented by an expert in the relevant field was suggested.

Leumit conducts a multisite continuing medical education program for primary care physicians. One group suggested that a lecture delivered in this format devoted to the subject of treating urinary tract infections would be beneficial in improving prescribing precision.

Sessions that used NGT reinforced with electronic voting technology were successful in actively involving small groups of target population physicians in the managerial decision-making process about how best to implement CPGs. This methodology enabled achievement of the objective: to facilitate presentation of population-based drug utilization data to practicing physicians and to elicit their interpretations of our findings. In view of the active and productive physician participation achieved during these sessions, the HMO studied may want to consider incorporating this methodology into its routine policy formulation processes. The combination of focus group discussion and NGT promoted less inhibited discussion than might otherwise be the case in a bureaucratic organization characterized by explicit hierarchy and chain of command.

Additionally, a major advantage suggested by our study is that although NGT was used by a group experienced in conducting focus groups, it appears that NGT also can be implemented without enlisting specially trained moderators or extensively training in-house personnel. The technique is likely to be applicable to other settings because it can be implemented by an in-house facilitator who is familiar with the data and the study objective. We therefore recommend that when feasible (as in this study), the primary investigator who evaluated the relevant prescribing data should function as the moderator.

The electronic voting technology proved highly effective for initiating group discussion. Being anonymous, it reduced reticence associated with exposing one’s opinions to peers, thereby enabling access to physician attitudes and knowledge levels. The method also allows for identification of knowledge gaps and promotion of awareness regarding misunderstandings concerning current clinical practice. 

The method described here at length has been proven to be useful for generating a quantitative study, one of the accepted roles for qualitative research. The utility of the survey designed on the basis of the focus group results has been reported elsewhere.

 Focus groups conducted with this method have been effective in generating information relevant to further study and management.

The ideas generated during the sessions about the reasons for suboptimal adherence to the guidelines as well as the suggestions for future efforts provide important information concerning these physicians’ perceptions of the organizational environment in this HMO. Although most of the ideas generated during the sessions have been well documented in the literature, these findings are important because they were obtained in a closed in-house managed care model. In attributing the nonadherence to lack of knowledge of the guidelines, these physicians are essentially identifying a flaw in the medical-managerial process, which failed to fulfill its obligation to transfer to them clinically relevant knowledge. This point is further emphasized by the persistent suggestion of feedback, indicating that these physicians perceive that dissemination of knowledge and monitoring of practice are legitimate and necessary practices in this organization. These findings are surprising because they indicate that these physicians accept managerial auditing of prescribing as a quality improvement measure and not as an invasive, paternalistic intrusion that suppresses the volitional nature of clinical practice.

Nominal group technique reinforced with electronic voting technology to stimulate conversation is a feasible technique for conducting focus groups with primary care physicians. Drug utilization data were successfully incorporated into this modality, which facilitated physician interpretation of the current prescribing patterns of the HMO studied. These findings will serve as the basis for quantitative surveys and subsequent design and evaluation of intervention strategies. This technique shows promise for becoming an integral part of the guideline formulation and dissemination process in the future and may be beneficial in other clinical settings.

Nominal group technique reinforced with electronic voting technology to stimulate conversation is a feasible technique for conducting focus groups with primary care physicians.