| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|---|
|
|
|
|||||||
|
|||||||||
Correspondence: Address correspondence to John F. Schnelle, PhD, University of California–Los Angeles, Department of Medicine, Division of Geriatrics, 7150 Tampa Avenue, Reseda, CA 91335. E-mail: jschnell{at}ucla.edu
 |
Abstract |
|---|
 TOP Abstract What Are the Key...Controlling Observer...Developing a Standardized...Identifying Care Domains for...Determining the Costs and...SummaryReferences |
|---|
Key Words: Observations • Nursing home quality assessment • Nursing home survey
Nursing homes across the nation are compared on quality indicators with the intent of identifying homes with outcomes that might reflect poor care quality. For example, homes are compared on the percent of residents with weight loss, and homes in the upper percentile of weight loss are "flagged" as having a potential quality problem (Nursing Home Compare, 2002). One intent of this system is to motivate nursing home staff to assess the quality of their care processes that might be related to poor quality-indicator scores. All homes are required to conduct such quality-assurance activities, even though the procedures used to do so are not well defined.
In addition, federal and state teams use these indicators to structure how they conduct surveys to determine if there are quality problems in facilities. For example, medical records are reviewed, residents and staff are interviewed, and observations are conducted by survey staff to determine if residents are being adequately assessed and treated for nutritional problems if a facility is flagged as having a high prevalence of weight loss.
It is obvious that effective care management minimally requires reasonably accurate information about the care delivered by nursing home staff. Unfortunately, there is evidence that the major method used by nursing homes to report care-process delivery is not useful for management and is even a barrier to improvement (Schnelle, Bates-Jensen, Chu, & Simmons, 2004). The medical record is the major source of information about care, and providers typically make handwritten entries at times remote from actual care delivery. For example, most information about daily care is provided by nurse aides, who typically check boxes at the end of the shift, to document that such critical care as feeding and toileting assistance was provided. It has been reported that the medical record documents better care than that actually provided, which should not be surprising given the time lapse between care delivery and documentation and incentives to document care that is consistent with federal regulations.
Despite serious limitations, there are good reasons to use staff reports from the medical record as a tool for management and quality assurance. Most notably, the act of self-recording care may increase the probability that the care is provided, and members of the staff, as the direct care providers, are in the best position to record it. However, it is also clear that system-level changes are required to ensure that the medical-record data are accurate, or at least not misleading, as a measure of care quality.
One system-level change that has been suggested is to use electronic systems that allow staff to document care at the point of service delivery (Wood, 2004). Such computerized systems reduce the need for memory of care, and time-stamp technologies ensure that care is documented during intervals when it is reasonable to expect it to occur (e.g., repositioning a resident at 2-hr intervals). However, even these electronic systems will not reduce the incentives for inaccurate documentation. One incentive is that nursing homes may be understaffed, and high staffing ratios create pressure to overreport care tasks to maintain the appearance of adequate care provision. Moreover, the medical record is used to reduce legal liability and ensure paper compliance with federal regulations, which only increases the pressures for inaccurate documentation.
In addition to electronic medical-record systems, we have argued that there are two obvious solutions to improving medical-record accuracy that should be implemented simultaneously. One solution is to provide staff with realistic workloads, and the second is to audit staff reports with independent measures (Schnelle et al, 2004). Here we focus on the auditing solution.
Direct observations of care delivery can provide the independent information needed to improve medical-record accuracy and gain important quality data that are absent from the medical record. For example, data about how well staff interact with residents are important to quality of life, and the medical record does not provide this type of information.
Federal survey teams and nursing home quality-assurance personnel (e.g., supervisory nurses, an administrator) are in the position to collect these observational data, and it is their responsibility to do so. Unfortunately, the observation procedures described in the nursing home quality-assessment literature are poorly defined and show little awareness of the measurement principles that must guide the collection of accurate and reliable observational data. For example, federal survey protocols require that observations be conducted to evaluate nursing home quality in most care areas (CMS State Operational Manual, 2004). Unfortunately, little specificity is provided about how to conduct these observations. The nonspecific instructions for assessing quality of life in the federal survey manual are typical of the observation procedures recommended for most care areas (Procedure 483.15a): "Throughout the survey, observe: Do staff show respect for residents? When staff interact with a resident, do staff pay attention to the resident as an individual?" The content of these instructions are clearly related to quality-of-life constructs, but it is almost certain that individual survey staff will use different methods to observe and different criteria to judge quality with such nonspecific instructions.
The most specifically defined example of an observational protocol to assess nursing home care quality was described by Rantz and colleagues (2000). The Rantz study described an observational protocol in which observers walk through the nursing home for 20 to 30 min, recording categorical data in 42 different areas. For example, observers recorded if they observed staff providing residents with assistance with food or fluid intake, using a scale from rarely seen to very often. This approach has the advantage of simplicity but does not allow the categorical quality data to be linked to a specific resident or provider (e.g., licensed nurses vs nurse aides). The absence of a link between the quality data and the resident or staff member precludes an accuracy check of corresponding medical-record data and the targeting of interventions to residents who might be receiving substandard care. However, the Rantz approach does provide reliable global quality measures.
In sum, observational protocols appear to be an important component of many nursing home quality-assessment efforts, but we are aware of no nursing home observational system that provides the specific information needed to audit medical-record accuracy or measure care at the individual resident level. Our purpose in this article is to describe (a) characteristics of care-process data that are useful for nursing home staff management and quality improvement, because these characteristics influence the design and evaluation of observational protocols; and (b) measurement principles that must guide the collection of observational data about care delivery to produce reliable and accurate conclusions about nursing home care quality.
Our intent is not to describe the principles that guide observational measurement in scientific research, because these principles are described in numerous textbooks. Instead, the intent is to describe how these principles can be applied to the methodological problems posed by nursing home quality-assessment activities so that the resources currently spent on observational assessment can be more effectively utilized for quality assessment and improvement.
|
What Are the Key Components of a Quality-Improvement Information System? |
|---|
|
TOPAbstractWhat Are the Key...Controlling Observer...Developing a Standardized...Identifying Care Domains for...Determining the Costs and...SummaryReferences |
|---|
Specificity refers to the detail about the care processes provided to residents. For example, it lacks specificity for nursing home staff to document that care was provided "as needed" (e.g., feeding assistance) or to provide staff feedback that care quality is "poor." Instead, specific information should be provided about the care processes that contribute to a quality conclusion (e.g., only 1 of 8 residents rated by staff as requiring assistance to eat received more than 5 min of assistance during dinner). The specificity of the care-process information allows two critical CQI events to occur. First, the accuracy of medical-record documentation related to the care processes can be determined (e.g., what was recorded in the medical record about feeding-assistance care provision?) What is equally if not more important is that the specificity of the care-process information allows nursing home staff to improve care quality in a targeted manner (e.g., the specific residents not receiving assistance during dinner are identified, which permits training interventions to be targeted to staff responsible for the care). In addition to the need for specific information about the occurrence of care, it is also important to have specific information about other quality issues. In our previous work, we have reported that observations during mealtimes show that few residents receive social interaction (e.g., "Hello. How are you today?") or even verbal cueing (e.g., "Here are some carrots.") prior to being physically fed. These observational measures assess how well staff interact with residents, which has been related to residents' quality of life and oral intake (Lange-Alberts & Shott, 1994). The medical record currently provides no data (much less specific or accurate data) about this aspect of care.
Timeliness of care-process information is necessary to identify factors that influence the ability of staff to provide care (e.g., are nurse aides short staffed during dinner?), which requires the availability and organization of information soon after care provision. Minimally, organized and interpretable information should be available during the same shift that the care is provided so that care provision can be linked with a specific provider or to the work conditions that might influence care. One of the major advantages of electronic systems, which permit point-of-service documentation, is that timely information can be made available to supervisors as the documentation occurs.
Accurate information about nursing home staff care provision is critical for staff management and CQI efforts to be successful. Medical-record accuracy issues have been reported in many health care settings, both in terms of errors of omission (not documenting care provided) and errors of commission (documenting care that is not provided; Aaronson & Burman, 1994; Ehrenberg & Ehnfors, 2001). Much of the error in nursing home medical-record documentation related to care-process delivery falls into the commission category (Schnelle et al., 2004). In general, medical-record documentation indicates a higher frequency of care than is indicated by independent measures. Thus, reliance on medical-record documentation alone may lead to erroneous conclusions about care-process delivery and related care quality.
Information collected with direct observational protocols can meet the specificity, timeliness, and accuracy requirements necessary for effective staff management and CQI efforts. In addition, direct observations can serve as an audit to ensure the accuracy of information recorded by nursing home staff and to provide data about quality of life and quality of care that are absent from the medical record. However, direct observation information can only be used for these important CQI purposes if the following issues are addressed:
|
Controlling Observer Subjectivity and Accurately Measuring Care: A Review of Measurement Principles |
|---|
|
TOPAbstractWhat Are the Key...Controlling Observer...Developing a Standardized...Identifying Care Domains for...Determining the Costs and...SummaryReferences |
|---|
Interobserver Agreement
Interobserver agreement statistics, such as kappa, are often reported in both nursing home quality assessment and clinical research to support the argument that the behavioral measures are defined with sufficient specificity that different observers can agree on what is being measured (Cohen, 1960). It is important to note that high interobserver agreement is necessary to achieve accurate measures of usual care practices, but it is not sufficient, as will be discussed in the following sections of this article. In addition, if extended periods of training or clinical expertise are required to achieve interobserver agreement, it is likely that the rules of observation have to be more specific. Ideally, acceptable levels of interobserver agreement should be achieved within 1 day of training, given adequately defined behaviors, and naive observers can be taught to agree. Periodic repeat agreement checks can then be conducted to prevent observer drift over time.
Observer drift in which observers gradually begin to modify the behavioral codes applied to the observed behaviors but still maintain interobserver agreement is a common phenomenon (Johnson & Bolstad, 1973). Observer drift is particularly problematic when observations are routinely conducted over long time periods by the same observers, as in the nursing home survey and quality-assurance process. Independent, and preferably unobtrusive, random observations by observers who do not routinely collect the data are necessary to prevent drift. Specific technical advice about how to collect interobserver agreement data is described in introductory behavioral analysis textbooks (Bailey & Burch, 2002).
Observational Schedule: Continuous Versus Time-Sampled Procedures
The schedule, or timing, of observations must be clearly defined to control observer subjectivity. A continuous observation schedule offers the most defensible information because it eliminates the possibility of a behavioral occurrence missed by the observer. However, a continuous observation schedule is not feasible to conduct for a large number of residents or over long time periods unless there are available technologies that can replace human observers. For example, a recent study showed that computerized movement monitors could be used to continuously measure the frequency of nursing home resident repositioning activities (Bates-Jensen et al, 2003). A continuous observation schedule or a technology that allows continuous monitoring is particularly important for behaviors that occur infrequently or require brief periods of time, such as repositioning care activities in the nursing home setting.
Time-sampled observational schedules can be used when continuous monitoring is not practical or necessary. Time-sampled observations involve observing residents for a brief interval (e.g., 1 min/hr). The parameters of the time-sampled observation schedule should be set based on information that the sampling intervals produce data similar to that collected with more frequent observational schedules (e.g., 1 min/15 min or continuous monitoring). For example, we have determined that a time-sampled observational schedule of 1 min/hr produces data similar to that produced by a time-sampled observational schedule of 1 min/15 min for the following nursing home resident behaviors: whether the resident is in or out of bed, asleep or awake, sitting on a pressure relief surface, or wearing a physical restraint device. Because evidence of care occurrence is visible and relatively stable in these domains, an infrequent schedule permits accurate observation. Care behaviors that do not leave visible evidence of implementation (e.g., walking assistance) require a more frequent observation schedule to produce accurate data.
There is also the related issue of how often observations must be conducted to assess care accurately when the care must occur over long time periods and multiple shifts. For example, incontinence care should occur throughout the 24-hr period. The number of different time periods that must to be observed to accurately measure incontinence care quality is unknown.
Reactivity of Behavioral Observations
A potential limitation of behavioral observations is that the observations may be "reactive," which means that the targeted behavior changes as a result of observation. A common recommendation made to reduce reactivity is to conduct observations in an unobtrusive manner. Unfortunately, there are sparse data about the severity of reactivity and little guidance about how to reduce obtrusiveness. There are sparse data because the best way to estimate reactivity effects is to simultaneously use two observational protocols that differ on the level of obtrusiveness. For example, simultaneous observations would be conducted of the same behavior for the same resident by one visible or obtrusive observer, and one invisible or unobtrusive observer. In practice, it is difficult for the observer to be completely unobtrusive. However, there are research data from two studies that approximate this design wherein the data of primary research staff observers were compared during time periods when they knew other research staff was present versus not present (Romanczyk, Kent, Diament, & O'Leary, 1973). There was a 10% to 20% difference in the data recorded between the two observation periods. These results suggest that the primary observers "reacted" to the presence of other observers.
The 10% to 20% reactivity estimate for direct observational methods does not appear to be unusually high when it is compared with other methods of measurement that depend on the self-report of caregivers or patients. For example, the results of drug trials for behavioral disturbance often report 40% to 60% placebo effects, with caregivers or patients reporting improvement in the absence of treatment (O'Brian, 1999). This placebo effect could reflect changes in behavior that are due, at least partially, to the task of self-recording behaviors as well as other factors that are encompassed in the so-called Hawthorne effect phenomenon. Finally, it should be noted that many observational studies of nursing home quality have reported significant care problems. If direct observations of care delivery were reactive, one would expect better conclusions about quality in these studies (Kayser-Jones & Schell, 1997; Simmons, Babineau, Garcia, & Schnelle, 2002).
|
Developing a Standardized Observational System for Use in Nursing Home and Survey Practice |
|---|
|
TOPAbstractWhat Are the Key...Controlling Observer...Developing a Standardized...Identifying Care Domains for...Determining the Costs and...SummaryReferences |
|---|
|
Identifying Care Domains for Observation |
|---|
|
TOPAbstractWhat Are the Key...Controlling Observer...Developing a Standardized...Identifying Care Domains for...Determining the Costs and...SummaryReferences |
|---|
We propose four criteria to select nursing home care domains for observation: evidence that (a) care is poor in an area recognized as important for quality of life or clinical functioning; (b) there is consensus about what care processes define higher quality in the domain; (c) medical-record documentation about the care processes is inaccurate, incomplete (lacks specificity), or absent; and, (d) human observers can assess the (non)occurrence of care process(es) that are recognized as reflecting care consistent with federal regulations or best practice guidelines. A review of the literature identifies observational studies that measure care processes that meet these criteria in both clinical and quality-of-life domains (Schnelle et al., 2004; Simmons et al., 2002; Sloane et al., 2004). However, the feasibility of adapting these observational protocols developed for research purposes to the demands created by the survey or internal quality-improvement process in nursing homes has not been demonstrated. The most important feasibility issue relates to costs.
|
Determining the Costs and Rules of Observation |
|---|
|
TOPAbstractWhat Are the Key...Controlling Observer...Developing a Standardized...Identifying Care Domains for...Determining the Costs and...SummaryReferences |
|---|
Accuracy Evaluation: Frequency of Observation
To determine the schedule of observations necessary to produce accurate information, observations should be conducted on a continuous schedule (e.g., a small number of residents are continuously monitored) for the purpose of comparing the data to that collected on a less labor-intensive time sampling schedule (e.g., every 15 min or hour). The key feasibility question is this: What is the least labor-intensive schedule that produces accurate information? In extreme cases, it might be demonstrated that continuous observations of a relatively few residents or staff are needed to produce accurate information. On the basis of our clinical experience, we hypothesize that toileting assistance and repositioning for pressure relief are examples of care processes that might require continuous observations of two or three residents over 6- to 8-hr daytime periods to produce accurate data. However, this hypothesis could be tested by comparing the frequency of these care activities measured on a continuous schedule with the frequency measured on a time-sampled schedule (e.g., resident observed for 1 min/15 min). In addition, repositioning and toileting assistance frequency continuously measured over a few hours (around mealtimes or when the resident is assisted in or out of bed) could be compared with information collected over 8 consecutive daytime hours to determine how many hours and when key time periods of observation are necessary to measure quality accurately in these areas. For example, the frequency of toileting assistance measured at key time periods might be highly correlated with frequencies measured over the entire daytime period, which would make continuous observations feasible and worthwhile to conduct during the key time periods only.
Accuracy Evaluation: Reactivity
Reactivity also contributes to the cost of conducting observations, because observational protocols that produce reactivity will require longer periods of observation to yield accurate information about the care process. The key to reducing reactivity is to minimize the obtrusiveness of the observers (see earlier discussion concerning the reactivity of behavioral observations) or to conduct observations over long time periods to diminish the uniqueness of the observations. One way to minimize obtrusiveness is to focus the observations on residents instead of staff. For example, a high level of reactivity might occur if an observer follows a staff member as he or she provides care to assess how the staff member interacts with residents. Alternatively, less reactivity should occur if the observer rotated through the facility to observe the quality of the interaction between residents and any staff member as it naturally occurred during care provision. Our experience suggests that nonreactive, time-sampled observational schedules can be developed that permit multiple care processes to be efficiently observed with 10 to 12 residents by one observer. For example, preliminary data suggest that the amount of time residents spend in bed and the frequency and quality of their interaction with staff can be accurately assessed with a time-sampled observational protocol.
Specifically, observations of care could be conducted over several weeks to determine the amount of time necessary to provide stable data. On the basis of preliminary data in a number of daily care areas, we hypothesize that there will not be trends in the observational data to indicate reactivity. Once stable information is evident, staff could reduce the frequency and hence the cost of the observations (e.g., from daily to weekly) to determine if the frequency of observations influences the results.
Translating Observational Data Into Quality Conclusions
Consistent or reliable conclusions about quality require that observers use the same rules to observe, as previously discussed, and to translate these observations into quality statements. For example, a measure of how much feeding assistance a resident receives, even if reliably observed by two observers, is only useful if it can be translated into the same quality conclusion by different observers. If observers use different criteria to determine how much assistance defines good care, then there will be inconsistencies in the quality conclusions. As an illustration of this point, it was determined in our previous work that if a resident required assistance to eat, then she or he should receive more than 5 min of assistance in order for this care to be judged as adequate. This rule was based on evidence that a dependent resident required 20 to 30 min of assistance to eat at optimal levels (Simmons et al., 2002). Similar rules, which operationalize the relationship of observations to quality conclusions, are necessary in all areas to prevent observers from inserting their personal bias into the quality-assessment process.
|
Summary |
|---|
|
TOPAbstractWhat Are the Key...Controlling Observer...Developing a Standardized...Identifying Care Domains for...Determining the Costs and...SummaryReferences |
|---|
These independent data could be collected from two sources (interviews with residents and direct observations of care), and a strong case can be made that both sources of information are of value. For example, we have reported that between 25% and 50% of residents can answer specific questions about care delivery, such as, "Did someone help you to the bathroom this morning?" (Simmons & Schnelle, 2001). A greater proportion can answer more subjective questions about quality of life, daily care preferences, mood, or pain status (Chu, Schnelle, Cadogan & Simmons, 2004; Kane et al., 2003). However, in this article we emphasized the use of observational data. We have made the case that quality-assurance personnel, including federal and state survey staff, do not have access to standardized protocols that facilitate the collection of defensible observational data. Despite language about the importance of observations in assessment efforts, observational protocols are treated with a surprising informality and under the apparent assumption that quality-assurance staff will intuitively know how to conduct accurate observations. Inconsistencies in the survey process are likely due to the lack of standardized observational protocols and, instead, a reliance on subjective expert judgment (Centers for Medicare & Medicaid Services AGG/Research Contracts and Grants Division, 2003).
The accuracy, reliability, and feasibility of observational quality-assessment protocols should be evaluated and described in a manner that permits their use in nursing home care practice. Survey and nursing home staff should be able to use at least some of the same protocols to assess care quality and medical-record accuracy within existing staff resources. If both nursing home and survey staff used the same standardized observational protocols, then nursing home resistance to the survey process would be reduced, and staff would be more motivated to collect observational data about care delivery for quality improvement on a more frequent basis. The use of standardized observational protocols represents a necessary first step toward both genuine CQI efforts and a more defensible survey process.
|
Footnotes |
|---|
2 Borun Center for Gerontological Research, Los Angeles Jewish Home for the Aging; Department of Medicine, University of California–Los Angeles.
Decision Editor: Linda S. Noelker, PhD
Received for publication December 15, 2004. Accepted for publication April 22, 2005.
|
References |
|---|
|
TOPAbstractWhat Are the Key...Controlling Observer...Developing a Standardized...Identifying Care Domains for...Determining the Costs and...SummaryReferences |
|---|
This article has been cited by other articles:
|
|
 |
|
  E. O. Siegel, H. M. Young, P. H. Mitchell, and S. E. Shannon Nurse Preparation and Organizational Support for Supervision of Unlicensed Assistive Personnel in Nursing Homes: A Qualitative Exploration Gerontologist, August 1, 2008; 48(4): 453 - 463. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
C. D. Phillips, M. Chen, and M. Sherman To What Degree Does Provider Performance Affect a Quality Indicator? The Case of Nursing Homes and ADL Change Gerontologist, June 1, 2008; 48(3): 330 - 337. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 E. A. Miller and V. Mor Balancing Regulatory Controls and Incentives: Toward Smarter and More Transparent Oversight in Long-Term Care Journal of Health Politics Policy and Law, January 1, 2008; 33(2): 249 - 279. [Abstract] [PDF]
|
|
|
|
|
|
L. Levy-Storms, J. F. Schnelle, and S. F. Simmons What Do Family Members Notice Following an Intervention to Improve Mobility and Incontinence Care for Nursing Home Residents? An Analysis of Open-Ended Comments Gerontologist, February 1, 2007; 47(1): 14 - 20. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
R. H. Lee, B. J. Gajewski, and S. Thompson Reliability of the Nursing Home Survey Process: A Simultaneous Survey Approach Gerontologist, December 1, 2006; 46(6): 772 - 779. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|---|
| All GSA journals | Journals of Gerontology Series A: Biological Sciences and Medical Sciences | Journals of Gerontology Series B: Psychological Sciences and Social Sciences |
