| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|---|
|
|
|
|||||||
|
|||||||||
Correspondence: Address correspondence to Jacqueline Zinn, PhD, Professor, Department of Risk, Insurance and Healthcare Management, Fox School of Business and Management, Temple University, 413 Ritter Annex, Philadelphia PA 19122. E-mail: Jacqueline.Zinn{at}temple.edu
 |
Abstract |
|---|
 TOP Abstract MethodsResultsDiscussionReferences |
|---|
Key Words: Nursing Home Compare • Medicare • Consumer information • Quality measures
Despite federal legislation enacted over a decade ago to improve quality, public misgivings concerning nursing home care persist, making nursing home placement a difficult decision that many patients and families would rather avoid (Harrington & Carillo, 1999; Institute of Medicine, 1986; U.S. General Accounting Office, 1999). Lack of public information about facility-specific nursing home quality has contributed to the uncertainty surrounding this decision. Making information available to the public on the performance of nursing facilities has been viewed as an important step in enabling consumers to better assess nursing facilities and to choose facilities based on information about quality (Mukamel & Spector, 2003).
In 1998, in an effort to assist consumers in evaluating the quality of nursing facilities, the Centers for Medicare and Medicaid Services (CMS) made available to consumers the Nursing Home Compare Web site (http://www.medicare.gov/NHCompare). This site provides consumers with information about Medicare- and Medicaid-certified nursing homes in the United States. It includes the quality-deficiency citations issued by state inspectors, staffing levels, and basic facility characteristics (Fermazin, Canady, Bauer, & Cooper, 2003).
In November 2002, the CMS launched the Nursing Home Quality Initiative, which introduced quality measures (QMs) to supplement the information about citations and staffing found on Nursing Home Compare. These QMs are intended to provide objective information that an individual can use as one indication of how well the nursing home manages various aspects of care provided to residents. The development of the Nursing Home Compare QMs began with the creation of the data set used to calculate the measures. The Minimum Data Set (MDS) was designed by a consortium of academic medical centers under contract with the Health Care Financing Administration (now the CMS) between 1989 and 1991 (Mor, 2004). Although the MDS was originally designed as a comprehensive clinical and functional assessment instrument to improve the care of nursing home residents, it has subsequently been applied to a number of policy functions, including case-mix reimbursement and quality monitoring (Mor).
In the early 1990s, under CMS's Nursing Home Case Mix and Quality Demonstration, the Center for Health Services Research and Analysis at the University of Wisconsin–Madison was charged with developing readily usable facility and quality indicators (QIs) based solely on computerized data from the MDS resident-assessment instrument (Zimmerman et al., 1995). Validation testing of this initial set of QIs demonstrated that the likelihood of quality problems increased with the QI scores, and that the QIs exhibited high levels of stability over two consecutive measurements, although stability varied with the characteristics of the QI (Karon, Sainfort, & Zimmerman, 1999). Subsequently, the CMS contracted with Abt Associates to review existing QIs to determine if they were suitable for public reporting. After cataloguing and evaluating 143 existing QIs, including those used by state surveyors, Abt recommended 39 risk-adjusted QIs to the CMS; 22 of these already existed, and 17 were newly developed. At about the same time, the CMS also contracted with the National Quality Forum to recommend a set of indicators to use in its planned six-state pilot and to develop a core set of indicators for national implementation in late 2002. The CMS initiated a six-state pilot in April 2002, in which facility-specific MDS-based QMs were promulgated for every Medicare- or Medicaid-certified facility in each state. In November 2002, the CMS rolled out a slightly modified set of indicators across the country, replicating the approach developed in the pilot (Mor, 2004).
The development of the QMs was not without controversy. Although there have been a number of studies pointing to the reliability and validity of the MDS data items used to construct the measures, there have been other studies that dispute these findings (Mor, 2004). On one hand, two studies of the reliability of MDS data conducted by the Office of the Inspector General (OIG) found significant discrepancies between OIG reviewer audits and facility records (OIG, 2001a, 2001b). On the other hand, a large field-reliability trial found that 85% of MDS data elements manifest adequate interrater reliability (Mor).
The team that developed the current MDS indicators reported an association between a nursing home's score on the pressure-ulcer-prevalence QM and pressure-ulcer-care processes as evidence of the validity of this QM (Morris, Moore, Jones, & Mor, 2002). However, several recent research studies, although small and focused in a single region, have called into question the accuracy of the QMs. For example, Bates-Jensen, Cadogan, and colleagues (2004) found no differences in nursing care in homes reporting high and low pressure-ulcer prevalence in 16 Southern California nursing homes. Chu, Schnelle, Cadogan, and Simmons (2004) found that 63% of residents with daily pain did not have moderate or serious pain reported on the MDS. In particular, residents with cognitive impairment were significantly less likely to have serious pain documented on the MDS. In addition, Cadogan, Schnelle, Yamamoto-Mitani, Cabrera, and Simmons (2004) found that facilities reporting higher prevalence of pain were more accurate than those reporting lower rates. Bates-Jensen, Schnelle, Alessi, Al-Samarai, and Levy-Storms (2004) observed high levels of residents' being left in bed all day, at variance with what was reported in the MDS. Schnelle, Bates-Jensen, Chu, and Simmons (2004) noted inaccurate reporting (mostly underreporting) in these and other care-process areas. The potential for detection or ascertainment bias may be due to differences in the ability of clinical staff to detect patient status and symptoms, particularly in clinical areas that are harder to define or diagnose such as pain, mood, or early-stage pressure ulcers (Mor et al., 2003). Lack of formal auditing to ensure the accuracy of MDS data (with the exception of states that use MDS data as a basis for payment) was cited as a validity issue in a U.S. General Accountability Office report (GAO, 2002a).
Although conflicting results preclude a definitive judgment regarding the accuracy of QM reporting, when used under proper research conditions, it appears that the MDS can yield reliable, valid, and sensitive measures of clinical care processes, as well as patient outcomes in important clinical dimensions. However, it appears to have some shortcomings in real-world applications (Mor, 2004; Sangl, Saliba, Gifford, & Hittle, 2005). A recent article suggests that at least some of these shortcomings could be remedied. Arling, Kane, Lewis, and Mueller (2005), after critically evaluating QIs from the perspective of theory, measurement, and application, recommended a number of strategies to improve the comprehensiveness, reliability, validity, and relevance for quality assessment.
Risk-adjustment procedures also were a key area of dispute in QM development. For example, two of the three consultants hired by the National Quality Forum recommended against the use of facility-level risk adjustment at the time of publication until the approach, along with other risk-adjustment methods, could be validated (GAO, 2002b). Consequently, the final recommendation of the GAO was that nationwide reporting be delayed until the validity of the indicators selected could be ascertained and the appropriateness of the risk-adjustment methodology was better understood.
Despite the GAO recommendation, the CMS moved forward with implementation in November 2002, citing the importance of public accountability and ongoing efforts to improve the QMs. The set of QMs included were based on the National Quality Forum recommendations and had only limited risk adjustment. The CMS advertised the introduction of the QMs in major daily newspapers to help raise awareness of the quality initiative throughout the country and promote consumer use. The Medicare Compare Web site has become one of the most popular sites on Medicare.com (National Quality Forum, 2004).
In this article, we provide an initial descriptive analysis of the first set of QMs that were published on the Medicare Compare Web site between November of 2002 and January 2004. Our objective is to offer a first look at how these reported measures vary across nursing homes by facility type and whether these measures changed over the first year of publication.
Description of Nursing Home QMs
The nursing home QMs are computed by the CMS from resident-assessment data (the MDS) that nursing homes routinely collect on all residents at specified intervals during their stay. The assessment records the residents' physical and clinical conditions and abilities, as well as life-care wishes. The methodology used in calculating the QMs is described in the Quality Measures for National Public Reporting: User's Manual (Abt Associates, 2002b). All of the measures are defined as rates of positive or negative health outcomes. Some have been adjusted to account, at least partially, for differences in case mix across nursing facilities. All QMs exclude residents who have missing MDS-assessment data elements needed for calculation.
The 10 measures chosen for public reporting on the Nursing Home Compare Web site from November 2002 to January 2004 covered two distinct resident populations: postacute residents, who enter a nursing home for a short stay following an acute-care hospitalization, and long-term-care residents, who enter a nursing home for a long period of time (typically until death) because they are no longer capable of living in the community. QMs were calculated for every facility in the United States. Some QMs are risk adjusted by using resident-level covariates, a facility admission profile, or both. In January 2004, the Nursing Home Compare Web site was updated with an enhanced set of 14 QMs (http://www.cms.hhs.gov/nhqi/quality). In this study we focus on the initial set of indicators that were published in the first five reporting periods.
For the chronic-care (long-stay) measures, calculations are based on any resident with a full or quarterly MDS survey in the target quarter beginning April 2002 through June 2002 (htpp://www.cms.hhs.gov/nhqi/quality). Because sample size affects the stability of facility scores, nursing homes with small resident numbers for a specific QM were excluded from measure calculations (Sangl et al., 2005). The facility had to have at least 30 cases within the QM category during the target time period to be included in this calculation. The six long-stay resident measures consisted of the percentage of residents in physical restraints, or with infections, pain, pressure sores, pressure sores as adjusted by the facility admission profile, or loss of ability in daily tasks:
For postacute measures, calculations were based on any resident with a 14-day MDS report in the two consecutive target quarters beginning January 2002 through June 2002. The facility had to have at least 20 cases during the target time period to be included in this calculation (http://www.cms.hhs.gov/nhqi/quality). The four postacute (short-stay) measures included the percentage of postacute residents with improved or maintained walking ability, or with pain, delirium, or delirium as adjusted by the facility admission profile:
The definition, exclusions, resident-level covariates, or facility-admission-profile adjustment and source for each of this first set of QMs are summarized in Table 1.
|
|
Methods |
|---|
|
TOPAbstractMethodsResultsDiscussionReferences |
|---|
Variable Measurement
We measured chain affiliation and hospital-based versus freestanding status as dichotomous variables (0, 1). We trichotomized occupancy rate as high if the facility was in the upper quartile of the occupancy distribution (95% + occupancy), medium if between the first and third quartile (78–94% occupancy), and low if in the bottom quartile of the occupancy distribution (less than 78% occupancy). We classified nursing home ownership status as for-profit, nonprofit, and public (governmentally operated). We trichotomized size as small (fewer than 60 beds, the bottom quartile of the size distribution), medium (between 60 and 124 beds, between the first and third quartiles) and large (more than 125 beds, in the upper quartile of the size distribution).
Statistical Analyses
We performed robust linear regression analyses to determine if the score at baseline and the pattern of change in these measures over the first five reporting periods were related to nursing home characteristics. We estimated separate models for each QM and for each facility characteristic of the following form (using chain affiliation as an example):
|
|
To interpret the results from this regression, note that the values for QMi,t for facilities that are not part of a chain are given by setting Chain = 0 and for facilities that are in a chain by setting Chain = 1. Therefore, 
is the average value of QMi,t at t = 0, that is, at baseline for nonchain facilities, and ß is the amount that reported quality changes each time period for nonchain facilities. Here 
is the difference in QMi,t at t = 0 between chain and nonchain facilities, and 
is the difference in the linear trend over time between chain and nonchain facilities. Thus, if and are not significantly different from zero, we conclude that there are no significant differences between the reported QM for chain and nonchain facilities at baseline and over time, respectively.
|
Results |
|---|
|
TOPAbstractMethodsResultsDiscussionReferences |
|---|
|
|
Tables 2 and 3 also report the number of facilities reporting on each QM. About 60% of facilities report the short-term QMs and 80% report the long-term QMs. QMs are not reported when a facility did not have at least 30 residents meeting the criteria for calculating the specific QM. Thus, small facilities or facilities that do not care for specific types of residents (e.g., short-term postacute patients) are excluded from the report card and from our analysis.
Trends in QMs Over Time
Although the statistical analysis identified significant trends at the 0.01 level for all but one of the QMs, an inspection of Figure 1 shows that, for some QMs, no real trend is apparent and the overall change between the QM at the first publication and the QM at the fifth are minimal. Pain (both long term and short term), physical restraints, and delirium (with and without risk adjustment) exhibit a clear downward trend, with differences between the first QM reporting period and the fifth ranging from 12.7% to 46.0%. Thus, those measures that show a clear trend are all declining and the decline seems substantial, given the relatively short period (15 months) over which they are measured. The other measures (pressure sores, infections, walking ability, and loss in ADLs) first decline and then increase, and the overall change in the QM does not exceed 6%. Therefore, we believe it is premature to conclude that there is a trend in these latter measures.
|
|
|
Discussion |
|---|
|
TOPAbstractMethodsResultsDiscussionReferences |
|---|
Our findings that QMs differ at baseline by organizational characteristics are consistent with prior research showing that facility attributes are associated with quality of care (Greene & Monahan, 1981; Spector et al., 1998; Spector & Takada, 1991; Zinn, Aaronson, & Rosko, 1993). In this study, for-profit and chain-affiliated facilities appear to do better on QMs (pain and delirium) for short-stay, postacute residents. Small, independent, nonprofit, high-occupancy facilities had better reported performance on QMs for long-stay residents. This suggests that some types of facilities may specialize in different aspects of nursing home care.
The trend toward improvement in five of the QMs suggests that the Medicare Compare report card, and possibly other aspects of the CMS Nursing Home Quality Initiative, may have the expected impact of improving nursing home care. However, the fact that only some measures exhibit a distinct trend raises the question of what distinguishes these measures from those that do not show a trend. Because there are a number of alternative explanations that could account for the observed positive trends on some QMs, we can only speculate on this issue. One possibility is that nursing homes that are operating with limited resources target those QMs with the highest rates, indicating the areas requiring the most attention. Indeed, the prevalence of short-term resident pain, one of the measures showing a distinct downward trend, had one of the highest prevalence rates, at over 25%. In contrast, the walking ability measure for short-term residents, with a prevalence of about 30%, did not exhibit any improvement. The lack of improvement in walking and in pressure sores may be because it is more difficult for nursing homes to affect these outcomes, especially within the short run, that is, the 15 months for which we have examined the trends. This suggests that the lack of a distinct trend toward improvement may not be due to lack of effort but rather to other factors, such as ability to affect changes in the short run. Finally, as previously noted, at least one study found no differences in nursing care in homes reporting high and low pressure ulcer prevalence (Bates-Jensen, Cadogan, et al., 2004). This suggests that the level of pressure ulcer care provided in some homes may not be good enough to make a difference.
It is also possible that some of the observed improvements reflect changes in data-reporting practices rather than actual improvements in care. The potential for detection or ascertainment bias may be due to differences in the ability of clinical staff to detect patient status and symptoms, particularly in clinical areas that are harder to define or diagnose such as pain, mood, or early-stage pressure ulcers (Mor et al., 2003). Pain in particular may be subject to reporting bias. A recent study found that pain was underassessed in facilities lacking pain specialists, suggesting that the documentation of low levels of pain may actually reflect incomplete assessment (Wu, Miller, Lapane, & Gozalo, 2003). Standards for the assessment of pain may vary as a function of differences in training in addition to underlying biological differences (Mor et al., 2003). Finally, similar to "DRG creep" (i.e., coding patients in the clinical category that generates the most revenue) observed in hospitals after the implementation of prospective payment, positive changes in pain and delirium may reflect perverse incentives to upcode in areas that are more difficult to audit.
The current set of QMs published on the Nursing Home Compare Web site are different from the original set we report on in several respects: new measures have been added, some from the original set have been eliminated, and, for some, risk adjustment was modified. The publication of the QMs was intended to be an evolutionary process. In commenting on the recommendations of the GAO report, the CMS reiterated its commitment to continuously improve the indicators as more evidence of validity and reliability becomes available. It is probably a good thing to rotate the published QMs to avoid "teaching to the test" incentives, that is, incentives to focus only on improving the published QMs to the possible detriment of other areas of patient care. The downside of changing QMs is that we will not be able to follow trends for the eliminated QMs in the future.
The limited association between organizational characteristics and improvement trends raises concerns that the gaps in outcomes by facility type at baseline may persist. An obvious objective for report cards is to provide incentives for all facilities to improve quality, and in particular for those who lag behind. In a few instances we indeed observe that gaps are closing. For example, the differences in pain among short-stay residents between chain and nonchain facilities decreased over the period. However, most of the gaps did not close. Researchers should consider several caveats when interpreting these findings. It is not clear whether the changes observed over time are attributable to quality improvement or to changes in data reporting over time. For example, is the decline in the reported prevalence of pain attributable to better treatment of pain or to less diligence by nursing homes in documenting pain? In addition, any trend in quality improvement may have preceded, rather than have been precipitated by, QM publication. One would have to examine values of these measures for periods prior to publication to determine if Nursing Home Compare reporting is influencing observed trends. In addition, because the QMs are outcome measures, their validity depends on whether they have been appropriately adjusted for differences in case mix. If the measures are not appropriately adjusted, then nursing homes could improve their QM score by changing their admission policies toward a less frail population that may be at lower risks for these adverse outcomes. In addition, if facilities do not measure the QMs comparably, measurement bias may influence the findings. Thus, researchers must pursue further studies to address these questions in order to establish the effectiveness of the reported measures in quality improvement. Nevertheless, report cards clearly do serve an important function in focusing attention on where quality improvement is needed for facilities, regulators, and consumers.
Finally, it is important to understand how facilities that have successfully made improvements were able to make these strides. What were the most important changes in the quality-assurance process that improved QM scores? Were the efforts of the quality improvement organizations instrumental in effecting improvements? Were facilities that worked with the quality improvement organizations to improve specific QMs successful? How can we change quality-assurance programs for ADL, walking, and pressure-sores QMs where no declining trends have occurred? The answers to these questions suggest the need for in-depth evaluation studies that go beyond whether nursing home quality has improved under the Nursing Home Quality Initiative to provide insights into how any gains were achieved and how improvements could be made in the future.
|
Footnotes |
|---|
2 Agency for Healthcare Research and Quality, Rockville, MD.
3 Center for Health Policy Research, University of California, Irvine.
Decision Editor: Linda S. Noelker, PhD
Received for publication November 12, 2004. Accepted for publication June 28, 2005.
|
References |
|---|
|
TOPAbstractMethodsResultsDiscussionReferences |
|---|
This article has been cited by other articles:
|
|
 |
|
  A. S. Groenewoud, MSc, N. J. A. van Exel, MSc, M. Berg, PhD, and R. Huijsman, PhD Building Quality Report Cards for Geriatric Care in The Netherlands: Using Concept Mapping to Identify the Appropriate "Building Blocks" From the Consumer's Perspective Gerontologist, February 1, 2008; 48(1): 79 - 92. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 D. B. Mukamel, W. D. Spector, J. S. Zinn, L. Huang, D. L. Weimer, and A. Dozier Nursing Homes' Response to the Nursing Home Compare Report Card J. Gerontol. B. Psychol. Sci. Soc. Sci., July 1, 2007; 62(4): S218 - S225. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 C. M. Hughes and K. L. Lapane Nurses' and nursing assistants' perceptions of patient safety culture in nursing homes Int. J. Qual. Health Care, August 1, 2006; 18(4): 281 - 286. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|---|
| All GSA journals | Journals of Gerontology Series A: Biological Sciences and Medical Sciences | Journals of Gerontology Series B: Psychological Sciences and Social Sciences |
