Summary of the Working Session Meeting
on Defining Health Statistics and
Developing Criteria for Evaluating Health Statistics

March 29-30, 1999

Hosts: Daniel J. Friedman, Ph.D.,
Member, NCVHS and Assistant Commissioner, Bureau of Health Statistics, Research and Evaluation, Massachusetts Department of Public Health
and Edward J. Sondik, Ph.D.,
Director, National Center for Health Statistics

Part of the ongoing process for Developing the 21st Century Vision for Health Statistics --a joint project of the National Committee on Vital and Health Statistics, the National Center for Health Statistics, and the HHS Data Council

DRAFT July 12, 1999

Table of Contents

A. Background

1. Commissioning papers
2. Convening experts in discussion groups and working sessions
3. Holding a workshop
4. Soliciting broader input through a consultative process

B. Goals

C. Defining Health Statistics

1. Content
2. Nature
3. Purpose

D. Roles for Health Statistics

1. Supporting policy
2. Setting standards and norms
3. Adding value to data

E. Health Statistics as a Field

F. Health Statistics as a System

G. Evaluation Criteria for Health Statistics

Appendix A: Agenda for the Working Session

Appendix B: Participant List for the Working Session

A. BACKGROUND

Dan Friedman began the meeting by describing the joint National Committee on Vital and Health Statistics (NCVHS), NCHS, and HHS Data Council process to examine and create a vision for health statistics in the 21st century. In 1997, Ed Sondik approached NCVHS and asked them to consider what health statistics should be in 10-20 years. As a result of this request, NCVHS, NCHS, and the DHHS Data Council have developed a process to create this vision.

Dan pointed out that the process has three goals:

First, to develop a vision of the health statistics system for the 21st century. The vision will reflect all manifestations of health and health care delivery; encompass population health, transactions between the population and the health care delivery system, and the health care delivery system; and address the relationship and potential synergy between public and private health data sets, and national, state, and locally maintained data.

Second, to describe and define the disciplines, components, resources, and other elements that are needed to implement this vision.

Third, to set forth a clear set of criteria and a process for evaluating the health statistics system and its individual components, now and in the 21st century.

Although health statistics is a commonly used term, it has not been systematically defined in the USA. This contrasts with surveillance, which has a small but helpful literature that defines its purpose and scope. The definition of health statistics should help us to envision its purposes and roles and guide its evaluation.

The NCVHS, NCHS and the Data Council process for achieving these three goals includes the following components:

1. Commissioning papers -

• Selected National Health Data Systems: A Background Paper for Planning the Future of Health Statistics in the United States, by Jennifer Zelmer et al, Canadian Institute for Health Information. This paper will provide a comparative overview of how different countries organize their health statistics systems.
• From National Health Statistics to Health Information Systems: A Model for the 21st Century, by Charlyn Black et al, University of Manitoba. This paper will discuss key assumptions and critical components of a health information system; outline the importance of being able to make links across key areas and to examine various types and levels of investment in medical care for different populations; describe the Manitoba experience with developing, implementing and expanding the POPULIS System, and discuss possibilities of developing these types of health information systems in the future.
• Health Statistics Needs for Children, by Lorraine Klerman, University of Alabama at Birmingham. This paper will review the history of the State-Children’s Health Insurance Program (S-CHIP) to determine what role health statistics data played in indicating a need for S-CHIP and the development of its provisions; describe the health statistics data that policy makers at the state and federal levels believe will be essential to demonstrate the impact of S-CHIP; and explore the need for additional health statistics data about the health status of children and the services that children use, as perceived by leaders in the field.
• Numbers We Need: Health Statistics and Health Policy, by Richard Kronick, University of California at San Diego. This paper will focus on the health statistics data needed for improving health policy, emphasizing: the financing and delivery of care to the uninsured, the extent to which public policy should encourage the growth of managed care, and mental health parity.
• Characteristics of a Robust and Useful Health Statistics System, by Daniel Melnick, Consultant. This paper will identify several subsystems of the national health statistics system that are generally regarded by their users as having responded successfully to diverse and changing data needs, such as the mortality statistics system and the National Health Interview Survey. The properties of those successful systems will be analyzed, and a general model for successful systems will be derived.

These papers should help to identify needs for improvements in health statistics, and help us to achieve common understandings of the scope and effects of health statistics.

2. Convening experts in discussion group and working sessions such as this meeting - These groups will help to define health statistics and develop evaluation criteria, as well as identify issues likely to emerge in health and health care technology during the next 10 to 20 years.

3. Holding a workshop under the auspices of the National Academy of Sciences’ Committee on National Statistics (CNStat) - The workshop will build upon the discussion and working groups, the commissioned papers, and input of additional national experts, and will present an overview of issues in health statistics likely to emerge in the next 10 to 20 years.

4. Soliciting broader input through a consultative process – NCVHS, NCHS and the Data Council recognize the need to establish a broad public consultative process for obtaining input into the 21st century health statistics vision from practitioners, users, advocates, health professionals, and consumers. Although details of the consultative process are still being defined, it will certainly entail public hearings, town hall meetings and forums at professional association meetings, state visits, and feedback on the NCVHS web site.

Ed Sondik added that the target that we have been aiming for in health statistics is not well defined. Health statistics have evolved to meet many needs, but with minimal vision to the evolution. Rather, health statistics tend to react to issues as they emerge, and trail events rather than anticipate them, making it difficult to provide the information that would be most helpful.

He noted that the joint NCVHS\NCHS\Data Council process involves discussions with a variety of people and organizations. This process is not really about NCHS specifically, but about health statistics broadly. Once the vision is formulated, and the CNStat workshop completed, the current effort could focus on the roles of different agencies that produce health statistics such as NCHS, other components of the CDC, AHCPR, and the NIH.

B. GOALS

The goals of the March 29-30th meeting were to:

• define health statistics,
• delineate the roles of health statistics,
• discuss the field and system of health statistics, and
• develop criteria for evaluating health statistics.

The underlying aim of this meeting was to elicit a range of opinions related to these four objectives, rather than to achieve consensus. As such, some seemingly incompatible thoughts may be represented in the summary that follows.

C. DEFINING HEALTH STATISTICS

Considerable discussion focused on whether health statistics has a clear definition and boundary. The term “health statistics” is used interchangeably to refer to numerical data concerning health, as well as the area of interest or academic specialization (although there was discussion as to whether there exists a “field” of health statistics as well.) As such the use of “health statistics” throughout this document is somewhat and ambiguous and often grammatically challenging.

Health statistics was characterized by some as a body of potential evidence that lies between data and information. Health statistics becomes useful when it forms the basis for health program knowledge. Its content, nature, and purpose can define health statistics.

1. Content - While statistics may be compiled on any type of entity, health statistics are (a) population-based, and (b) focus on distributions of conditions, events, or outcomes related to the population. Health statistics relate to the association that those distributions have with the health and wellness of the human population, and extend beyond the medical model to a more expansive model of health and wellness. The distributions of interest may include:

• people and their
  • health behaviors
  • health status
  • work life
  • other individual, familial, economic, social and environmental factors that affect health and wellness
  • utilization of, access to, costs of, and outcomes of health services;
• providers, both individuals and institutions, and their
  • organization
  • characteristics
  • quality of services
  • resources; and
• financing mechanisms, including health insurance plans and their
  • organization
  • characteristics
  • impact on the provision of health care.

2. Nature - While the characteristics and qualities of health statistics are not unique to health statistics, they are an integral part of its definition. Among the attributes mentioned by the group:

• Health statistics consist of quantitative data.
• Health statistics encompass ongoing data, rather than individual point-in-time studies.
• Health statistics’ underlying tenets are (a) public disclosure of nonidentifiable data, (b) public access to nonidentifiable data, and (c) availability of data collection and analysis methods for peer and public review. The public must have access to the data and the methods used so that the credibility of analyses can be tested, and data can be used for multiple purposes. These three tenets are required for all health statistics, whether publicly or privately generated.

Other possible characteristics are: representative, generalizable, descriptive and predictive.

3. Purpose - Health statistics is defined by its uses. Health statistics is a supportive science. Statistics alone are just a tool that program professionals and policymakers apply to the questions they have. Without a user, there is no purpose for health statistics. Producing information to support programs and policies is the key to health statistics.

The public health surveillance system, for example, looks for changes in health events, and reports and investigates those changes to stimulate appropriate interventions to control the events. Public health surveillance, which often uses health statistics, is targeted to developing public health interventions to control the occurrence of adverse events, while health statistics are compiled for a variety of purposes, surveillance or intervention being only one of them.

During the working session, there was agreement that the use of the data often is not adequately considered in the developmental stages of data collection systems. It is important to know what is to be analyzed in order to build the collection mechanism appropriately. Involving health statisticians in the development of data collection strategies would improve the data collected and thereby the utility of the statistics.

In addition, participants recognized the need for closer relationships with those in public affairs roles to aid in appropriately translating health statistics into information for the public. Public affairs staff, along with those who produce health statistics, need to do a better of job of educating the public so they can discern credible statistics, as well as understand the need for the seemingly intrusive questions that sometimes appear on health surveys.

D. ROLES FOR HEALTH STATISTICS

Health statistics plays an intermediary role through translating data into information for supporting policy, particularly to improve health care and public health. Some participants also felt health statistics has a responsibility to set standards and norms, and bring added value to data that are collected.

1. Supporting policy – Participants agreed that the use of health statistics to support health policy is a defining characteristic of health statistics. Health statistics necessarily has a central role in formal and informal policy development and monitoring. In this role, health statistics touches actors in many phases of the policy process:

• researchers and analysts who formulate policy recommendations,
• decision-makers involved in health policy and legislation and implementation, and
• analysts responsible for evaluating those policies and their implementation.

Health statistics were noted as playing a special role in two policy areas: improving care and public health. Health statistics are needed to provide information on the quality and efficiency of health care delivery. One participant noted that while we have data on health care expenditures, health statistics should be more encompassing than monitoring health care services. Health statistics are particularly useful for developing public health interventions, which extend beyond the health care delivery setting. Another participant highlighted the lack of data to monitor and evaluate public health programs and actions.

2. Setting standards and norms - Considerable discussion occurred on the role of health statistics in fostering credible data and research. Discussion also occurred about the establishment of a set of minimum standards. One problem with surveys supported by private organizations, as well as surveys by the Federal government, is that the definitions used for the variables are not common across surveys, making it difficult both to interpret data and to make comparisons with other data in order to validate results. A recommended role of health statistics is to set the standards that define variables and set out how they are measured.

It was suggested that the HHS Data Council could coordinate a process to set norms for quality and standards for procedures and protecting confidentiality. The public should be involved in the process so that they understand the value it brings to them both in determining and producing credible statistics.

3. Adding value to data - Some participants felt that health statistics should make something more of collected data through:

• organizing data rationally,
• translating data into information that can give context to and focus issues, and
• publicly disseminating data and methods so that replication can occur.

One participant noted that technology has allowed health statistics considerably to expand its contribution and reach.

E. HEALTH STATISTICS AS A FIELD

Following up on the ambiguity mentioned earlier as to whether the term “health statistics” refers to data about health or to something that is known as a field of health statistics, the group began a dialogue as to what exactly the field of health statistics is. Is it a profession, a formal field, or an inherently multidisciplinary activity? Health statistics was conceptualized by some as a subspecialty of the field of statistics in general. Health statistics encompasses people who have diverse training and come from many disciplines, including medicine, statistics, computer science, behavioral science, epidemiology, and informatics. Few of these professionals come to the practice of health statistics with all the tools of the trade, but rather become health statisticians on the job. Health statistics may be more of an orientation than a field.

One participant noted that in order to have a field, there must be (a) formal, university- based training and (b) research in the discipline. These two conditions do not exist in health statistics. Other requirements of a field that are currently missing are a conceptual framework, integration of data, and an organized constituency of users.

F. HEALTH STATISTICS AS A SYSTEM

Participants were unable to identify an existing coherent health statistics system in the United States. This was seen as partially reflective of the fact that we do not have a coherent health care system. It generally was agreed that any system of health statistics would be made up of a series of statistics. A coherent system would be an organized link between the health care system and the users of health data, as well as between the population and users of health data. Given the time constraints of the working session, and the multiplicity of users and uses of health statistics, participants were unable to define a coherent health statistics system. Rather they noted the distinctions between multiple activities that do not hold together as a system largely because they are not all playing by the same rules or using the same definitions, standards and guiding principles.

It was suggested that if identifying such a health statistics system in the United States were desirable, then one might start with developing a conceptual model such as those developed by Evans and Stoddard, and Roos et al, which would assist in identifying current data availability and data gaps. The analysis should also examine the degree of data collection overlap. A health statistics system should strive for integration of collected data, rather than one grand data collection scheme.

Integration of collected data will also depend upon the use of common standards for terminology, data collection and data definitions. The proposed development of a conceptual model and analyses employing that model would move the U.S. closer to a coherent health statistics system. Such a system could accomplish goals that cannot be achieved by agencies alone. A collective process is needed to sort through health statistics system goals, and develop a model for the roles of both federal and non- federal players.

G. EVALUATION CRITERIA FOR HEALTH STATISTICS

A framework or model for health statistics is an important prerequisite for developing a coherent evaluation strategy. Participants agreed that probably the most important criterion by which a health statistics series or a health statistics system should be evaluated is the contribution it makes to health.

When selecting evaluation criteria, there are a variety of views to be considered – those of the producers, analysts, users and the broader policy and public arenas. The participants considered the first priority for evaluating health statistics to be the utility of the health statistics to users. It was suggested that a “successful” health statistics series or system requires a constituency. This can include health policymakers, health care providers, public health practitioners, program managers, researchers, legislators, business and consumers.

An evaluation must begin with the objectives of the health statistics series or system, which may be multiple. Examples are monitoring, tracking, informing decisions, directing interventions, explaining phenomena and predicting. How well does the series or system fulfill these objectives? Feedback loops are essential in assessing the quality and value of the statistics system and its components. Is the system flexible and adaptable to changing circumstances and needs, prescient rather than solely reactive, translating information adequately to users and capable of providing methodological feedback and improvement?

Evaluation criteria will want to consider different characteristics of the data or statistics. One participant suggested that the most important ones are whether the data and methods are available for peer review and public scrutiny, publicly accessible, confidential and secure, coherent and credible. Do the data make sense and fit together? Integration was determined an important evaluation criteria, whether unnecessary redundancies are eliminated and critical linkages are possible. Other important criteria include independence, timeliness, balance and comprehensiveness – of populations, resources, and determinants.

Participants concluded that we don’t systematically evaluate health surveys and data systems to examine whether we are collecting the “right” data and the different ways of collecting and disseminating information. A critical unmet need is investment in research on the statistics system, itself, and the infrastructure to assure that this research is an integral part of the system.

Finally, one participant reminded the group that this is not a perfect world; we are often “running between the rain drops.”

Appendix A

Working Session on Definition and Criteria for Health Statistics
Developing the 21st Century for Health Statistics
March 29-30, 1999

Objectives for the working session:

• initiate dialogue on the definitions of the field of health statistics, it purposes, its roles, and its boundaries, building on discussion group on future directions
• develop draft criteria for evaluating the current status of health statistics and the health statistics system, and progress towards implementing the 21st century health statistics system

develop draft criteria for evaluating the current status of health statistics and the health statistics system, and progress towards implementing the 21st century health statistics system

¨ help refine and shape discussions for the workshop on A Health Statistics System for the 21st Century

This agenda is intended only to be a guide and will be as flexible as the group deems appropriate.

Appendix B

Participant List for
March 29 - 30, 1999 Working Session towards
Developing the 21st Century Vision for Health Statistics