Completeness

Completeness can be considered as having two separate dimensions; internal and external completeness.

Internal completeness refers to whether there are missing records or data fields and can be defined as “the frequency of unknown or blank responses to data items in the system”.

External completeness relates to whether the data available to the surveillance system reflects the true number of cases diagnosed with notifiable conditions [Doyle]. One approach to evaluate external completeness consists in the comparison of at least two datasets from different sources of information that are supposed to provide surveillance information on the same disease (e.g. laboratory and notification data for case reporting of salmonellosis). A common method to measure external completeness is “capture-recapture”. However other methods can certainly be used to compare datasets depending on the disease under surveillance, the nature/accessibility of data sources, and other parameters to be defined.

Note: consider completeness of data within a case data or completeness of data within a database (set of cases collected for a time window).

Validity

Validity in the context of surveillance would be the capacity to capture the “true value” for incidence, prevalence or other variables that are useful analysis of surveillance data. The “true value” should be viewed in the context of the surveillance system and objects, for example, it may relate to only those cases diagnosed by health services under surveillance. Validity can be considered to comprise of both internal and external dimensions, where:

Internal validity relates to the extent of errors within the system, e.g. coding errors in translating from one level of the system to the next.

External validity relates to whether the information recorded about the cases is correct and exact. Evaluating external validity implies the comparison of a surveillance indicator measured in a dataset to a “gold standard” value [Doyle]. One possible way to conduct validation study is to compare data recorded in the studied dataset to the original medical records. If data on a same patient are recorded at different points in time for the same information (disease/variable), differences can be due to a “real change“ over time or a bias in the measurement. Reliability studies can help identify this type of bias.

Sensitivity

Sensitivity as a proportion of persons diagnosed with the condition, detected by the system:Sensitivity of the EU-wide surveillance sub-network reflects the combined sensitivity of the national surveillance systems and of the international network, and can be defined on three levels relevant to EU-wide surveillance: (a) the proportion of cases notified to the national system that were reported to the international coordinating centre of the sub-network; (b) the proportion of cases fulfilling the standard case definition, diagnosed at the local level, that were notified to the national system; (c) the proportion of cases detected by the national surveillance system out of all cases truly occurring in the population without respect to whether cases sought medical care or a laboratory diagnosis was attempted (can usually only be determined by special studies). In practice, the sensitivity of the national surveillance systems will determine the sensitivity of the overall surveillance system. However, the sensitivity of national surveillance systems, ie the ratio between (a) and (b) will vary widely from country to country for specific diseases.

When knowledge on the differences in the sensitivity of national surveillance systems is of importance to the objectives of the EU-wide surveillance sub-network, country-specific investigations need to be implemented with defined methodology to determine the sensitivity of the national surveillance systems and to form a basis for comparability of the country-specific data. Stringent criteria for inclusion of cases in several of the existing networks will make EU-wide sensitivity lower than national [Ruutu et al]

The sensitivity of a surveillance system can be considered on two levels. First, at the level of case reporting, sensitivity refers to the proportion of cases of a disease (or other health-related event) detected by the surveillance system. Second, sensitivity can refer to the ability to detect outbreaks, including the ability to monitor changes in the number of cases over time. [CDC guidelines]

Predictive value positive (PVP) is the proportion of reported cases that actually have the health-related event under surveillance [CDC guidelines]

Timeliness

Timeliness reflects the speed between steps in a public health surveillance system [CDC guidelines]

Reactivity would reflect the retro-information and delay necessary to initiate a public health action

Representativeness

A public health surveillance system that is representative accurately describes the occurrence of a health-related event over time and its distribution in the population by place and person [CDC guidelines]

Representativeness: system accurately describes the occurrence of disease over time and its distribution in the population. Knowledge on the representativeness of surveillance data on the national level is important for some of the proposed specific objectives of EU-wide surveillance. Cases notified to a surveillance system may be derived, e.g. for practical reasons, unevenly from the population under surveillance, and therefore not be representative of the events in the population in general. The data would thus reflect poorly the situation nationally and on EU level. Over time, the representativeness of surveillance data may change for a number of reasons, such as changes in legislation, surveillance infrastructure, clinical practice and reimbursement policies. A change in representativeness will lead to wrong conclusions when, e.g. trends in surveillance data are compared between countries or on EU level. Stability in the representativeness on the national surveillance level needs to be monitored, and the quantitative effect of changes on representativeness assessed [Ruutu et al.]

Usefulness

A public health surveillance system is useful if it contributes to the prevention and control of adverse health-related events, including an improved understanding of the public health implications of such events. A public health surveillance system can also be useful if it helps to determine that an adverse health-related event previously thought to be unimportant is actually important. In addition, data from a surveillance system can be useful in contributing to performance measures, including health indicators that are used in needs assessments and accountability systems [CDC guidelines].

Simplicity

The simplicity of a public health surveillance system refers to both its structure and ease of operation. Surveillance systems should be as simple as possible while still meeting their objectives [CDC guidelines]

Flexibility

A flexible public health surveillance system can adapt to changing information needs or operating conditions with little additional time, personnel, or allocated funds. Flexible systems can accommodate, for example, new health-related events, changes in case definitions or technology, and variations in funding or reporting sources. In addition, systems that use standard data formats (e.g., in electronic data interchange) can be easily integrated with other systems and thus might be considered flexible [CDC guidelines].

Acceptability

Acceptability reflects the willingness of persons and organizations to participate in the surveillance system [CDC guidelines]

Acceptability is influenced substantially by the time and efforts required to complete and submit reports or perform other surveillance tasks.

The simplicity of a surveillance system refers to both its structure and ease of operation. Surveillance systems should be as simple as possible while still meeting their objectives.

Methods

A chart describing the flow of information and the lines of response in a surveillance system can help assess the simplicity or complexity of a surveillance system. A flow chart for a generic surveillance system is illustrated in Figure 1.

The following measures might be considered in evaluating the simplicity of a system:

  • Amount and type of information necessary to establish the diagnosis
  • Number and type of reporting sources
  • Method(s) of transmitting case information/data
  • Number of organizations involved in receiving case reports
  • Staff training requirements
  • Type and extent of data analysis
  • Number and type of users of compiled case information
  • Method of distributing reports or case information to these users
  • Time spent with the following tasks:
    • Maintaining the system
    • Collecting case information
    • Transmitting case information
    • Analyzing case information
    • Preparing and disseminating surveillance report

Discussion

 

It may be useful to think of the simplicity of a surveillance system from two perspectives: the design of the system and the size of the system. An example of a system that is simple in design is one whose case definition is easy to apply and in which the person identifying the case will also be the one analyzing and using the information. A more complex system might involve some of the following:

 

Special laboratory tests to confirm the case

 

Telephone contact or a home visit by a public health nurse to collect detailed information

 

Multiple levels of reporting (e.g., with the Notifiable Diseases Reporting System, case reports may start with the doctor who makes the diagnosis and pass through county and state health departments before going to the Centers for Disease Control) Simplicity is closely related to timeliness and will affect the amount of resources that are required to operate the system

Stability

Stability refers to the reliability (i.e., the ability to collect, manage, and provide data properly without failure) and availability (the ability to be operational when it is needed) of the public health surveillance system [CDC guidelines]

The adequacy would refer to the ability of the surveillance system to address its objectives.


References: