VA is committed to ensuring that those who use VA’s reported performance information to make decisions can do so with the confidence that our data are reliable and valid. Developing policy to ensure data quality, establishing oversight authority, using the expertise of the Office of the Actuary, and using performance audits to objectively assess the reliability, validity and integrity of the data will provide senior managers with needed assurances about the quality of VA’s data.

VA needs to establish sound policy for data quality at the Department level that would include among other things, standardization of data definitions; use of internal controls; data sources; and data reliability, validity, and integrity checks. Senior managers are considering the establishment and need for a VA Data Council to provide oversight on data verification issues and practices.

Upon establishment of Department key performance measures, it was critical to senior managers that the quality of the data reported be objectively verified for accuracy. The Office of the Inspector General (IG), through performance audits, provides an important and objective assurance of data quality. To date, the IG has completed performance audits on five key measures and has plans to initiate performance audits on other key measures.

In order to ensure a greater understanding among VA staff and managers, IG auditors provided the following definitions:

• validity—do the data represent what they are supposed to or intended to;
• reliability—are the data consistent and can they be replicated; and
• integrity—can the data be gamed or manipulated.

In reviewing data validity, reliability, and integrity, the IG work is being performed in accordance with GAO’s Assessing the Reliability of Computer Processed Data, popularly known as the Gray Book.

During FY 1999, the IG completed audits of VHA’s number of unique patients and NCA’s percentage of the veteran population served by the existence of a burial option within a reasonable distance of place of residence.

Based on its audit of unique patients, the IG concluded that we overstated the patient count by 5.7 percent. The IG cited two major reasons for this:

• inaccurate SSNs were entered into the National Patient Care Database
• patients with undocumented appointments or who did not keep their appointments were counted as being treated.

The Acting Under Secretary for Health agreed with the recommendations in the IG’s report and provided an acceptable implementation plan.

The NCA performance audit showed that NCA personnel generally made sound decisions and accurate calculations in determining the percent of veterans served by a burial option. However, inconsistencies in NCA’s estimate of the percent of the veteran population served by a burial option were identified. Although these inconsistencies did not have a material impact and no formal recommendations were made, adjustments have been made to data collection practices by NCA. The validity and reliability of the NCA measure was based on a review of adjustments made by VA personnel to veteran population data received from the Census Bureau, an evaluation of the decision to define a cemetery’s service area (in most cases, as the area within a 75-mile radius), an assessment of the mapping software used by NCA personnel, and data input and output for a stratified random sample of cemeteries.

During FY 1999, the IG began two new audits. One audit is to determine whether VBA personnel accurately computed the foreclosure avoidance through servicing (FATS) ratio and the other is to determine the accuracy of the VHA performance measures on the chronic disease care index (CDCI) and the prevention index (PI).

The VBA audit will include an assessment of pertinent internal controls at selected regional offices and the Austin Automation Center, reviews of random samples of successful interventions, refundings, voluntary conveyances, compromises, and foreclosures recorded in the Liquidation and Claims System (LCS) to test authenticity, reviews of random samples of cures and payments to test completeness of data in LCS, and an assessment of the program used to compute the ratio. The audit is expected to be completed in 2000.

The VHA audit will evaluate the statistical sampling methodology and assumptions to determine if it produces results that are representative of actual treatment provided by VHA, examine the data processing systems in which CDCI and PI data were input to determine whether the data were processed accurately, whether there were adequate controls to prevent bad data from processing, and compare source documents and data from the automated systems to determine whether the proper data were input accurately and if there is sufficient supporting documentation in the medical records. The audit is expected to be completed by early FY 2001.

As a standard practice of accountability, the IG will follow-up all recommendations made regarding data integrity, validity, and reliability on all performance measure audits. The IG is responsible for maintaining the Department’s centralized, computerized follow-up systems that provide for oversight, monitoring, and tracking of all IG recommendations through both resolution and implementation. Resolution and implementation actions are monitored to ensure disagreements between the IG and management are resolved as promptly as possible and corrective actions are implemented as agreed upon by management officials. Disagreements unable to be resolved between the IG and management are decided by the Deputy Secretary, VA’s audit follow-up official. Management officials are required to provide the IG with documentation showing the completion of corrective actions. IG staff evaluates information submitted by management officials to assess both the adequacy and timeliness of actions and to request periodic updates on an ongoing basis.

Internal controls exist for many of VHA’s data collection efforts. However, the quality of the data is receiving greater scrutiny. In response to criticisms, VHA is making significant progress toward developing a data quality strategy to provide the necessary internal control processes that have been lacking in the system with regard to data validity, reliability, and integrity. Specifically, a GAO draft report on VA Access to Care and the Need for Better Oversight of Veterans Equitable Resource Allocation (VERA) System, noted throughout the report that data quality issues had a negative impact on the ability of GAO to make an accurate assessment of issues and that standardization of collecting and analyzing data was lacking. In response to the GAO report , the Under Secretary for Health requested that VHA hold a Data Validation Summit to develop strategies to eliminate such problems as lack of standard definitions, decentralized approaches to data collection and implementation of automated systems, local modification, lack of knowledge/understanding, and difficulty of coordination of more than 140 VHA data bases contributing to data validity problems. In addition to addressing these problems, the Data Validation Summit identified corporate data issues impacting VHA’s ability to provide comprehensive, accurate, business-needs data as efficiently as possible to a variety of VA business data users and other data customers. Summit participants are working on five initiatives to address data quality in VHA: form a data quality council to provide leadership (currently awaiting approval from the Acting Under Secretary for Health); define a standards process; define and implement local accountability for data; establish on-going employee education, training, and communication programs for VHA data users; and provide patients with access to their data. These efforts provide a foundation for VHA’s continuing process to improve the quality, availability, and accuracy of information to support the decision process.

In order to provide a VISN level management tool, the VHA Decision Support System (DSS) Steering Committee has just completed a test of four national measures. Based on the outcome of these tests, recommendations regarding standardization and data quality will be forwarded to VA’s National Leadership Board for action.

The validity of VHA’s electronic databases has been assessed in a number of studies by researchers, with adequate validity being found for most data elements. For those measures where data are collected as a result of chart review, medical record reviews have been performed with computerized algorithms to enhance their reliability. In addition, abstractors have received intensive training in the application of the criteria prior to abstraction and have a "help desk" available to them during abstraction to answer questions about difficult charts. Inter-rater reliability has been assessed with the level of agreement being at least adequate for all performance indicators, when compared to generally accepted standards. Extensive psychometric testing of the customer feedback instruments has been performed to establish their reliability and validity. In addition, validity has been enhanced by risk adjusting facility data for age, gender, and health status, and by using a wide variety of survey procedures to obtain high response rates. The validity of the self-report measures has been considerably enhanced through on-site visits for randomly selected facilities.

C&P’s program automated information system was vulnerable to reporting errors and the ability to erroneously enter data to show better performance than was actually achieved. VBA has taken several steps to ensure it has accurate and reliable data for planning and management purposes. VBA collects and stores in a database all end product transactions from all stations since October 1, 1997. From that database, VBA extracts and reviews transactions, and identifies questionable actions for the following types of claims: original and reopened compensation, original and reopened disability pension, original dependency and indemnity compensation, and original death pension. In August 1998, the C&P Service issued a letter providing written clarification of the most commonly misunderstood control and work credit issues.

On a weekly basis, data are extracted and records of questionable transactions within and among regional offices are made available via the Intranet. C&P tracks the questionable transactions for each office and prepares quarterly summaries which are also available to local field managers via the Intranet.

The C&P Service also tracks the percent of questionable end product transactions for each office. For those stations having the highest percentage of questionable transactions, these sites were identified for case call-in review. The first case call-in review of approximately 500 cases, from five selected regional offices, took place during April 1999. Based on the results of this review, Office of Field Operations and the C&P Service management met with the regional office directors and staff representatives in June 1999 to discuss the findings. Each office was required to submit an action plan for addressing end product improprieties.

NCA workload data are collected monthly through field station input to the Management and Decision Support System (MADSS), the Burial Operations Support System (BOSS), and the Automated Monument Application System - Redesign (AMAS - R). Headquarters staff review the data for general conformance with previous report periods, and any irregularities are validated through contact with the reporting station.

NCA conducts an annual survey of the families of individuals who are interred in national cemeteries and of other visitors to measure how the public perceives the appearance of the cemeteries and the quality of service provided. This information provides a gauge by which to assess maintenance conditions at the cemeteries and our success in delivering service with courtesy, compassion, and respect. The survey provides us with data from the customer’s perspective, which is critical to developing our objectives and associated measures. VA headquarters staff oversee the survey process and provide an annual report at the national level. NCA Area Office and cemetery level reports are provided for NCA management use.

Efforts are also underway to expand the use of information technology to collect performance data for recently developed performance measures. NCA has established a Data Validation Team whose goal is to ensure that performance data collected and reported for timeliness of scheduling interments and setting headstones and markers are accurate, valid and verifiable. The team’s major tasks include defining performance measurement terms to ensure standard interpretation and application throughout NCA; identifying training needs to ensure accuracy of data and consistent data entry processes; and recommending necessary changes to the Burial Operations Support System to help ensure accurate data are entered.

In its December 1996 report, the Veterans Claims Adjudication Commission observed many critical decisions relative to VA programs were not supported by "valid data and long-term analyses of program needs." To this end, the Commission recommended, and the Secretary of Veterans Affairs agreed, VA should establish a capacity for actuarial analysis at the Department level. In establishing the position of Chief Actuary, the Department acknowledged actuarial analysis will significantly benefit the evaluation of the long-term financial commitment of VA programs to individual veterans and their dependents. Further, VA expects this function to influence such other areas as the demographics of beneficiaries, disability rates, life-time utilization of VA programs, and projections of future beneficiaries and VA workload. In July 1999, VA successfully recruited its first Chief Actuary.

To effectively serve VA and its beneficiaries, the Chief Actuary will need to ensure he is positively impacting the validity and accuracy of VA data. As his role evolves, VA anticipates improvements will ultimately result from the Chief Actuary’s efforts to create new data sets and improve old ones. Most immediately, the Chief Actuary, supported by a major contract with a nationally recognized actuarial firm, is implementing significant enhancements to the national estimates of the number and characteristics of veterans. Additionally, as a heavy user of many of VA’s administrative data sets, the Chief Actuary will explore relationships between the data elements and ask questions that may not have been contemplated when the data set was created. In this process, data are validated. The Chief Actuary is available to provide actuarial assistance to data developers throughout the Department and will also research exogenous data for useful information.

As a profession, actuaries apply Actuarial Standards of Practice to their work. According to Actuarial Standard of Practice No. 23 "Data Quality," data should be reviewed for reasonableness and consistency, any actual or potential material biases should be disclosed, and documentation to support the use of specific data should be maintained. Consequently, VA expects the results of an actuarial review will be valuable feedback to data developers to help them improve the validity and accuracy of their data.

Over the last year, VA has made progress within the Department to begin the process of addressing both the data verification methods used by our three major operating elements as well as data limitations. In that regard, VA has continued to work to develop a cooperative relationship with the IG, communicated the importance of internal controls to program managers, and monitored ongoing efforts within VA to improve data reliability, validity, and integrity.

It is this cooperative partnership that sends the message to VA’s employees and managers that data integrity, validity, and reliability must be taken seriously and that VA expects to be held accountable for reported performance information.

Initiating a data verification process policy will increase confidence that there is a high level of data validity and reliability. Additionally, such a process will help ensure there is a lack of evidence for systematic bias.

VA recognizes that performance measure auditing should not be the only source for ensuring validity, reliability, and integrity of our data. As we meet our responsibility for providing accurate performance reports, we need to establish additional mechanisms for ensuring data quality. We recognize that VA must develop, implement, and monitor a Departmental policy on data verification methodologies to reduce, and ultimately eliminate, questions about the quality of our data.