ImPact Test-Retest Reliability: Reliably Unreliable?

This study is an observational study of the reliability of follow up tests. In my personal opinion, ANY patient who is SYMPTOMATIC but has a normal test AND any ASYMPTOMATIC patient with an abnormal test should have an expert evaluation .

Do experts review all these baseline tests and provide an interpretation?

Do parents get results and an option to evaluate their child?
Are schools  putting students with abnormal tests in the game?

hmmm...

 JR

ImPact Test-Retest Reliability: Reliably Unreliable?

Jacob Resch , PhD, ATC*; Aoife Driscoll , BSc†; Noel McCaffrey , MD†; Cathleen Brown , PhD, ATC‡; Michael S. Ferrara , PhD, ATC, FNATA‡; Stephen Macciocchi , PhD, ABPP§; Ted Baumgartner , PhD‡; Kimberly Walpert , MD‖

*University of Texas at Arlington;

†Dublin City University, Ireland;

‡University of Georgia, Athens;

§Shepherd Center, Atlanta, GA;

‖Georgia Neurological Surgery, Athens

Context: Computerized neuropsychological testing is commonly used in the assessment and management of sport-related concussion. Even though computerized testing is widespread, psychometric evidence for test-retest reliability is somewhat limited. Additional evidence for test-retest reliability is needed to optimize clinical decision making after concussion.

Objective: To document test-retest reliability for a commercially available computerized neuropsychological test battery (ImPACT) using 2 different clinically relevant time intervals.

Design: Cross-sectional study.

Setting: Two research laboratories.

Patients or Other Participants: Group 1 (n = 46) consisted of 25 men and 21 women (age = 22.4 ± 1.89 years). Group 2 (n = 45) consisted of 17 men and 28 women (age = 20.9 ± 1.72 years).

Intervention(s): Both groups completed ImPACT forms 1, 2, and 3, which were delivered sequentially either at 1-week intervals (group 1) or at baseline, day 45, and day 50 (group 2). Group 2 also completed the Green Word Memory Test (WMT) as a measure of effort.

Main Outcome Measures: Intraclass correlation coefficients (ICCs) were calculated for the composite scores of ImPACT between time points. Repeated-measures analysis of variance was used to evaluate changes in ImPACT and WMT results over time.

Results: The ICC values for group 1 ranged from 0.26 to 0.88 for the 4 ImPACT composite scores. The ICC values for group 2 ranged from 0.37 to 0.76. In group 1, ImPACT classified 37.0% and 46.0% of healthy participants as impaired at time points 2 and 3, respectively. In group 2, ImPACT classified 22.2% and 28.9% of healthy participants as impaired at time points 2 and 3, respectively.

Conclusions: We found variable test-retest reliability for ImPACT metrics. Visual motor speed and reaction time demonstrated greater reliability than verbal and visual memory. Our current data support a multifaceted approach to concussion assessment using clinical examinations, symptom reports, cognitive testing, and balance assessment.

Key Words:  intraclass correlation, concussions, mild traumatic brain injuries, neuropsychological 
testing, athletes

FULL study here

Address correspondence to Jacob Resch, PhD, ATC, University of Texas at Arlington, Box 19259, 113 Maverick Activities Center, Arlington, TX 76102. Address e-mail to resch@uta.edu.

Concussion tests are outpacing scientific research

Computerized concussion testing is becoming more common in sports even though there is little scientific research regarding the tests' effectiveness. Professionals should learn the benefits and limitations of the computerized tests they use.

Computerized neurocognitive testing for concussions is widely used in amateur and professional sports, but little research over the past decade proves its effectiveness, a paper published this month in the journal Neuropsychology Review says.

Jacob Resch, director of the Brain Injury Laboratory at The University of Texas at Arlington, is lead author on the review, which updates a 2005 look at the available research on computerized neurocognitive testing. In 2005, researchers said not enough evidence existed to support clinical use of the then relatively new assessments.

The more recent work acknowledges that computerized tests, such as those marketed under the name ImPACT, HeadMinder, CogState, and ANAM, have become extremely commonplace across the sports world. But, the authors still urge caution with their use and point out a need for more peer-reviewed studies.

"Limited data has been published since 2005 to assist clinicians in determining the clinical value of this form of testing," Resch said. "While these products are an important component of concussion management, their development, marketing and sales seem to have outpaced the evidence. So, some caution is needed."

Co-authors are Michael McCrea, an author of the 2005 study and director of brain injury research at the Medical College of Wisconsin, and C. Munro Cullum, professor and head of the neuropsychology program at UT Southwestern Medical Center.

"Given the attention that concussion in sport has gained in recent years, it is surprising there has not been more research into the some of the newer computer-based methods used to evaluate post-concussion symptoms," Cullum said. "Since there is no single brain-test or biomarker for concussion at this point, the diagnosis of concussion remains a challenge in many cases, as it relies upon reported and observed symptoms."

Resch, McCrea and Cullum found 29 peer-reviewed articles since 2005 have addressed the characteristics of commercially available computerized neurocognitive tests. After a detailed analysis, they concluded that evidence on reliability and validity of the tests wasn't consistent.

For example, in a May 2013 study published by the Journal of Athletic Training, Resch and other researchers found that the ImPACT test misclassified healthy study participants as impaired as much as 46 percent of the time for some evaluation factors. ImPACT stands for Immediate Post Concussion Assessment and Cognitive Testing and is by far the most used computerized neurocognitive test for concussion management.

An Institute of Medicine report early this year said that the number of people 19 and under treated in U.S. emergency rooms for concussions and other non-fatal, sports and recreation-related traumatic brain injuries increased from 150,000 in 2001 to 250,000 in 2009. With the recent settlement of a landmark lawsuit filed by former NFL players, concussions remain in the headlines and on the minds of athletes, parents, coaches and others in the sports world.

Recently, nearly 40 percent of athletic trainers reported using a computerized neurocognitive tests as part of their response to a sports-related concussion, according to the new paper. The convenient computerized tests are typically used, just as pen and paper versions were in the past, to establish a baseline to use as a point of comparison after an athlete is injured. But, because concussion symptoms can vary widely across individuals and may be subtle, clear documentation of mild brain injury can be difficult.

The findings of the new survey should serve as a caution to those utilizing and interpreting computerized cognitive test results, the researchers said.

"Neurocognitive testing is an important component of the concussion assessment, but should not be used as a stand alone method to diagnose injury or determine an athlete's level of recovery and fitness to return to play," said McCrea. "A multi-dimensional approach is supported by the evidence as best practice."

Until more research is done, the new paper from Resch, McCrea and Cullum recommends healthcare professionals who use the tests take some precautions, including:

• Making an informed decision when choosing a test and investigating its limitations.
• Making sure the test is part of a multi-faceted concussion management approach.
• Incorporating a clinical neuropsychologist into the sports-related concussion management team to assist in interpretation of test results.
• Ensure proper training of those administering the test.
• Conducting quality control reviews of baseline testing.

Read more here