Fall 2001

This newsletter has previously reported on the early success of ARCH, one of the components of Youth Matters' 3-R Campaign to have all 3rd graders reading at grade level by 2010. ARCH stands for AmericaReads Richmond Chesterfield Henrico and is an in-school tutoring program for first and second graders. These early results showed that 83 percent of ARCH students improved at least one grade level in reading.

A closer look reveals that ARCH is a good example of the UHI's core value that "campaigns use a data-driven process to inform decisions about where and how to focus their efforts, and employ the most effective strategies."

Youth Matters and its local evaluators at Virginia Commonwealth University's Survey and Evaluation Research Laboratory meticulously gather data to gauge progress scientifically and to determine whether ARCH continues to be a worthwhile investment. This data gathering-and-use effort is serving as a model for other Youth Matters' endeavors and for the school systems themselves.

Jennifer Kelman directs ARCH and works with the VCU lab as program director for VCU Americorps and AmericaReads. Susan Kennedy Spain manages the data collection system as senior project manager for the VCU lab. They described the elements of the data gathering system:

Dosage data: Tutors report the actual number of hours of tutoring kids receive and other "dosage data" in order to assure the intervention is administered as intended and thus adding confidence to the final analysis of cause-and-effect. Plus, this information can help determine what dose (eg, number of hours tutored) is most effective.

Pre and post common measure: Dr. Kathleen Heubach at VCU developed a customized pre and post test for ARCH participants. Each school has other reading tests, but they differ so there was no way to compare scores across jurisdictions. "We needed some way to talk about all kids and measure progress," says Spain. There was some concern within the schools about over-testing of kids, as well as some initial lack of understanding of the need for the common measure, which is given to students upon entry into the ARCH program and at the end of the school year.

"Running records": The participating schools also administer commonly used reading tests, but scales are different among the jurisdictions, ARCH has no control over how they are administered, and sometimes either a pre or post test is not given. These test scores are gathered and they provide some context for the ARCH effort, but their limitations led to the development of ARCH's common measure.

PALS (Phonological Awareness and Language Skills) data : This statewide assessment is given to kids K-3, and the ARCH team uses it to select the kids to participate in ARCH. But not all kids take a posttest, so it can't be used as a measure of efficacy. For this info, though, jurisdictions differed in their capacity to provide it. The VCU lab helped get new procedures established.

Standards of Learning (SOL) data: This is another statewide test, given in the third grade. It is used as another benchmark to measure progress.

Report card data: This data also helps provide context for the ARCH program and its measurement of progress. For example, the report card data can answer what a "normal" change in reading would be from one reporting cycle to the next. However, while some jurisdictions could send these electronically, others could only provide hard copies, which made it difficult to pull out only the desired information. These schools now fill out forms with only the data needed. ARCH officials work with the schools to overcome any resistance to filling out yet-another form.

Qualitative data: Teacher surveys, satisfaction surveys of schools and more are administered to assure that ARCH is meeting the needs of all involved.

Youth Matters will apply similar techniques to other strategies. For example, the team is now gathering data on kids who are tutored via ARCH and also receive after-school assistance: Does the combination lead to better results? Not all after-school partners have the capacity to gather and report the needed data, so the VCU lab is assessing how much technical assistance would be needed.

Kelman says schools are becoming more engaged with ARCH as a result of the data gathering. "Schools are becoming more interested in how their students are doing," she says. "They recognize that the ARCH data gathering and analysis process is a valuable service that helps them gauge if this program is making an impact. As a result, they have become more actively engaged in ARCH over the years and are more willing to provide tutoring space and other supports to make the program happen.

"We hope that the data gathering system for ARCH will provide one model for the schools to consider in looking at which programs and services are making an impact," she says. "Schools have many well-meaning programs offered to them, and often there is no system in place to weed out what works and what doesn't, or what is too much."