26-member panel conducts independent reviews
In March 2014, 32 health and social services providers, program directors, trainers and supervisors volunteered to assess the understandability and actionability of Beginnings Pregnancy Guideusing the PEMAT-P — Patient Education Materials Assessment Tool for Printable Materials from the Agency for Healthcare Research and Quality. Twenty-six completed the assessment. The secondary purpose of the review was to implement the new PEMAT-P and get a sense of its utility. Each of the six booklets comprising the Beginnings Pregnancy Guide was reviewed separately by four or five individuals working independently.
Beginnings Pregnancy Guide Earns High Marks Understandability: 96.5% Actionability: 98%
I am particularly pleased with the nearly perfect actionability score. Goethe said it centuries ago, “Understanding is not enough, we must act.”
Understandability: (a multisyllabic bit of Latinized jargon that is questionable in the context of health literacy) is the quality of information that enables users comprehend its meaning. The PEMAT-P asks reviewers to rate printable materials on 17 factors in six categories known to affect understandability. A score of 1 indicates the factor is present; 0 indicates it is not; NA indicates the factor is not applicable. Factor scores are expressed as the percent of possible scores of 1. The final score is calculated as the average of reviewers’ combined scores in each category. Here are the category scores:
Content: 94% Word Choice/Style: 96% Use of Numbers: 100%
Organization: 97% Layout/Design: 96% Visual Aids: 96%
understandable for a low level reader.”
Actionability (more jargon) is the quality of information that enables users to take action. Reviewers score seven contributing factors. The final score is calculated as the average of the factor scores. The Pregnancy Guide earned 100% on five of the actionability factors.
“An additional plus is the links to other resources for specific topics.”
Detailed results are displayed in Tables 1 and 2. Table 3 shows reviewers’ unsolicited comments on the materials.
PEMAT-P shows good reliability
Testing during development of the tool showed acceptable validity. Results of this project suggest the tool has good inter-rater reliability, meaning that multiple reviewers of the same materials rate the factors similarly.
About one third of the reviewers struggled with the PEMAT web page. Technical difficulties may have discouraged some of the six who did not submit a competed form. SeeTable 4.
Personally, I recommend relabeling the buttons in the top menu. I expected the PRINT button to print something; it brings up the Printed Materials form. The bottom menu buttons are inactive on my machine. Those buttons and the frame around the form take up space and require printing on two pages in too-small type. I, and some others, found the numbering on the Printable Materials form confusing; it skips items related only to audio-visual materials.
Overall, the PEMAT-P is a useful at-your-desk review that can improve materials in the development process and weed out complex, fact heavy, concept-dense materials. It cannot replace testing by intended users - both teachers and learners.
Kudos to the developers of the PEMAT: Michael Wolf and Cindy Brach
Thanks to the reviewers: Betsy Rubin, Lori Lake, Pamela Cho, Michelle Breuer, Dora McKean,, Kath Anderson, Joanne Martin, Tennessa Dallas-Theus, Hudelaine Deus, Oscar Flores, Cheryl Underwood, Marisela Rosales, Kobe Rives, Alli McClennen, Eva Perez, Lina Rooney, Elizabeth Burleson, Cynthia Smith, Denise Powell, Katie Burnett, Leslie Munson, Mary Rosecky, Jeffrey Wynnyk, Linda Wollesen, Margarita Franco, Maryellen Miller
Find the PEMAT-P at http://www.ahrq.gov/professionals/prevention-chronic-care/improve/self-mgmt/pemat/pemat-p.pdf