The Test and Evaluation Process


Throughout its development, the Countering Illicit Power courseware was tested on a variety of training audiences. The model curriculum was designed to support a combination online and residence program at the graduate level taught by the Naval Postgraduate School. Students were given reading and listening assignments in advance. Their independent study was then supported by eight hours of classroom instruction and facilitated discussion, reinforced by small group projects based on Impunity case studies that were not covered in the classroom. The group project required students to retrospectively apply the principles and frameworks they had learned to the real world, historical examples. They were graded on oral presentations and written analysis. 

The materials were repackaged in a seminar format for regional National Guard units assigned to a counterdrug mission. The purpose was to improve understanding of the intersection between strategy, policy, and the tactical mission those units were implementing at the time. The focus was on strengthening analytic understanding of targeted networks, the networks’ enabling environment, and the Guard’s ability to more effectively collaborate with its interagency partners. 

A third variation was developed for the Reserve Component. In this test, a civil affairs battalion participated in 20 hours of iterative training during weekend drills over a period of several months. Using one, “model syllabus” module at a time, students were gradually introduced to the enduring insights. On-site facilitation enabled them to digest the case studies and apply their lessons to a number of different real-world missions the unit was performing throughout the Middle East. Podcasts and reading assignments prepared participants for distributed learning over a variety of different communications platforms to accommodate individuals who were engaged in train, advise, and assist missions. The webcasts created further opportunities to add depth to their understanding. The test unit concluded that the value of the courseware was indisputable, and is looking to institutionalize the curriculum into an intensive active-duty-for-training (ADT) program at both Brigade and Battalion level. 

Finally, outside of DOD, the materials were used as the core components of graduate and post-graduate courses and seminars for international rule of law and governance development professionals. In these experiments, the courseware was supplemented by other research to add academic rigor. The Countering Illicit Power courseware proved to be some of the most effective implementer-focused material, adding an operational perspective not normally seen in this type of program. The international students, most of whom were practicing professionals in their home countries or international organizations, and representing more than 30 countries, also affirmed the validity and relevance of the insights and lessons presented in Impunity and Convergence.  

In all instances, participant audiences praised the content of the courseware for its operational focus. In pre- and post-learning assessments, students rated themselves as having increased their knowledge and understanding of illicit power by 1.5 – 3.0 points on a scale of five, depending on the module and their prior experience. In no case was the post-assessment “endpoint” ever lower than four. Demand for further training rapidly outstripped the project team’s ability to support.