Qualitative Results: IDEA Survey Student Comments Many students found the class challenging. Moreover, many of them found the value of having a service-learning component in class. The shorter time frame in the summer semester led the class being taught in a faster pace than usual. The class would probably be better taught in the regular 15-week semester and/or with a second follow-up (more advanced) program evaluation course. Overall this class has been challenging compared to other classes taken thus far. I have put a lot more effort on reading and working toward completion of each assignment. I definitely recommend this class on a 15 weeks session because the 10 weeks session is quite hectic. Excellent instructor and really thorough on papers feedback. The service-learning component is a great idea. While I don't have any suggestions at the moment, some thought into modifying that may be worthwhile. Perhaps is it is the compressed summer term, hard to say. It just felt difficult to meet expectations for both the program and class. The course was challenging, but I learned so much from it!Course, Students, and Community Partners
HS A628: Program Evaluation was taught for 10 weeks in Summer Semester 2013 via distance delivery method, using the Blackboard platform. There were a total of nine students in this class, of which three were males and the rest were females. Four of the students resided in Anchorage, one in Sitka, one resided in Juneau, one resided in Bristol Bay, one resided in Guam, and another resided in Colorado. The students were given an option to work with a partner or to work on their own for their service learning project. Due to different availability, geography, and time zone, most of the students decided to work on their own, while two partnered with each other.
For this class, students were asked to find a community organization or public health agency to partner with in developing an evaluation plan for one of the organization’s/agency’s health-related program. Overall, the students worked with a total of eight different programs from eight different community organizations/agencies; they are as follows:
Safe Kids Alaska: Child Passenger Safety Program
Northern Marianas College: Childcare Services Program
Southcentral Foundation: Lose to Win Program
Alaska Literacy Program: Get Active Program
Catholic Community Services of Juneau: Time Limited Family Reunification Program
SEARHC: Park Prescriptions Program
University of Colorado Health North: Bicycle to Work Program
Providence Alaska Medical Center: Lung Cancer Screening Program
Confidence in Performing Program Evaluation
The C-Scale is a set of five questions, each scored at five points each, giving a total score ranging from 5 to 25 points, with 5 as not confident and 25 being extremely confident in performing program evaluation in the real world setting. Mean confidence score in posttest was higher than pretest (13.0 ± 3.2 S.D. vs. 8.1 ± 3.8 S.D.), however, their difference was not statistically significant (P = 0.06).
Figure 1. Pretest and posttest mean confidence score of students with regards to performing program evaluation in the real world setting (N=7 pretest; N=4 posttest). Total score ranges from 5 to 25, with a score of 5 as “not confident” and a score of 25 as “extremely confident.
Student Learning Outcomes
Six student learning outcomes were assessed.Each item were scored at four points each, giving a total score ranging from 4 to 24 points, with a score of 4 to mean the student strongly disagrees the student learning outcomes were achieved and 24 points to mean the student strongly agrees the student learning outcomes were achieved.Mean student learning outcomes score at posttest was significantly higher than pretest (20.1 ± 1.7 S.D. vs. 13.9 ± 2.3, P < 0.001).
Figure 2. Pretest and post test mean student learning outcomes score of students (N = 7). Total score ranges from 4 to 24 points, with 4 points being student strongly disagreeing that the student learning outcomes were achieved and 24 points being student strongly agreeing that the student learning outcomes were achieve.Mean scores are significantly different, P < 0.001.
Public Health Core Competencies
The course addressed five public health core competencies. Each competency was scored at four points each, giving a total score ranging from 4 to 20 points, with 4 points to mean the student strongly disagrees they are competent in the five core competencies and 20 points to mean the student strongly agrees they are competent in the five core competencies. Mean core competency post test score was significantly higher than pretest score (16.1 ± 2.1 S.D. vs. 12.7 ± 1.8 S.D. P < 0.05).
Figure 3. Pretest and post test mean public health core competency score of students (N = 7). Total score ranges from 4 to 20 points, with 4 points being the student strongly disagrees he/she is competent in five core public health core competency and 20 points being the student strongly agrees he/she is competent in five core public health core competency. Mean scores are significantly different (P < 0.05)
Qualitative Results: Reflection Journals
Based on the students’ reflective journals, the students learned important lessons relevant to program evaluation and/or being an evaluator for each assignment provided. The three common lessons learned by the student throughout the process are as follows:
Creating a quality evaluation plan takes a lot of time and effort.
I learned the quite time consuming process to conduct an evaluation program and to have to deal with several pages of information that are needed to be put together in order to write a final evaluation report… This learning matters because of the complexity of being an evaluator or in a position where I may have to represent stakeholders for a program being evaluated.
I learned that program evaluations take a tremendous amount of time and effort, much more than I had put forth in the past and perhaps to the detriment of those programs… This learning matters because I see the actual benefit of a structured and complete program evaluation.
Constant communication with the appropriate person in the program/agency is important. Communication can help clarify important issues in the evaluation, but more importantly it can help improve the relationship between the evaluator and the agency.
One of the important lessons I have learned these last two weeks is to be the squeaky wheel. I kept emailing and checking in via email with Mr. Olpin; this diligence at least allowed me the security that I was doing everything possible to be an effective communicator even if I didn’t receive emails on a regular basis. I also learned the face to face meetings might have more merit, as I can directly ask follow up question…
I learned persistence and ongoing communication, even if brief, are necessary to maintain a connection. All the information needed may not be readily available, so being resourceful in finding information is an important skill. Also, determining the accuracy and reliability is critical as well in order for the evaluation to be useful. Again, setting realistic time frames is easier said than done. In a more flexible setting, I would have renegotiated deadlines. At this point, having Carin’s input for a final paper would have been very helpful in addressing weakness or making suggestions. It’s clear having consistent input from stakeholders would really improve the evaluation process, and likely make it a more useful tool for everyone.
Having a service-learning component in a program evaluation course provides valuable real-world experience.
I learned that a program evaluation can require tons of information and different documents that are needed to be sorted out in order to get the type of information you are looking for… This learning matter is particularly important for the student that I am in a way that it gives me rudiments for real world experience on program evaluation and it also teach me the technics, concept and methods needed to conduct program evaluation for the field of health profession.
The last several weeks have helped me better understand the inner workings involved in the implementation of the Lung Screening Program at Providence. Taking the assignments from throughout the semester and combining them to create the final evaluation plan has taught me how to better break down large projects into smaller more manageable pieces. Putting together the final evaluation plan during the past few weeks has been a helpful process, because I am able to see how each element is relevant to the other. By combining pieces, I was better able to see interworking of the process, which will be helpful to know for future similar projects. I feel better prepared to ask relevant questions and have a general framework for this evaluation process… The increased experience that I have gained over the last three assignments has allowed me to be better prepared for the meetings with Mr. Olpin as well as the stakeholders. In addition, the experience as allowed me to better express the intent of what was required by assignment four as well as being prepared with any questions Mr. Olpin may have had or I may have needed to ask to get a better picture of updated participation by lung screening participants. It would be very interesting to see if this project can develop into a thesis.
Qualitative Results: IDEA Survey Student Comments
Many students found the class challenging. Moreover, many of them found the value of having a service-learning component in class. The shorter time frame in the summer semester led the class being taught in a faster pace than usual. The class would probably be better taught in the regular 15-week semester and/or with a second follow-up (more advanced) program evaluation course. Overall this class has been challenging compared to other classes taken thus far. I have put a lot more effort on reading and working toward completion of each assignment. I definitely recommend this class on a 15 weeks session because the 10 weeks session is quite hectic. Excellent instructor and really thorough on papers feedback. The service-learning component is a great idea. While I don’t have any suggestions at the moment, some thought into modifying that may be worthwhile. Perhaps is it is the compressed summer term, hard to say. It just felt difficult to meet expectations for both the program and class. The course was challenging, but I learned so much from it!