Preparing for the Real World Through Service Learning

By Dr. Gabriel Garcia, Assistant Professor
Department of Health Sciences, University of Alaska Anchorage

Context of the Inquiry

Context of Inquiry: HS/SWK A628: Program Evaluation (Summer 2013)

I teach in the Master's of Public Health Program in UAA's Department of Health Sciences.  One of its required courses is HS/SWK A628: Program Evaluation.

In the past four years of teaching Program Evaluation, I realize that several students have difficulty understanding and application of program evaluation concepts. Moreover, I was concerned that at the end of the semester, some students still feel that they lack confidence in doing their own program evaluation in the real world setting.  Thus, in teaching this class in Summer Semester 2013, I added a service learning component in my class, wherein each of my students would find a partner nonprofit organization and develop an evaluation plan for one of its health or social work related program.

It is important to note that this course that is currently being evaluated was taught in a 10-week period.  Also, the course was 100% distance delivered via Blackboard platform and asynchronous format.

Course Artifacts

Focus of the Inquiry

Question of Inquiry 


Does the addition of a service-learning component in class increase student confidence in doing program evaluation in the real world setting and enhance student learning?

 

Course Artifacts

Course Design and Implementation

Course Design

HS/SWK A628: Program Evaluation was delivered all online, using the Blackboard platform, and in asynchronous format.

The course was offered in Summer Semester 2013 for a period of ten weeks.

The course is mainly discussion driven, with brief lectures to supplement and clarify reading assignments.


Methods

Study Design. Pre- and post-test self-completed surveys and results of IDEA Course Evaluation Survey given at the end of the semester will be used to assess changes in level of confidence and enhanced learning among the students. Additionally, qualitative data will be collected to confirm and explain quantitative findings. Qualitative data will come from open-ended questions from the post-test, IDEA survey comments, and reflection journals.
    
Instruments.
To assess level of confidence in performing program evaluation in the real world setting, I’m adapting the Confidence Scale (C-Scale) developed by Grundy (1993), which she used in assessing the confidence of nursing students in performing physical assessment. It has an alpha-Cronbach between 0.84 and 0.95. This C-Scale will be added to the pre and post-test survey. The questions of the C-Scale are as follows, and the answer choices to these questions uses a five-point Likert-type scale (1 = I do not feel confident to 5 = I feel extremely confident):

1. If I were to conduct program evaluation right now, I feel that I will be doing it correctly.
 
2. If I were to conduct program evaluation right now, I feel that I will able to do the tasks without hesitation.
 
3. If I were to conduct program evaluation right now, I feel that my performance would convince an observer who has experience in evaluation that I’m competent.
 
4. If I were to conduct program evaluation right now, I feel sure of myself as I perform the tasks.
 
5. If I were to conduct program evaluation right now, I feel that I will be satisfied with my performance.
 
 
 I will assess whether there was an improvement in learning in four different ways:   
 
First, I will ask the students to write a journal addressing the following questions after each assignment:  
 
DESCRIBE (objectively) the following: (1) When did this experience take place?  Where did it take place? (2) Who else was there? Who wasn't there (but should have been there)? (3) What did you do? What did others do? What actions did you and the agency take? What did you and the agency communicated (i.e., what did you talk about)?
  
EXAMINE your actions: Talk about what were some of your successes, challenges, and compromises in working with the agency in completing your assignment.
  
ARTICULATE LEARNING:
In your reflection, use the following as starting point of your paragraphs on learning: (1) "I learned that..."; (2) "I learned this when..."; (3) "This learning matters because..."; (4) "In light of this I will..."; and (5) "The questions/concerns that I have at this point is/are..."
 
Second,
I will add a series of questions in the pre and post-test survey, asking the students to rate their ability to do the course’s student learning objectives. The questions are as follows, and the answer choices to these questions uses a four-point Likert-type scale (1 = strongly disagree to 4 = strongly agree):
 
 
1.  I feel that I am able to recognize the role and challenges of program evaluator.
2.  I feel that I am able to analyze the compromises often necessary in conducting program evaluation.
3.  I feel that I am able to explain various approaches to program evaluation and their related methodologies, including limitations and constraints to effective program evaluation.
4.  I feel that I am able to integrate theory and community based participatory framework in program planning, implementation, and evaluation.
5.  I feel that I am able to explain the cultural, political, and ethical issues involved in conducting program evaluations and the necessity for protecting human subjects through ethical research procedures.
6.  I feel that I am able to design and prepare an evaluation plan that integrates input from the evaluation team and community partner and its stakeholders.
 
Third, I will also add a series of questions in the pre and post-test survey regarding the student’s ability perform core public health competencies specific for this course. The questions are as follows, and the answer choices to these questions uses a four-point Likert-type scale (1 = strongly disagree to 4 = strongly agree):
 
1.  I am able to monitor health status to identify community health problems.
2.  I am able to diagnose and investigate health problems and hazards in the community.
3.  I am able to mobilize community partnerships to identify and solve health problems.
4.  I am able to develop policies and plans that support individual and community health efforts.
5.  I am able to evaluate effectiveness, accessibility, and quality of personal and population-based health services.
    
Fourth, I will assess the grades that students obtain in all of their assignments, discussion, and final paper.



Course Artifacts

Findings

Qualitative Results: IDEA Survey Student Comments Many students found the class challenging. Moreover, many of them found the value of having a service-learning component in class. The shorter time frame in the summer semester led the class being taught in a faster pace than usual. The class would probably be better taught in the regular 15-week semester and/or with a second follow-up (more advanced) program evaluation course. Overall this class has been challenging compared to other classes taken thus far. I have put a lot more effort on reading and working toward completion of each assignment. I definitely recommend this class on a 15 weeks session because the 10 weeks session is quite hectic. Excellent instructor and really thorough on papers feedback. The service-learning component is a great idea. While I don't have any suggestions at the moment, some thought into modifying that may be worthwhile. Perhaps is it is the compressed summer term, hard to say. It just felt difficult to meet expectations for both the program and class.            The course was challenging, but I learned so much from it!Course, Students, and Community Partners

HS A628: Program Evaluation was taught for 10 weeks in Summer Semester 2013 via distance delivery method, using the Blackboard platform. There were a total of nine students in this class, of which three were males and the rest were females. Four of the students resided in Anchorage, one in Sitka, one resided in Juneau, one resided in Bristol Bay, one resided in Guam, and another resided in Colorado. The students were given an option to work with a partner or to work on their own for their service learning project. Due to different availability, geography, and time zone, most of the students decided to work on their own, while two partnered with each other.

For this class, students were asked to find a community organization or public health agency to partner with in developing an evaluation plan for one of the organization’s/agency’s health-related program. Overall, the students worked with a total of eight different programs from eight different community organizations/agencies; they are as follows:

Safe Kids Alaska: Child Passenger Safety Program
Northern Marianas College: Childcare Services Program
Southcentral Foundation: Lose to Win Program
Alaska Literacy Program: Get Active Program
Catholic Community Services of Juneau: Time Limited Family Reunification Program
SEARHC: Park Prescriptions Program
University of Colorado Health North: Bicycle to Work Program
Providence Alaska Medical Center: Lung Cancer Screening Program

Confidence in Performing Program Evaluation


The C-Scale is a set of five questions, each scored at five points each, giving a total score ranging from 5 to 25 points, with 5 as not confident and 25 being extremely confident in performing program evaluation in the real world setting. Mean confidence score in posttest was higher than pretest (13.0 ± 3.2 S.D. vs. 8.1 ± 3.8 S.D.), however, their difference was not statistically significant (P = 0.06).

   Confidence in Performing Program Evaluation Graph
Figure 1. Pretest and posttest mean confidence score of students with regards to performing program evaluation in the real world setting (N=7 pretest; N=4 posttest). Total score ranges from 5 to 25, with a score of 5 as “not confident” and a score of 25 as “extremely confident.

Student Learning Outcomes

Six student learning outcomes were assessed.Each item were scored at four points each, giving a total score ranging from 4 to 24 points, with a score of 4 to mean the student strongly disagrees the student learning outcomes were achieved and 24 points to mean the student strongly agrees the student learning outcomes were achieved.Mean student learning outcomes score at posttest was significantly higher than pretest (20.1 ± 1.7 S.D. vs. 13.9 ± 2.3, P < 0.001).


 Student Learning Outcomes Graph

Figure 2.  Pretest and post test mean student learning outcomes score of students (N = 7).  Total score ranges from 4 to 24 points, with 4 points being student strongly disagreeing that the student learning outcomes were achieved and 24 points being student strongly agreeing that the student learning outcomes were achieve.Mean scores are significantly different, P < 0.001.


Public Health Core Competencies 

The course addressed five public health core competencies. Each competency was scored at four points each, giving a total score ranging from 4 to 20 points, with 4 points to mean the student strongly disagrees they are competent in the five core competencies and 20 points to mean the student strongly agrees they are competent in the five core competencies. Mean core competency post test score was significantly higher than pretest score (16.1 ± 2.1 S.D. vs. 12.7 ± 1.8 S.D. P < 0.05).

 Public Health Core Competencies Graph

Figure 3.  Pretest and post test mean public health core competency score of students (N = 7). Total score ranges from 4 to 20 points, with 4 points being the student strongly disagrees he/she is competent in five core public health core competency and 20 points being the student strongly agrees he/she is competent in five core public health core competency. Mean scores are significantly different (P < 0.05)

Qualitative Results: Reflection Journals

Based on the students’ reflective journals, the students learned important lessons relevant to program evaluation and/or being an evaluator for each assignment provided. The three common lessons learned by the student throughout the process are as follows:

 

Creating a quality evaluation plan takes a lot of time and effort.

I learned the quite time consuming process to conduct an evaluation program and to have to deal with several pages of information that are needed to be put together in order to write a final evaluation report… This learning matters because of the complexity of being an evaluator or in a position where I may have to represent stakeholders for a program being evaluated.

I learned that program evaluations take a tremendous amount of time and effort, much more than I had put forth in the past and perhaps to the detriment of those programs… This learning matters because I see the actual benefit of a structured and complete program evaluation.

 

      Constant communication with the appropriate person in the program/agency is important. Communication can help clarify important issues in the evaluation, but more importantly it can help improve the relationship between the evaluator and the agency.

 

One of the important lessons I have learned these last two weeks is to be the squeaky wheel. I kept emailing and checking in via email with Mr. Olpin; this diligence at least allowed me the security that I was doing everything possible to be an effective communicator even if I didn’t receive emails on a regular basis. I also learned the face to face meetings might have more merit, as I can directly ask follow up question…

 

I learned persistence and ongoing communication, even if brief, are necessary to maintain a connection. All the information needed may not be readily available, so being resourceful in finding information is an important skill. Also, determining the accuracy and reliability is critical as well in order for the evaluation to be useful. Again, setting realistic time frames is easier said than done. In a more flexible setting, I would have renegotiated deadlines. At this point, having Carin’s input for a final paper would have been very helpful in addressing weakness or making suggestions. It’s clear having consistent input from stakeholders would really improve the evaluation process, and likely make it a more useful tool for everyone.

 

Having a service-learning component in a program evaluation course provides valuable real-world experience.

 

I learned that a program evaluation can require tons of information and different documents that are needed to be sorted out in order to get the type of information you are looking for… This learning matter is particularly important for the student that I am in a way that it gives me rudiments for real world experience on program evaluation and it also teach me the technics, concept and methods needed to conduct program evaluation for the field of health profession.

 

The last several weeks have helped me better understand the inner workings involved in the implementation of the Lung Screening Program at Providence. Taking the assignments from throughout the semester and combining them to create the final evaluation plan has taught me how to better break down large projects into smaller more manageable pieces. Putting together the final evaluation plan during the past few weeks has been a helpful process, because I am able to see how each element is relevant to the other. By combining pieces, I was better able to see interworking of the process, which will be helpful to know for future similar projects. I feel better prepared to ask relevant questions and have a general framework for this evaluation process… The increased experience that I have gained over the last three assignments has allowed me to be better prepared for the meetings with Mr. Olpin as well as the stakeholders. In addition, the experience as allowed me to better express the intent of what was required by assignment four as well as being prepared with any questions Mr. Olpin may have had or I may have needed to ask to get a better picture of updated participation by lung screening participants. It would be very interesting to see if this project can develop into a thesis.


Qualitative Results: IDEA Survey Student Comments

Many students found the class challenging. Moreover, many of them found the value of having a service-learning component in class. The shorter time frame in the summer semester led the class being taught in a faster pace than usual. The class would probably be better taught in the regular 15-week semester and/or with a second follow-up (more advanced) program evaluation course.

     Overall this class has been challenging compared to other classes taken thus far. I have put a lot more effort on reading and working   toward completion of each assignment. I definitely recommend this class on a 15 weeks session because the 10 weeks session is quite hectic. Excellent instructor and really thorough on papers feedback.

     The service-learning component is a great idea. While I don’t have any suggestions at the moment, some thought into modifying that may be worthwhile. Perhaps is it is the compressed summer term, hard to say. It just felt difficult to meet expectations for both the program and class.

     The course was challenging, but I learned so much from it!


Course Artifacts

Reflections

Course Artifacts

Faculty Contact