2b.1. What are the processes and timelines used by the unit to collect, compile, aggregate, summarize, and analyze data on candidate performance, unit operations, and program quality?
COE has made slow but significant progress in establishing a system of data collection that provides useful information to assess candidate performance, program quality, and unit operations. Policies and procedures for data collection, analysis, and evaluation are delineated in the Assessment Handbook (Std2b.Exh6). COE’s limited ability to access more sophisticated information technologies, or to take full advantage of UA’s existing technologies for achieving unit goals, has been a factor, but COE continues making progress as central administration grows in its understanding of the College’s data needs. Also, faculty have grown in their understanding of the value of data collection to inform practice in the various departments and unit-wide, thereby increasing their participation in the College’s efforts.
Data are collected in a variety of ways depending on the type of data. Transition point data are collected using the institution’s Banner database and OnBase for final decisions on applications for admission and applications for graduation. Departments maintain records for other program transition points. Exit, alumni, and employer survey data are collected annually using Zoomerang through email invitations using subject contact information generated by Banner, the institution’s Advancement database, and State of Alaska Department of Education and Early Development employment information.
Prior to AY10, the TaskStream portfolio system collected course-based key assessments for most initial programs. For other programs, the institution’s Blackboard course management system collected evaluation scores on course-based assessments. Effective AY10, all programs use TaskStream. Final course grades are collected via UAOnline, a component of the Banner database. Test score data, such as Praxis I and Praxis II, are transmitted from the Educational Testing Service to the institution electronically and on hardcopy score reports and entered into unit databases and Banner by unit and institutional staff.
COE Clinical Experiences and Certification Office staff enter information on candidate internship placements into a database for all Department of Teaching and Learning initial programs. Department staff enter information into databases for other initial and advanced programs in the Department of Educational Leadership and the Department of Counseling and Special Education. Prior to AY10 instructors and clinical faculty entered internship evaluations in TaskStream for most initial programs and into department databases for other initial and advanced programs. In AY10, all programs use TaskStream. Staff in the Clinical Experiences and Certification Office enter institutional recommendations issued by COE in a database.
Various data are collected on the unit’s file sharing system. Department staff collect course syllabi each semester. These syllabi are also maintained in PETaL. The Associate Dean’s Office collects and maintains Course Content Guides whenever new courses are created or existing courses are revised. These guides are also maintained centrally on UAA’s Curriculum website. Dean’s Office staff collect Workload Agreements, Annual Activity Reports, and Curriculum Vitae on an annual basis. Effective Fall 2009, CVs are entered in PETaL. Student record document images are collected by department staff on an ongoing basis. Unit staff create and maintain meeting documentation on an ongoing basis.
COE’s assessment system data are collected from a variety of sources including program applicants, candidates, alumni, employers of graduates, instructors, clinical faculty, school-based mentors, school districts and other stakeholders, the State of Alaska Department of Education and Early Development, the institution’s Advancement Office, and the institution’s Enrollment Services Office (including Admissions, Registrar, and Degree Services staff). These data are analyzed and summarized on an annual basis in accordance with the timelines of the unit’s assessment system. The responsibility for summarizing and analyzing data rests with the Leadership Team, which consists of the Dean, Associate Dean, department chairs and directors, Data Manager, and Fiscal Manager. The team disseminates data to appropriate faculty and unit governance committees and boards for consideration and possible actions for operations and program improvement.
The Data Manager provides survey data, disaggregated by program area, to the Leadership Team and the Assessment and Accreditation Committee. Program area results are passed to program faculty for discussion and action. The Data Manager also prepares reports on test score data for dissemination to advisory boards and councils, departments, Arts and Sciences, and Title II. Similar reports are generated on enrollment numbers, degrees and certificates awarded, institutional recommendations, and candidate diversity. These data are shared with internal and external stakeholders. Faculty report workload productivity data to department chairs, the Dean, and central administration through Annual Activity Reports.
The unit uses a variety of information technologies to maintain the assessment system (Std2b.Exh10), including the Banner database, OnBase document imaging system, IDEA on-line instructor and dean evaluation system, TaskStream, the Blackboard course management system, Excel-based UAA Outcomes Assessment Report system, Excel-based department databases, the unit’s file sharing system, and the unit’s PETaL system.
2b.2 How does the unit disaggregate candidate assessment data for candidates on the main campus, at off-campus sites, in distance learning programs, and in alternate route programs?
COE determines if candidate assessment data must be disaggregated by identifying those programs that, in addition to serving the main campus with face-to-face courses in regular routes to certification, serve a significant population of candidates whose courses are delivered at off-campus sites or via distance delivery, or whose candidates opt to choose an alternate route.
If 5 or more candidates earned more than 50% of credits in their program’s major requirements at a community campus site, then the program must disaggregate the assessment data for them by first identifying the community campus candidates using Banner, then using TaskStream to generate a disaggregated report. The program coordinator presents the report to the A&A Committee. Currently, the BA in Elementary Education is the only program to which this applies.
In AY09, COE had 7 programs delivered via distance: MAT, M.Ed. in Special Education, Graduate Certificate in Special Education, M.Ed. in Early Childhood Special Education, M.Ed. in Educational Leadership, and two Post-Graduate Certificates in Educational Leadership (principal and superintendent). The MAT is the only program that offers at least 50% of the courses (excluding internships) both by distance and face-to-face. Therefore, COE does not disaggregate distance/hybrid-delivered candidate assessment data from onsite delivered candidate assessment data except for MAT candidates. Elementary Education often provides candidates the opportunity to take some Education course sections by distance or face-to-face to facilitate program completion for those who live in the community campus areas. However, this program does not fit NCATE’s definition of a distance program, so candidate data are not disaggregated.
If 5 or more candidates enroll in an alternative route to initial teacher certification, then the program is required to disaggregate the assessment data for those candidates. This is done by identifying the alternative route candidates using PETaL and then using TaskStream to generate a disaggregated report for them. The program coordinator presents the report to the A&A Committee. The Department of Teaching and Learning currently offers an alternative route to initial teaching certification to secondary (MAT) candidates in high-need endorsement areas. The Department of Counseling and Special Education recently admitted candidates to an alternative route in special education through the initial teacher certification option of the Graduate Certificate in Special Education. There are no program graduates to date, and data from key assessments are insufficient to disaggregate since only 2 or fewer candidates have completed any one assessment. When sufficient data exist, the program area will be expected to examine the transition point and key assessment data to ensure that the alternative route candidates perform as well as the candidates enrolled in the non-alternative option.
Summaries of the unit’s disaggregated data are displayed in Standard 2B Exhibit 7 Sample of candidate assessment data disaggregated by alternate route, off-campus, and distance learning programs.
2b.3. How does the unit maintain records of formal candidate complaints and their resolutions?
In Chapter 5 of the UAA Catalog, candidate rights, freedoms, and responsibilities are outlined. Candidates have a variety of procedures available to them if they have been adversely affected by the actions or inactions of members of the University community. COE handles formal candidate complaints in accordance with the University of Alaska Board of Regents Policy, Part IX – Student Affairs, Chapter 09.03 - Student Dispute Resolution. The procedures are outlined in the UAA Fact Finder Student Handbook, Student Dispute/Complaint Resolution Process, and the College of Education Handbook, Appendix D - Academic Dispute Resolution Procedures (Std2b.Exh8).
The COE Associate Dean for Student and Curriculum Affairs manages all formal candidate complaints about academic matters. As appropriate, the Associate Dean will notify the UAA Dean of Students or other UAA officials about formal complaints that may need to be reviewed by entities other than the unit. Records are maintained in a secured file on the COE’s shared drive (Std2b.Exh9). These files are accessible to only the Dean, Associate Dean, and the Assistant to the Associate Dean.