Pitt | Swanson Engineering
Outcomes and Assessment

The department has an active undergraduate committee that incorporates ABET accreditation needs into its normal operations. At present, the department utilizes various assessment tools to evaluate how program and class objectives are realized. These methods include: student and alumni surveys, peer and self-evaluations, assessment of senior design project, and compilations of students' work within courses.
In evaluating the changes in the proposed curriculum, these methods will continue to be employed, but will be coupled with additional methods to ensure that the objectives of the current proposal are also being realized. Specifically, under the direction of Dr. Besterfield-Sacre from Industrial Engineering, we will employ a series of cohort studies that takes advantage of the roll-out plan, so that we can investigate students in the current curriculum and as additional cohorts become more and more exposed to the pillar experience. To evaluate each proposed pedagogical objective and implementation strategy, we intend to employ at least three methods. The triangulation of these methods include not only third party assessments (e.g. board of visitors) and student self-assessments (e.g. employed by the School of Engineering), but also "graded evidence" of students' abilities (e.g. performance on pillar concept inventories, scores on concept maps). By using this approach, we meet not only on the goals of the proposed curriculum, but also maintain the current program objectives. The table below provides a summary of the evaluation plan. For each pedagogical objective, the corresponding program objective(s) is provided along with the proposed implementation strategy(s). Further, for the objectives and strategies provided, an overarching evaluation is provided with the proposed assessment strategies. Descriptions for a selection of the methods to be employed follows.

Table: Summary of the Evaluation Plan.

Pedagogical Objective (ChE ABET objective in italics) Proposed Implementation Strategy Evaluation Questions and Strategy
Give Students a strong fundamental foundation by concentrating on the essential core of scientific and engineering basics in a given discipline

-> Provide students with a broad knowledge of the principles of ChE and their applications

-> Provide students with the knowledge and skills required to design and analyze chemical processes, taking into account health, safety, environmental, and societal impacts -> Modularize and integrate the critical "core" of Chemical Engineering into six Pillar Courses

-> Re-evaluate and re-align supporting courses and elective courses Do students have a more comprehensive understanding of their Chemical Engineering core and related courses?
-> Develop (where necessary) and apply concept inventories to traditional and pillar cohorts; investigate anticipated gains in the core areas as a result of the Pillar courses. For those Pillars that are not conducive concept inventories (i.e. design) rubrics will be developed.

-> Map current curriculum summary table evaluations of groups to new pillar courses; investigate probable gains in the number of "goods" for improving students' skills in a particular engineering criteria related to the core, supporting courses, and technical electives.

-> Develop rubrics and conduct external review by ChE board of visitors of students' work in both pillar and complementary courses.
Enhance systems thinking by helping students to integrate their knowledge across courses so that they are better prepared to address open-ended problems

-> Provide students with the knowledge and skills required to design and analyze chemical processes, taking into account health, safety, environmental, and societal impacts

-> Provide students with the skills necessary to perform in the multi-disciplinary environment of the 21st century -> Re-evaluate and re-align supporting courses (mathematics, chemistry, physics, and biology) and elective courses.

-> Re-align each individual course so that, in addition to the traditional macroscopic and continuum level descriptions, we also include "sub-continuum", molecular or "fundamental unit" analysis. Do students have enhanced knowledge integration and organization of their discipline both in terms of integration across the core and within the individual pillars?

-> Apply the use of concept maps at various levels of students' undergraduate studies and within pillars; investigate anticipated gains in comprehension, organization, and correctness.

-> Continue to employ the all-level survey summary grades for the ChE program; investigate grade gains in a-k outcomes.

-> Conduct alumni surveys; investigate anticipated changes in alumni's opinions with respect to: (1) more "real world" examples, (2) computer skills and software exposure, (3) writing experience, (4) positive faculty-student communications, and (5) exposure to state-of-art technologies.
Prepare and provide for continuing education and life-long learning

-> Provide students with the skills necessary to perform in the multi-disciplinary environment of the 21st century -> Make certain that students are well exposed to topical "tracks" that address both traditional and emerging ChE markets. Does the pillar approach to the curriculum give students a better awareness of emphasis areas and opportunities?

-> Investigate students' reasons for track-hoping through student records and focus groups.

-> Track (through student records) students and alumni tendency to (not) track hop to determine if a relationship exists with continuing education.

-> Conduct alumni surveys; investigate anticipated changes in alumni's opinions with respect to advising and career selection.
• Concept Inventories. Because a major objective of the pillar structure is to give students a strong foundation of the core basics, we intend to measure that students, indeed, understand the fundamentals. The development of concept inventories can play an important part in relating teaching techniques to student learning. NSF has recently funded several concept inventory studies in engineering subjects, one of which focuses on Thermal and Transport Processes - two of the pillars. We have secured with the authors Drs. Streveler and Miller at the Colorado School of Mines to pilot and use this CI. In addition, Drs. Miller and Besterfield-Sacre intend to develop CI's for Reactive Processes and Fundamentals.
• Concept Maps. Concept maps have been extensively used in literature to measure knowledge integration. In a previous study, Dr. Besterfield-Sacre and colleagues have developed a rubric to expeditiously "score" maps on elements of comprehension, organization, and correctness. These elements help to determine if students have properly integrated their knowledge across courses and within courses - one of our pedagogical objectives.
• School of Engineering Surveys. Drs. Besterfield-Sacre, Shuman and Wolfe have developed a number of survey instruments and reporting mechanisms for the School of Engineering to facilitate assessment on a college wide basis. At the School of Engineering level, each department receives two reports annually: the "Curriculum Summary Table" and "All-Level Survey Summary Grades". These reports provide an overview of the outcomes with respect to student course evaluations and annual attitude surveys at all levels, respectively. A third report, "Comparisons of Alumni Reactions" is conducted and analyzed on a bi-annual basis.

For more information on previous uses of these assessment techniques please refer to:
D. L. Evans et al. Progress on concept inventory assessment tools. Frontiers in Education, 2003.
J. Gerchak, M. Besterfield-Sacre, L.J. Shuman, and H. Wolfe. Using concept maps for evaluating program objectives. Frontiers in Education, 2003.
J. Gerchak, M.R. Lyons M. Besterfield-Sacre, L. Shuman, and H. Wolfe. Creating an integrated rubric for scoring concept maps in an engineering program. working paper, 2003.
M. Besterfield-Sacre, M. Kolar, L.J. Shuman, and H. Wolfe. Beyond accreditation: How to sustain the use of assessment in quality educational processes. ABET Annual Conference, Pittsburgh, PA, 2002.
J. McGourty, L.J. Shuman, M. Besterfield-Sacre, and H. Wolfe. Improving academic programs by capitalizing on alumni's perceptions and experiences. Frontiers in Education, San Juan PR, 1999.