The objectives of this paper are to describe in detail the steps to be followed in order to plot. Direct assessment chart for Student Learning Outcomes (SLOs) for an academic course following ABET accreditation system, and Student Indirect Assessment (SIA) for the learning outcomes of the same. Assessments to be followed are those defined by ABET as one of sixteen categories named a to k. Two methods are used to achieve the stated objectives: the first method is done using MS Excel spreadsheet program, while the second one relied on a special tool that was developed by the authors to serve the purpose of plotting Radar Charts for SLOs.

Keywords: ABET, student learning outcomes, student indirect assessment, radar chart, accreditation system

Academic courses are prepared and delivered to students under specific learning outcomes and objectives. Outcomes refer to the theoretical or practical knowledge and abilities students attain by the time they are graduated from the course.¹ Objectives refer to potential skills and abilities students will have after graduating from the course.¹ ABET (Accreditation Board for Engineering and Technology) categorizes the student outcomes of a given course into 16 categories, depicted by the letters (a to k):

1. “An ability to apply knowledge of mathematics, science and engineering,
2. An ability to design and conduct experiments, as well as to analyze and interpret data,
3. An ability to design a system, component, or process to meet desired needs within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability,
4. An ability to function on multidisciplinary teams,
5. An ability to identify, formulate, and solve engineering problems,
6. An understanding of professional and ethical responsibility,
7. An ability to communicate effectively (orally, written),
8. The broad education necessary to understand the impact of engineering solutions in a global, economic, environmental, and societal context,
9. A recognition of the need for, and an ability to engage in life–long learning,
10. A knowledge of contemporary issues,
11. An ability to use the techniques, skills, and modern engineering tools necessary for engineering practice”.¹

One important measurement in the academic field is the measurement of the degree to which an academic program has succeeded in delivering its content to the students, and how much did the students achieved with regards to set objectives and outcomes of the specific program? Furthermore, ABET defines different types of assessments including direct, indirect, qualitative and quantitative ones.¹ In this paper the authors described the vital steps needed to prepare direct and indirect student assessment tools in a simple and easy format.

ABET direct assessment

ABET defines assessment as “Assessment is one or more processes that identify, collect, and prepare data to evaluate the attainment of student outcomes and program educational objectives. Effective assessment uses relevant direct, indirect, quantitative, and qualitative measures as appropriate to the objective or outcome being measured. Appropriate sampling methods may be used as part of an assessment process.”^2,3 ABET elaborates more on direct assessments by stating that “Direct assessments provide for the direct examination or observation of student knowledge or skills against measurable learning outcomes. Direct assessments of student learning could be conducted throughout a course using such techniques as exams, quizzes, demonstrations, simulations, portfolio, and reports. These techniques provide a sampling of what students know and/or can do. Besides they provide strong evidence of student learning”.⁴ Examples of direct assessment methods include comprehensive examinations; commercial, and standardized designs; certification or licensure examinations; course assignments; research papers; capstone or senior design project evaluations; internship portfolio valuations; reflective journal appraisals; writing proficiency examinations; oral presentation judgments with criteria or rubrics; case study review and national standard tests; etc.^5–7

ABET indirect assessment

ABET defines indirect assessment as “Indirect assessments of student learning ascertain the perceived extent or value of learning experiences. They assess opinions or thoughts about student knowledge or skills. Indirect measures can provide information about student perception of their learning and how this learning is valued by different constituencies. An indirect assessment is useful in that it can be used to measure certain implicit qualities of student learning, such as values, perceptions, and attitudes, from a variety of perspectives. However, in the absence of direct evidence, assumptions must be made about how well perceptions match the reality of actual achievement of student learning”.⁴ Examples of indirect assessment methods include archival data, interviews, questionnaires, surveys, internal/external peer reviews of performance; exit interviews/surveys; alumni surveys; employer surveys; student satisfaction surveys; student course evaluations; graduation and retention rates; job placement of graduates; focus group discussions; graduate school acceptance rates; alumni honors, awards, and achievements; etc.^5,8,9,10 More information may be obtained from the “ABET assessment webinar series” prepared by ABET (ABET 2017c).⁷

The objective of this paper is be to delineate the steps to follow in order to plot direct assessment chart for Student Learning Outcomes (SLOs) for an academic course following ABET accreditation system. Another chart for Student Indirect Assessment (SIA) for the learning outcomes will also be shown.

In the following sections, the steps to plot SLO chart will be shown using two methods. The first method will be using Microsoft Excel (MS Excel) to draw the charts. The second method will use a tool that was specifically programmed by the authors to achieve the same results. For purposes of demonstrating the steps involved in calculations, a sample class of (5) students will be used throughout the discussion.

First method (MS Excel plot)

In the first method, the assessor will be using MS Excel spreadsheet software to accomplish the task of plotting a radar chart for the direct assessment of students SLOs.^11,12

Task: Draw Student Learning Outcomes (SLOs) radar chart for the academic course “Library skills” (course code: HUMN 203), offered at the Environmental Engineering Dept. of the College of Engineering at Imam Abdulrahman Bin Faisal University, given the following information:

1. Overall grade records (Table 1).
2. Record of activities (Table 2).
3. Records of mid–term examination results (Table 3).
4. Records of final examination results (Table 4).

Detailed solution for task: The program must have documented student outcomes that prepare graduates to attain the program educational objectives. Student outcomes are outcomes (a) through (k) plus any additional outcomes that may be articulated by the program (ABET).

Step 1: Formulate course ABET outcomes.

As an example, select ABET Student Outcomes (SO) for the academic course in “Library skills, HUMN 203” to include the three (3) selections out of ABET (a) through (k) outcomes i.e. (d), (f) and (i).

Step 2: Prepare needed data to plot chart (include all marks and results attained by all students for different identified assessment components. In the case under consideration, the data gathered contained: quizzes, assignments and activities, plus mid and final term examinations):

1. Record quiz results (Quiz number, students enrolled and related marks in any exam question as related to its affiliated (a) through (k) outcomes) (Table 1). In this case, two (2) quizzes were offered and ten (10) marks were allocated for the two (2) of them (five (5) marks for each) out of the course total as presented in Table 5.
2. Note results for “assignments and activities” as presented in Table 2. Assignments ought to be distributed as per their actual weights. For instance, in this worked example twenty (20) activities were introduced to students with outcomes of (d), (f) and (i) of (13), (2) and (5) for each corresponding activity, respectively (This gives a ratio distribution of 65, 10 and 25 percent between their 30 allocated marks) (Table 6). Total marks reserved for assignments are thirty (30) marks out of hundred (100) for the course. Distribution of marks for students will be as presented in Table 7.
3. Register midterm examination results for each student as given in Table 3. In this case, twenty (20) marks were reserved for this component. Number of questions given in the midterm exam was five (5) questions addressing items (f) and (i) only with ratios as depicted in Table 8. The adjusted midterm examination results are shown in Table 9.
4. Final examination results are shown in Table 4. In this case forty (40) marks were reserved for this issue. Four (4) questions portrayed the exam addressing (i) and (f) outcomes in ratios given in Table 10. Then final examination adjusted results are as given in Table 11.

Step (3): An assessment tool need to be assumed and selected as depicted in Table 12. In this case, the ratios selected for course ingredients were adhered–to in formulating the SLOs assessment tool.

Step (4): Find weighted actual marks for all students as indicated in Table 13. Average for each category (d, f and i) should be calculated for the whole class of students. Total ought to be related to the assessment tool of Table 12.

Step (5): Prepare results from Table 13 as shown in Table 14.

Step (6): Plot the radar chart by selecting “Insert” from MS Excel menu. Then select “Radar” from Charts → Other Charts. The inserted Radar chart is shown in Figure 1.

Figure 1 Sample course SLOs radar chart in MS Excel.

Step (7): Manually connect the dots and add to the plot the triangle (d f i) of LOs on top of the radar web as presented in Figure 2.

Figure 2 Final SLOs radar chart for sample course.

Second method (Abdel–Magid RadarChart Creator Tool)

In the second method, you will be using a tool that the authors have prepared specifically for this task. The tool is called “Radar Chart Creator Tool” and can be obtained by contacting the author (Dr. Mohammed Isam Mohammed Abdel–Magid) through his email. Task: Draw Student Learning Outcomes (SLOs) radar chart for the academic course “Library skills” (course code: HUMN 203), offered at the Environmental Engineering Dept. of the College of Engineering at Imam Abdulrahman Bin Faisal University, given the information detailed in Tables (1–4).

Detailed solution for task

After obtaining the tool executable file, double click on it in Windows Explorer to run the tool. Afterwards, follow these steps in order to accomplish the task at hand:

Step (1): The first screen of the program is shown in Figure 3. Fill in the fields with the course data, as shown in Figure 4.

Figure 3 First screen of the Radar Chart Creator Tool.

Figure 4 First screen of the Radar Chart Creator Tool with fields filled out.

Step (2): Click on the “ABET Learning Outcomes” tab (at the top of the window, just below the menu). Tick on the selected ABET SLOs, in this case they are (d), (f) and (i), as shown in Figure 5.

Figure 5 The ABET Learning Outcomes Tab in the Radar Chart Creator Tool.

Step (3): Click on the “Students List” tab (see Figure 6). There are two options to load the names and IDs of the students to the tool:

1. Create an MS Excel sheet that contains only two columns: the first for student IDs, and the second for student names. Save the sheet somewhere on your computer. Return to the tool, and click on the “Browse…” button. This will open a dialog box in which you can browse and select the Excel workbook you just created.
2. Copy the IDs and names from MS Excel (or any other program) by highlighting the needed range and then pressing CTRL+C (or selecting Copy from the Ribbon, or right–mouse clicking and selecting Copy). Click on the first cell to the left in the “Students List” tab (immediately under the “ID” header), then paste the copied IDs and names by pressing CTRL+V.

Figure 6 The Students List Tab in the Radar Chart Creator Tool.

At any time, if any mistake is made the students list can be reloaded. Click on the “Clear student list” button. This will erase whatever names and IDs are loaded and start with a clean, empty sheet.

Step (4): Click on the “Quizzes” tab. Fill in the data required for each quiz in turn. This data includes the quiz’s total mark, the mark of each question and the ABET SLO for each question in each quiz. The quizzes can be navigated by using the buttons marked with (<<) and (>>) at the top–right of the “Quizzes” tab see Figure 7 & 8.

Figure 7 The Quizzes Tab in the Radar Chart Creator Tool.

Figure 8 The Assignments Tab in the Radar Chart Creator Tool.

Step (5): Click on the “Assignments” tab. For each assignment, enter the assignment mark and ABET SLO (see Figure 8).

Step (6): Click on the “Midterm” tab. For each question in the midterm, enter the respective mark and ABET SLO (see Figure 9).

Figure 9 The Midterm Tab in the Radar Chart Creator Tool.

Step (7): Click on the “Final Exam” tab. For each question in the final exam, enter the respective mark and ABET SLO (see Figure 10).

Figure 10 The Final Exam Tab in the Radar Chart Creator Tool.

Step (8): Click on the “Enter data” button, at the bottom–right corner of the window. This will open the Data Entry window (see Figure 11). The first two columns contain students IDs and names that were entered in the first window. They can’t be changed in the Data Entry window. If the students list needs to be changed, close this window and go back to the main window and open the “Students List” tab as shown in Step (3) above.

Figure 11 The Data Entry Window in the Radar Chart Creator Tool.

Fill in the data for the quizzes, assignments, midterm and final exams. If the students grades are in an Excel sheet (or any other program), one can copy the grades from the original file and paste them on the tool using CTRL+V. If there are students who missed some of the assessments (e.g. attended quizzes but missed midterm or final exam), those students will have their missed results zeroed out automatically by the tool, and they will be included in the final count, which would be used to count overall class averages. If this is not the desired behavior, the tool can be instructed to include/exclude students from the final calculation step by clicking on the “Change attendance list” button that is found in the bottom–left corner of the Data Entry window. This will open a small dialog box with the list of all students in the class, each with a check box next to it. Students with ticks inside the checkboxes are considered “attending” and will be included in the final calculations. Students with their checkboxes clear (no tick beside their name) are considered “absent” and will not be included in the calculations.

Step (9): Click on the “Calculate” button, at the bottom–right corner of the Data Entry window. This will open the Radar Chart Plot window. In this window, the adjusted students results that correspond to (Tables 5,7,9,11) are shown, each in its own tab page. The last tab page (to the right), is titled “Radar Chart”. Click on it to open the tab, as shown in Figure 12.

Figure 12 The Radar Chart Plot Window in the Radar Chart Creator Tool.

Student indirect assessment (sia) radar chart for student learning outcomes (slos)

Similar to Section (3) above, in the following sections, the steps to plot SIA chart will be shown using two methods. The first method will be using Microsoft Excel (MS Excel) to draw the charts. The second method will use a tool that was specifically programmed by the authors to achieve the same results. Again, a sample class of 5 students will be used throughout the discussion.

First method (MS Excel plot)

In the first method, MS Excel spreadsheet software will be used to accomplish the task of plotting a radar chart for the indirect assessment of students SLOs.

Task: Draw Student Indirect Assessment (SIA) radar chart for the academic course “Library skills” (course code: HUMN 203) offered at Environmental Engineering Dept. of the College of Engineering at Imam Abdulrahman Bin Faisal University, given the questionnaires filled by the students.

Indirect Student Outcomes Survey (Table 15)

(Library skills, HUMN 203)

Imam Abdulrahman Bin Faisal University – Faculty of Engineering, Environmental Engineering Department

Indirect Student Assessment Survey^4,9,8,12 Based on your course work experience, including inside and outside class activities, at what level would you rate the following?

1. (Use the attached rubric to help you select your rating level. Put a mark in the column to the right that best represents your skill level for each item).
2. The scale of marks is as follows:

5 = Excellent – highest mark

4 = Very good

3 = Good/Intermediate

2 = Poor

1 = Very poor–lowest mark

Detailed solution for task comment: A similar radar chart to the aforementioned one may be drawn for Student Indirect Assessment, SIA.^13,14

1. Table 16 gives an illustrative example for an SIA ratios for ABET (d), (f) and (i) as drawn from questionnaire results.
2. An assessment tool need to be assumed and selected as depicted in Table 17. In this case values given to each question within the questionnaire are used.
3. Find actual percentages for all students as indicated in Table 18. Average for each category (d, f, and i) should be calculated for the whole class of students. Total ought to be related to the assessment tool of Table 16.
4. Table 19 may be generated from Table 18 as related to the SIA tool for the purpose. These results may then be used for plotting the radar chart.
5. Plot the radar chart by selecting “insert” from the ribbon in MS Excel. Then select “radar” under “Other Charts”. Radar chart is shown in Figure 13.

Figure 13 Radar chart for SIA of a sample class in MS Excel.

Second method (Abdel–Magid RadarChart SIA Creator Tool)

In the second method, a tool that the authors have prepared specifically for this task will be used. The tool is called “Radar Chart SIA Creator Tool”.

Task: Draw Student Indirect Assessment (SIA) radar chart for the academic course “Library skills” (course code: HUMN 203) offered at Environmental Engineering Dept. of the College of Engineering at Imam Abdulrahman Bin Faisal University, given the questionnaires filled by the students as in Task (6.1).

Detailed solution for Task: After obtaining the tool executable file, double click on it in Windows Explorer to run the tool. Afterwards, follow these steps in order to accomplish the task at hand:

Step (1): The first screen of the program is shown in Figure 14. Fill in the fields with the course data, as shown in Figure 15.

Figure 14 First screen of the Radar Chart SIA Creator Tool.

Figure 15 First screen of the RadarChart SIA Creator Tool with fields filled out.

Step (2): Click on the “(d)” tab (at the top of the window, just below the menu). You can fill in the data by hand, or if you have the results in an Excel sheet (or any other program) you can copy them by highlighting the needed range and then pressing CTRL+C (or selecting Copy from the Ribbon, or right–mouse clicking and selecting Copy). Click on the second cell to the left in the first row (immediately under the “5” header), then paste the copied IDs and names by pressing CTRL+V. as shown in Figure 16.

Figure 16 The first SIA Tab in the Radar Chart SIA Creator Tool.

Step (3): Repeat the same process done in Step (2), this time filling out the fields for (f) and (i) SIA learning outcomes.

Step (4): Click on the “Plot chart” button, at the bottom–right corner of the window. This will open the Radar Chart Plot window. In this window, the only tab page is titled “Radar Chart”. It shows the radar chart, as depicted in Figure 17.

Figure 17 The Radar Chart Plot Window in the Radar Chart SIA Creator Tool.

This paper has discussed two different ways of plotting both direct and indirect assessment chart for Student Learning Outcomes (SLOs) for an academic course following ABET accreditation system. A sample class of 5 students was used, with ABET LO’s of (d), (f) and (i) included in the calculations. For the SLO plot using the same method, the averages for the class were 92.8, 85.49 and 80.2 for (d), (f) and (i) LO’s, respectively. The averages calculated using the second method was 92.08, 85.49 and 80.20, respectively. For the SIA plot using the first method, the averages for the class were 82.42, 83.04 and 83.11, for (d), (f), and (i) LO’s, respectively. The averages calculated using the second methods were 82.42, 82.94, and 83.11, respectively. We conclude that both methods generate similar results and are comparable when assessing academic course objectives and outcomes as delineated by ABET.

None.

Author declares that there is none of the conflicts.