TA Experience Reflection: A Conversation about Course Improvement Suggestions

The Course: 15150 Functional Programming

In my junior year, I became a teaching assistant (TA) of the introductory functional programming (FP) course at CMU 15–150: Functional Programming (150). The Spring 2021 instance of 150 has 2 instructors, 39 TAs, and 335 students with limited in-person lecture and lab options. This course has the following learning objectives on its website:

The Origin: A Piazza Conversation

In the middle of the semester (at the end of March), we received a long Piazza post concerning & suggesting changes about Gradescope autograder as the student lost more than half of the points in programming homework albeit a close answer, because of a failure to thoroughly test their code before submission. To quote¹ from the student:

I understand that this is a failure on my part for not having sufficient tests, but the autograder would give someone the same score I got for only putting the empty list and edge cases. I would much rather the autograder test the proportion of our output which is correct for problems like these, rather than simply giving you a 0 if one part of your output is incorrect. It seems like it’s unfair for students who put in a lot of effort in solving the problem and came really close to doing so to give them no points whatsoever in comparison to a student who put little to no effort.

This piazza post received a lot of attention and follow-up conversations between 150 students and teaching staff.

Students’ Words

Many students were frustrated, confused, and disheartening because they feel like their hard work devoted to the homework wasn’t being fairly assessed, for example:

Agreed!! It’s not fair for students to put in 95% of the work and only earn 20% of the points.

… I think that the mentality of “we are training you to be perfect” is far and away the most concerning part of any of this…At best, you’re training students to be anxious about the correctness of their code. CMU requires that you include a statement of student wellness in the syllabus for this course… This post and the follow-ups are students asking for you to really embody that message and support their mental health. Having an “all or nothing” mentality when it comes to perfection and satisfactory work is a symptom of almost every form of anxiety. I hope we can all agree that that level of scrutiny is not in any student’s best interest. All anyone here is asking for is a grading system that is more conducive to your student’s having some semblance of good mental health.

Some students also shared their experiences to encourage their peers,

All in all, don’t let these several/many points on these assignments define you. I let it define me and I let it ruin my morals and ultimately I got the worst possible grade…My biggest regret was not having faith in myself and believing that I was the only one who was failing. We are all students. We all want to be the best and we all want to try our hardest. Obviously it sucks when our effort isn’t being translated into good results. But this will be the case anywhere and anytime in life as well… In the moment, it may be extremely taxing on your mental health and I understand that, but use me as an example of someone who obsessed over grades too much and ultimately failed even harder.

Just to make you feel better, I got nearly half of the points of an assignment in (another class) because I forgot to output my answer with the first character capitalized.

Students had also tried to propose concrete solutions, which are further elaborated below.

Teaching Staff’s Words

This post and the threads of students’ feedback indeed need to be addressed, so various teaching staff responded by

clarifying learning goals and appealing to the ethos of education for life,

… A goal of this course is to get everyone to mentally prove correctness of their code. We used to not even allow sharing of test cases, since that is part of the thought process involved in coding. If a plane crashes and hundreds of people die, it does not matter that 100000 other flights were successful. That is the world you will be coding in.

… When I’ve had these kinds of things happen, I felt disappointed as well, but learned from my mistakes as I aimed to not repeat them again. Consequently I became a much stronger programmer. Without setbacks and struggle, we can’t really improve. Although it hurts now, I hope you all learn from this mistake and become stronger from it. In this scope of this course and your entire life, this is only a minor obstacle.

reassuring the students it won’t influence their grade as much

… Even 16 points, which might seem like a sizable chunk of your grade on this particular assignment, in the end, this might affect your grade by around half a percentage point, but even then it would not make a big difference unless this is a recurring issue, in which case you may want to consider writing more rigorous test cases. In this case, you could have caught your error by running one of the test case provided in the writeup, so you would not even have had to come up with your own test cases to have caught this error.

answering the different things 150 tried but failed in the past or are not quite feasible given the technologies

I understand your frustration, and these partial credit cases are something that we’ve been working on for many semesters. As mentioned in other replies, it’s very difficult for an autograder to be able to differentiate between a small error/typo in the code versus a serious misunderstanding. And manual grading is simply unfeasible. While we’d like to reply to all of the regrade requests and give back points in reasonable cases such as yours, we have to maintain fairness and short of manually regrading every submission, there’s no way to ensure that we give points equally in all cases.

My Message: Course Improvement Feedback

I also wrote a lengthy message in our teaching staff’s internal communication Slack channel, and below is my original feedback:

As someone coming from a psychology/learning science background, I’ll offer some ideas that I learned from the higher-level education class I took. Understanding others’ perspectives is a whole field of research so it’s definitely hard, but as teaching staff, we all wish the students to learn and thrive, to achieve this, we need to clearly communicate it to students. Although it’s sad to receive harshly-worded negative feedback, it’s actually good to have students come out and complain, so it helps expose problems and we can keep improving students’ educational experience in the future.

To clarify our teaching philosophy & learning objective early in the course

On the current website, the learning objectives are mostly about conceptual knowledge; however, we do expect students to learn a lot of other skills, so we should make them clear as well. To paraphrase some ideas that I saw in the discussion, students should learn

1. Responsible and transferable programming practices such as writing provably correct code, or as a heuristics, at least extensively test their code before submission.
2. Metacognitive skills of monitoring their learning, reflecting on failure, adjusting learning styles and mindset, seeking help, and don’t give up.

To align our teaching & grading rubrics with learning objectives

1. Probably the prerequisite courses didn’t prepare students to write provably correct code or extensive and rigorous test cases, so to encourage and assess students’ learning on these responsible programming practices, we should not only teach it by providing “worked examples” of test cases in lectures and labs but also grade it explicitly.
   For example, if it’s technically feasible and we have some buggy implementation at hand, we might run students’ test cases against the buggy code to see how many errors can the students’ test cases catch, and provide some potential bonus +X pts for comprehensive test cases (low enough to prevent coding auto test case generator but something as an incentive).
2. We shouldn’t expect novices to be able to understand whether they learned the materials or reflect on their own failure maturely and positively, it needs to be taught & scaffolded. Students come from different cultural and social backgrounds, so we should provide support for the shift of mindset.
   Various TAs have made a ton of good study suggestions here and there, but we need to integrate these suggestions into our objectives & assessment system so that students can actually learn these abilities.
   To support self-reflection, I do agree on part of the student’s suggestion, e.g., if a student loses ≥ X% pts in some tasks after grading, they can resubmit that problem within a window of Y days, and if their code passes the autograder they got Z% back.
   If this automatic point adjustment is technically difficult, we can ask the students to come to OH and show the TA their reflection on the error and the corrected solution like how they make up for labs. If X% is 50%, I don’t expect a lot of students to lose more than half the points in tasks so it’s really trying to incentivize the process of reflecting on error.

To let students know how they are assessed and why

Students are frustrated because their expectations and outcomes don’t align, which is because they can’t accurately predict their performance and can’t evaluate their own learning. This is again the metacognitive ability that we need to teach.
We may provide students with the rubric beforehand and even provide some opportunities for students to self-grade their own responses. This is less true for auto-graded programming tasks, but we should still explicitly let students know how they will be assessed and why:

1. How the assessment aligns with the learning objectives as abovementioned
   We really need to tell a consistent story in our assessment design and our teaching philosophy. For example, if we want the students to compare against their previous selves rather than others, nonetheless we grade them not based on their own progress & effort but in comparison to other students, it will be a conflicting story. Students will unavoidably ask about the statistics, without which they don’t know how to assess their position in the class and assess their own learning. So I do believe that some general statistics like median, number of hours spent, or even just some anecdote like “in past semesters you can get 70 on midterms but still get an A” will be helpful.
2. How the assessment is operationalized given the practical constraints of time, fairness consideration, and technologies. E.g.,

• autograder can’t differentiate between a small error vs. a serious misunderstanding
• manual grading is unfeasible
• we choose to expose only 1 failed test case out of a bucket because students need to learn to write provable code & generate test cases themselves,
• we choose to not reveal average statistics because it may discourage those who did not do well, etc.

Part of how well a student learns in the class will be implicitly assessed on how they respond to errors. So besides trying to integrate it into our assessment, we can also put some reflective prompts as bonus points in our first homework (the one with the AIV/syllabus reading questions).

To acknowledge & pre-address anxiety

• Acknowledge that many students consider 150 as a hard class, but provide them with some reference statistics or anecdotes.
• We can also make a post or example of “what does -20 pts on assignment or midterm mean to your grade” somewhere in the class. Walk them through the calculation and show it’s what percentage of their grade, what should they do to move toward their desired grade, and how to understand the lesson learned if they don’t.

CMU has the culture of sink or swim but it does more harm than good both in terms of educational effectiveness and mental health. It’s what we have to live with but admittedly, when I was a freshman I can be very upset when I didn’t get full scores on assignments and I was so afraid of earning a B in any class. It takes time for students to grow, while 150 may be one of the first places for students to learn this life lesson. Sometimes students just need affirmation, and we can try our best to make 150 an even better educational experience.

TA’s Feedback: An Internal Conversation

I received some feedback and also pushback from my fellow TAs. For example, more experienced TAs pointed out that they have been aware of and trying to fix these issues for a while, but unfortunately, no good solution came out yet:

Grading test cases explicitly — we used to require students to write “exhaustive” test cases which would be manually graded by a TA and they were allowed to fix missing test case deductions after grades were released. The issue here is defining what “exhaustive” test cases mean, and where to draw the line. …In the end, we decided that the students weren’t really learning much from being forced to write their own test cases and the risk of losing points should be enough to teach students to test their code. For running test cases against buggy code, (another course) does this and I don’t really think it’s a very successful assignment for motivating writing better test cases.

Getting points back after grading — the main issue here is that once we release grades, those who receive full points will know that their solution is correct. … Since many 150 problems result in correct solutions all looking alike, it becomes difficult to detect cheating.… I’m also worried that it’ll be an extra burden on students who are struggling, … they may be even more stressed out with the pressure of having to fix all of their code from the week before … and then have less time for the next homework assignment and then it just becomes a vicious cycle… given our relatively strict homework schedule.

Some TAs also feel like we already gave the rubrics to students in different forms and good enough starting place, so they worried that providing additional information may even be harmful to students, e.g.,

We give them through the lab where we walk through examples, emphasizing important points, and through the lab solutions as references.

I think releasing additional information about what kinds of tests we run on the code (and what will give points) just encourages writing to the test cases and making lots of bad partial solutions rather than a few good complete ones. I think we do encourage students to come to OH if they are confused about the grades they get so that we can help them try and assess their errors but the vast majority of students do not take advantage of this, and i’m not really sure of anyway to make them.

Still, I was more from a “telling consistent story” point of view: if we want students to learn something then we test it in some ways and provide feedback. So if reflection on mistakes should be encouraged, we can probably borrow the resubmission system of other courses, or think about some lighter ways to incentivize it, e.g., if students come to OH and discuss with a TA, they earn some tokens that may accumulate to extra late days.

Some concerned about releasing grade statistics, like

we never have any pre-determined percentage of A’s and B’s, and we’re very concerned that once we release stats people will ask about the curve we’re giving.

Summary: Thoughts & Reflection

Understanding different perspectives are key in communication

From all these discussions, I learned again that people can possess really different perspectives on the same thing, and if we can’t understand others’ perspectives, it can develop tension and frustrate both sides of the communication.

For example, the students suggested changes to the course, which were pushed back for various reasons, so they feel like their concerns are still not acknowledged in the end.

I have given multiple suggestions, tackling each counterargument as they come to try to come to some sort of compromise and middle-ground on the issue. … The first suggestion was shut down due to objectiveness and feasibility to implement. Understandable. The second suggestion was shut down to objectiveness and manual labor involved. Understandable. The third suggestion was shut down for “students will not try as hard” as a legitimate argument. Respectfully, this argument holds no water. No student will ever have the mindset to not try as hard on a specific problem to gain 20%…, especially when new deadlines and new assignments will be coming out non-stop, … and to imply so is not only rude but disrespectful to every student in this course.

They can stay calm in their first several responses, but it gets frustrating when one’s feedback is not going anywhere and thus they went on and said

The reality is: you don’t want to create a compromise, even if said compromise tries to tackle all your concerns. … Multiple suggestions have been made, trying to empathize with TAs and the concerns of the professors and they were all subsequently shut down, and some with counterarguments that have little to no actual empathy or thought put into them. And that is completely fine, I can understand the want to not change your course and or it’s structure. But don’t try to present the course with anything relating to helping the mental health of students. Because, clearly, those considerations weren’t considered seriously.

I definitely understand how TAs can feel sad when seeing this harshly-worded feedback, but I can also understand how some students are very disappointed because they feel like they are not heard.

Nonetheless, since the mistake of the student who wrote the original piazza post could have been fixed by running a test case provided in the homework writeup but they wrote some threads in this “antagonistic fashion²,” some TAs see the students as very demanding. They believe that this shouldn’t be a “negotiation with students,” and they also couldn’t see why students are so obsessed with and anxious about their grades to the degree of influencing their mental health:

I’m rather saddened by the tone you use there. It is one thing to say that you don’t like the current course policy, but it is altogether another to claim course staff has no empathy and doesn’t consider the perspective of students. I can certify to you that is simply untrue. Please do not conflate getting more points on assignments with mental health. I can empathize that I myself have been frustrated, saddened, etc., because of a gap in my expectations versus outcomes, but I don’t think it is reasonable to say the proper resolution comes about merely from saying the outcome is unfair and demanding it be changed.

In hindsight, it’s evident that these students who feel like their feedback is not being acknowledged were strongly discouraged, stopped being actively involved in the course, and gave up on some of the following work. Therefore, how to build a community where feedback is acknowledged and different perspectives are understood is important for a large-scale class like 150.

Educational theories may not be well-accepted or understood

What I saw in the piazza post is that students didn’t know how to interpret their low scores and assess their errors, and from learning science theories, novices are known to be immature in how they deal with errors & metacognitively reflect³. It is a skill to be learned and students who come from a different background may not have enough opportunities to practice this skill before they enter college. However, some other TA had a totally different take on my comment that

We shouldn’t expect novices to be able to understand whether they learned the materials or reflect on their own failure in a mature and positive way.

They believe that this is a wrong starting point, as it

… seems exceedingly patronising to me. I’m all for best-effort attempts to get students to reflect on their learning, but really? I expect students to be able to behave in a mature way. Starting from a baseline expectation that students are mature adults is not unreasonable to me, and I think it’s very disrespectful to students if you start off with the expectation that they have no idea how to reflect on their errors. It is one thing to make resources available…, but it is another with the perspective that you know better than students and they need to be coddled to do any manner of self-reflection.

From my perspective, if we assume that our students are masters of self-reflection and can apply this skill to various fields (which I don’t think is true for 150’s student body), reflecting/ assessing/ understanding one’s errors in a new and challenging area (functional programming) still takes practices. But I can see other TAs’ concerns that the reflection opportunities may be manipulated:

I certainly don’t think its inherently a bad thing to create more opportunity for students to reflect, but if we want to put serious thought into something of that nature we should make sure that it effectively creates the ability to reflect, and isn’t just a conduit for point-obsessed students.

Another comment that I received also triggers my thoughts, and it is consistent with a recurring theme in the post discussions:

Sometimes students have to lose points in order to learn, and although it’s sad to lose points it gives students an opportunity to learn and improve. So personally, I’m less concerned about partial credit and more concerned about providing as many learning opportunities for students as possible.

Life is full of failures and we need to learn to accept and reinterpret those failures as important educational moments, but I wouldn’t agree with the logic that people need to fail to learn. Providing more learning opportunities also doesn’t mean that students would fail less or learn less from their failures. I’ve observed this frequent association between the notion of “learn from failure” and “can’t learn without failure,” accompanying the belief that courses should simulate real life for students to best learn. But this position makes implicit assumptions about the skills that students are actually lacking and that are not explicit to students or integrated as learning objectives of the course.

Incorporate feedback and improve educational quality is hard

I know translating theories to practice is hard in various of my teaching experiences before. However, being engaged in such an intense discussion and seeing improvement stuck still strike me as to how difficult education could be in reality.

Since I was a new TA for 150 and I only have 2 semesters left in CMU, it’d be hard to get to a head TA position to push forward changes in the course, so I decided to gain more TA experiences in different courses and expose myself to different sets of course policies, student experiences, and similar or different challenges.

I’m planning to be a researcher in learning science & educational technology, especially focusing on bridging the gap between theories and practice. This conversation again confirms the importance of this body of research, and I definitely need more experience to navigate this complicated and frustrating reality of education.

Regarding 150, feedback has been given and this whole conversation about course improvement has received attention. There may be some systematic reasons behind all these frustrations, as similar complaints occurred in many other CS courses at CMU. My current abilities couldn’t let me come up with a really good solution, but I’ll continue to learn and grow. One thing is clear though:

We all deeply care about students’ learning and the teaching staff will keep trying, making 150 a better course.

Credit to Mia Tang, a former 150 TA

Notes and Reference

¹ Throughout the article, I’ve used anonymized original quotes from the discussion. If your original quote appeared, please let me know if you want me to cite you, or if you want me to paraphrase instead of using your words.

² I mostly used calmer quotes in this article, the original conversation could be more intense.

³ Ambrose, S. A., Bridges, M. W., DiPietro, M., Lovett, M. C., & Norman, M. K. (2010). How learning works: Seven research-based principles for smart teaching. John Wiley & Sons.