Using AI to Predict Student Performance
As Deep Analysis’s most recent analyst, I wanted to start by writing on a topic close to my heart. Many years ago, like many of my fellow high school students this time of year, I waited, with eager anticipation, on my A-level results. The grades received in these exams determine whether or not students can confirm a place at the university of their choice. I was fortunate to receive the grades I needed to secure a place at Imperial College to read for a Physics degree. Had I not received the grades necessary, I may have secured a place at another institution or maybe none at all.
So, this year, the COVID pandemic presented unique problems for the UK Government’s Department of Education. Without being able to sit their final exams, how could students confirm their places and go on to study at university? Naturally, the government turned to technology to solve the problem. Computer models are used all the time to make predictions; they are critical to weather forecasts, provide guidance in economic forecasting and budgeting. So why not utilize AI to solve the problem?
Throwing AI at a Problem
AI has become such a common term it would be easy to think that the addition of a little AI can improve everything. The reality is that no organization ever became more successful by going out and purchasing a ‘bucket load of AI.’ AI does not have any intrinsic value until it is applied to a specific business problem.
While there were slight differences in how things were managed elsewhere in the UK, I will focus on England. (Which is, after all where I originally hail from). Because students did not sit their final exams, their teachers were asked to supply the following information regards students studying a particular subject, e.g., History:
- An estimated grade – the grade the teacher expected the student to receive if they had completed the exam
- Student ranking – an assessment of how the student ranked when compared to every other pupil in their class
This information was then inputted into an algorithm and combined with additional information, critically the prior performance of students in the same subjects from the same schools from the previous three years.
Be Careful what you Assume
Now, here’s where things get messy. The assumption was that the exam results of schools would be consistent with those in previous years. This information was given more weight than the teachers estimated grade because it was felt that teachers were likely to be more generous in predicting a student’s performance, leading to a more significant proportion of students attaining the higher grades.
While on the surface, this would appear to be a reasonable hypothesis, there are some scenarios that it fails to accommodate:
Scenario #1 – A bright student at an underperforming school
If a particularly bright student just happens to attend a poorly performing school, they would be more likely to have their grades lowered through no fault of their own.
Scenario #2 – A school demonstrating continuous improvement in student performance
If a school had seen consistent, rapid improvement in test scores in the last few years, this trend would not continue due to the assumption mentioned earlier that the predicted results would be consistent with previous years rather than extrapolating an upward trend.
Another critical factor is that predictive computer models are good at predicting things in aggregate, but their accuracy tends to degrade when asked discrete questions. For example, a weather forecast can tell me if it will rain tomorrow in my area but will struggle to predict precisely how much rain will fall in my garden and at what time. In this case, while a variance of a grade here or there does not seem a lot, the variation is not experienced evenly with specific students or with schools consistently more disadvantaged than others.
AI fails to get a passing grade
The result of this exercise was that nearly 40% of all grades were lowered by at least one grade, e.g., a student expecting an A in a given subject was given a B. Such a downgrade could have serious ramifications.
Figure 1 – How estimated grades were affected (Source: Ofqual)
Ultimately, it would appear that the UK’s Department of Education has reversed its decision and will, after all, use the teacher’s estimated grades. However, this decision has come too late for some who have already lost their place and may have to defer starting their course until next year.
What can we learn from this?
AI, like many technologies, is a tool, and, like many tools, merely possessing it does not make things better. I can use the same golf clubs as Tiger Woods, but I shouldn’t assume that I can just turn up on the first tee at US Masters at Augusta,
Think of AI as a new employee. Useful, but it needs to be provided with clear instructions to be productive. So, if you are considering incorporating AI, the bear in mind the following:
- Adding AI won’t always improve the results, it may just cost you more money
- AI cannot miraculously turn bad data into good results
- Think long and hard about the assumptions you make, as these can come back and bite you
- AI will only do what you tell it to do, so don’t blame the AI if you don’t like the answers it gives you
In conclusion, AI, like any new technology, should only be deployed where it offers tangible business value and not merely because it’s cool.
Work with us to ensure you are a disruptor not one of the disrupted!
Get trusted advice and technology insights for your business from the experts at Deep Analysis. [email protected]