data, photography, education
11 December 2020
Introduction and background
With the recent surge and adoption of educational technologies to ease the transition to remote learning in these unprecedented times amidst a pandemic, some schools have adjusted by adopting platforms offering student testing surveillance systems to replicate a classroom testing environment. These platforms are most concerned with how to best prevent cheating. Some platforms even hire “professional cheaters” to help test the systems and develop new ways to prevent any conceivable act of cheating on remote exams. However, far from the forefront of their concerns lies the many potential privacy, inclusion, and justice issues surrounding these video, audio, and algorithmic surveillance systems. In this paper, we will be focusing on the ethical and legal implications of student surveillance technologies through the lens of the leading testing surveillance platform, Proctorio.
Proctorio is the most popular testing surveillance platform used by over 400 universities. Many schools are rushing to sign deals with the platform as a part of their adjustment to online schooling amidst the pandemic. Given this newfound traction and the need for schools to replicate a proctored testing environment, Proctorio and similar platforms have developed advanced analytics to detect students at risk for cheating on a given exam. These analytics and resulting information pose many concerns around student data privacy, inclusion, discrimination, and negative emotional impacts: risks that very much outweigh those that Proctorio tries to prevent.
Disability and testing anxiety biases
While Proctorio states that they aim to “remove bias and human error” through the use of their automated proctoring system, their purely automated system introduces biases related to disability status, race, gender, income, and access. Proctorio uses advanced analytics to detect abnormalities that subsequently are used to calculate a student’s risk of cheating on a given exam. The inputs to these algorithms include audio recordings, eye tracking data, facial recognition data, detected movements, student written answers, and background motion. Proctorio may also require a full 360 degree “room scan”, where the student must pick up their camera and show the full extent of their surroundings, so as to reveal any hidden help that may be used to cheat.
Labeling certain behaviors as abnormal, and therefore risky, is inherently biased towards certain groups of students. Most notably, abnormalities in eye movements occur much more frequently among students with learning or behavioral disabilities, specifically ASD, ADHD, anxiety, or vision impairments. Students with these disabilities are far more likely to be labeled suspicious because the algorithm judges them on the same scale as their peers. Students with disabilities often receive differentiation in classroom learning and on the tests themselves, such as differently worded questions, steps mapped out to complete a problem, extra time on a given exam, etc. However, this differentiation is not present in the proctoring software, where it very well should be.
Platforms like Proctorio seek to replicate the classroom environment in an online one; however, this transition may warrant completely different student needs. Within a classroom, a student with ADHD will often still take their exam with the rest of the class, with access to something like a fidget spinner and perhaps seated in a minimally distracting place in the classroom. However, the actual proctoring of the exam does not need much conscious differentiation. The teacher knows the student in a way the algorithm never could; they may know the student often needs a few short breaks to stand up and move around before coming back to the test, or that the student may need a few reminders to focus on the exam when the teacher sees them staring off into space. With online proctoring, the student no longer has the chance to get up and move around the room and will not receive those reminders to focus on the exam. Instead, the fidgety movements or spacey stares will be immediately deemed suspicious since these behaviors are being compared to their differently abled peers, with additional stress from the knowledge of the potential flag as “suspicious”.
Additionally, being subject to software such as Proctorio may force students to disclose their physical disorder, disability, or health condition that was formerly irrelevant in terms of their education. Although schools will have information on a student’s learning disabilities and certain health information like allergies or asthma, other physical conditions may have been irrelevant to the student’s education previously. For example, a disorder involving involuntary facial movements or tics may not have been noticeable to teachers in an in-person classroom, but now it becomes necessary for the student to disclose to school leadership in order for the cheating “flags” by proctoring software to be appropriately disregarded by the school. Such a disclosure to school officials would not prevent the flags from happening in the first place, but would allow the school to label the flags as non-cheating behaviors by that student. Although the disclosure does not go to Proctorio, forced disclosure of health data to school officials is not within the scope of what proctoring software should require.
In a classroom environment, the teacher (proctor), mostly subconsciously, can differentiate between cheating behavior and non-cheating behavior for each individual student: behaviors that look different for every student. Without this same knowledge about individuals, the algorithm cannot fairly differentiate between suspicious and unsuspicious behaviors. Thus, the very design of an algorithm that is equally applied to every student is inherently biased against students whose test taking behavior does not, for any reason, conform to the “normal” student behavior during a test.
With constant video and audio surveillance, students are well aware that they are being watched, recorded, and labeled. Knowing that small movements may immediately label them as suspicious of cheating, students’ level of anxiety is extremely heightened. Particularly for students who already experience test-taking anxiety, a platform like Proctorio has adverse effects on students’ emotional states and their subsequent performance on the test. One student stated that they “feel anxious and sick to [their] stomach about having to use this program” while another student with a visual impairment “is terrified of a sketchy eye movement because [they] need to shift them once in a while”. Platforms like Proctorio disproportionately negatively affect students with existing levels of anxiety, particularly in a test taking environment. Each individual student experiences test taking very differently; there needs to be a human being on the other end of the test taking experience with knowledge of these differences, so individual student intricacies are viewed not as suspicion but rather as what they are: intricacies.
Racial Biases
Testing surveillance systems are woven with racial biases at different stages of the test taking experience. Proctorio and similar platforms use a facial recognition software to verify a student’s identity before they begin the test. Because the training data mostly consisted of faces of lighter skin tones, the software is significantly better at detecting faces with lighter skin tones than those with darker skin tones. The difference in accuracies across student races forces darker skinned students to often find workarounds when their faces are not recognized. There is also a similar problem in identifying faces of transgender or genderqueer students, since they are also underrepresented in the training data.
However, the problems do not only lie in facial recognition technology. Eye tracking software is generally more sensitive to movements from darker colored eyes than it is for lighter eyes or asian eyes. The software will be more likely to label a given eye movement as anomalous for a student with darker eyes than for a student with lighter eyes. Students with darker skin and darker eyes, a group consisting primarily of black, indian, or latinx students, will experience the surveillance platform differently than their lighter skin and eyed peers. These surveillance platforms reinforce biases towards traditionally marginalized groups, thus perpetuating racism within our educational system and ultimately contributing to inequitable student outcomes.
Access
Proctorio and similar platforms also introduce many issues surrounding access, both to technology and to the expected environment. Proctorio requires a certain strength of WiFi and quality of computer hardware to run smoothly, thus disadvantaging students without access to this technology. Students often worry they do not have enough RAM to run Proctorio effectively or that their WiFi is too weak to maintain connection to the platform. Therefore, these surveillance platforms disadvantage students with less technological access, often lower-income students. Since race and income are largely linked, this bias against technological access can also be seen as introducing further racial bias into the system.
Additionally, the platform records other voices in the audio stream or movements in the background as indicators of suspicion. The assumption that every student has access to a quiet environment with no distractions disproportionately disadvantages those who may live in a crowded home, may not have a room to themselves, may not live in a home at all, etc. A single parent, for example, may not have the ability to leave their kids in another room while they complete an exam. It is very likely there will be audio in the Proctorio recording that will immediately deem them as suspicious. By making the environment in which a student takes a test a qualifying factor on their level of suspicion inherently biases those who do not have access to such an environment.
Data collection, ownership, and consent
One of the major concerns around the data collected by Proctorio centers around the fact that students, for the most part, are minors. Not only is Proctorio collecting sensitive data including PII and facial recognition information, but that data is being collected from and tied to minors. Additionally, the structures around consent to data collection are questionable at best. The use of the platform is often framed by Proctorio as “opt in” for students. However, when the options are handing over biometric and other private information or failing the class, using the platform is not a choice for students. While there are some controls over the information collected about students on the institution’s end, the use of the platform requires most of the sensitive information mentioned throughout this paper to be collected. For example, the platform must collect facial recognition information, but whether to retain the audio and video recordings of students is up to the instructor or institution. Perhaps most importantly, none of these decisions are made by the students themselves, regardless of whether they are a minor.
To address concerns about collecting minors’ data, Proctorio offers an entire page to address the requirements laid out by Children’s Online Privacy Protection Act (COPPA). However, half of the page is merely explaining COPPA, and Proctorio addresses their COPPA compliance in a single paragraph. Their compliance lies in the fact that the data is encrypted, parents are notified of data collection from individuals under the age of 13, and Proctorio does not ever share information with third parties. While they may not explicitly share or sell data to third parties, Proctorio (and similar platforms) require the installation of a third party software to run the program, which is relatively easy for hackers to exploit.
In its privacy policy, Proctorio claims they “never require personally identifiable information (PII) to use [their] software… the only time [they] record you for your exam is if and when [the user’s] instructor chooses the video or audio recording option for [the] assessment.” However, Proctorio markets itself through the use of these video and audio recordings and the ability to flag potential cheating. Thus, it is highly unlikely that an institution will partner with Proctorio and opt out of using its main feature. Additionally, the privacy policy does not actually state what data is collected and retained when the instructor enables audio and video recordings. Given the functionality of Proctorio, one can infer that, at the very least, facial recognition data is collected, which is arguably the most personally identifiable information possible.
The video recording does not just contain information about the limited background behind the student. Students are also asked to perform a “room scan”, where they must rotate their camera 360 degrees to demonstrate that no one else is in their testing space. This room scan offers potential for Proctorio to collect further information on their surroundings and arguably invades a student’s, and their family’s, personal space. The Proctorio data collection practices breaches not only on individual data privacy but also on an individual’s privacy of their own home.
Proctorio partners with institutions to decide on the appropriate time to retain their students’ data, with the default retainment of 30 days. While the privacy policy makes it clear that institutions will not have access to student data after this retainment period, it is unclear whether Proctorio retains the student data for their own purposes. In addition, because students are not the owners of their data, they do not have the ability to delete, access, or edit the data being collected about them. Through the lens of structures such as GDPR and CALOPPA compliance, users must be able to obtain a record of the data collected from them. Proctorio considers institutions to be its users and thus gives them the required ownership and access rights. However, we argue that students are also users and should be given similar data ownership rights.
The data collection practices and ownership rights Proctorio purposefully extends to institutions makes the blame for the platform’s misuse ambiguous. Proctorio allows institutions and/or individual teachers to configure their settings to allow video and audio recordings. Given this, Proctorio argues that any misuse that results from these data is on the shoulders of the institutions or teachers who configured the settings to provide it. However, Proctorio also does not provide adequate information to the institutions and teachers about how these features work. We argue that without a high level of transparency about the functionality of the audio and video features, Proctorio cannot off-load the blame for their misuse to the institutions themselves.
Proctorio may preach about its data privacy practices on its website, but it does not always live up to its words. The CEO recently tweeted a string of student chats obtained from the platform. After the appropriate backlash, he subsequently removed the post. However, the initial privacy breach, although seemingly small, serves as an example of how lightly Proctorio takes the importance of student privacy as a company. Additionally, it brings into question Proctorio’s actual privacy practices; they claim not to retain any data, including chat logs, but here the CEO is tweeting out screenshots of precisely that data. As discussed, there are valid concerns over the potential of third party sharing and external data breaches; however, this particular example raises concerns over internal breaches from Proctorio employees. The trust that institutions place in Proctorio to safeguard their students’ data is not met with an equal level of transparency and standards of protection from Proctorio. Thus, the lack of importance placed on the protection of student data warrants a lack of public trust in providing their data.
Recommendations
We must consider the context in which the data collection and use is occurring when evaluating the need for data privacy. This framework is especially useful to consider in the context of student data being used to predict riskiness of cheating. In this case, the student data is being used in such a way that affects students’ educational outcomes. Given that our education system is currently very inequitable, consistently failing disproportionate numbers of students of color, LGBTQ students, and low income students, we must take additional precaution in contexts like this in order to help dismantle the current system as opposed to contributing to it. Based on our discussion of the many biases that could and do result from such a platform, Proctorio and similar platforms are contributing to the systemic oppression and racism within the educational system. Their data collection practices and data use disadvantages some and rewards others, helping to widen educational and subsequent opportunity gaps that persist.
There are a few recommendations we would propose to try to mitigate the potential harm done by surveillance platforms like Proctorio. Since there are many issues that stem from the collection, storage, and use of audio and video recordings of students, there is a need for the platform to become a live test proctoring session. If the teacher is present in the room and sees that a student’s behavior is being flagged, the teacher would be able to interpret the behavior through a human eye and with their understanding of the individual student: a level of context not able to be interpreted by an algorithm. The teacher would then have the ability to dismiss the behavior upon interpretation or further investigate it with the individual student. This reintroduces the level of trust and familiarity students have with their teacher, mitigating the fear students may have of algorithmically being labeled as a cheater. Additionally, Proctorio could limit the potential for data breaches or hacks by not using a third party software to run the program. There also needs to be a level of data ownership in the students’ hands; students (and/or parents) should have the ability to access, correct, and delete personal information about themselves, including their facial recognition data and other personally identifiable information. There should be additional options for student verification, possibly allowing the teacher to verify their identity live or the ability for the teacher to disallow the particular feature. Lastly, the default settings on the teacher administration page should be set to every feature being deselected, with an individual agreement displayed to the teacher upon selection of each feature. This agreement should clearly state the potential harms that could come from the collection of the corresponding data as well as the specific ways in which that particular data will affect the predictions of cheating for a particular student. While the aforementioned recommendations may mitigate some of the harm caused by testing surveillance platforms, they in no way address the many harms that these platforms pose within their core design.
There are a seemingly endless number of potential issues with testing surveillance platforms like Proctorio stemming from the biases they inherently introduce, the potential privacy implications of collecting sensitive data, and the legal concerns around data ownership and consent. While there are some potential fixes to mitigating the problem, the very idea of an automated proctoring system detecting anomalous behavior is flawed in concept. We simply cannot have an algorithmic system determine whether a student is cheating from a limited view of their world behind a webcam. Our biggest recommendation, therefore, is to abandon the need to replicate a classroom testing environment online. Online learning will simply not look the same as in-person learning and we need to treat the two very differently. In the context of testing, this might mean that, in an online environment, we assess students based on projects, group discussions, or open-note tests as opposed to traditional proctored closed-note exams. In a classroom, everyone is in the same environment and has access to the same materials. Online learning will never replicate this aspect of in-person learning and thus our teaching methodology should change accordingly.
“In a classroom, everyone is in the same environment and has access to the same materials. Online learning will never replicate this aspect of in-person learning and thus our teaching methodology should change accordingly. “