JoWUA

Abstract

GAI applications such as ChatGPT, Gemini, and Perplexity have brought a revolution in the field of academic writing and raised the issues concerning academic integrity in higher education institutions. The emergence of GAI applications such as ChatGPT, Gemini, and Perplexity has brought a revolution in academic writing and raised new issues concerning academic integrity in higher education institutions. In order to address these issues, universities have started using more and more AI content detection tools. However, there is a lack of empirical research regarding the fairness and effectiveness of these tools when assessing content that is not written in English, particularly in Arabic language. This paper evaluates the efficiency and algorithmic fairness of the AI content detection tools when it comes to Arabic and English student writing. Comparative empirical assessment of these tools was performed based on bilingual academic writing of undergraduate students. The data set used included 300 text samples, both human and AI-generated, in Arabic and English languages. For each text sample, approximately 900 detector-level evaluation results were gathered from three popular AI detection systems: Turnitin, QuillBot, and ZeroGPT. The metrics used for evaluation of the performance include Confusion Matrix analysis, Accuracy, Precision, Recall, and F1 score. A fairness gap measurement was also considered as a way of capturing the difference in performance between languages. The results clearly show a great indicator of how there is a difference in the performance of detectors in different languages. All the three software had a perfect classification on English texts, with 100 % accuracy, precision, recall, and F1-score. However, the performance of this software is considerably poor in case of Arabic texts compared to other cases. Turnitin has 26% accuracy, Quillbot 28% and ZeroGPT 23%, with zero accuracy in Arabic AI text detection in terms of precision, recall and F1 score. The calculated fairness gaps ranged from 72% to 77% – this is considerable linguistic disparity. From this study, it is clearly shown that AI content detection systems available now are very good at detecting English content but very poor in Arabic content detection. This raises a number of concerns regarding issues of fairness, transparency and equitable evaluation in relation to school activities. The results indicate the importance of cross-linguistic datasets and validation.