From black box to pedagogical partner: Students’ sense-making of LLM-based automated assessment

Creative Commons License

Kara A., YILDIRIM S.

Participatory Educational Research, cilt.13, sa.4, ss.251-270, 2026 (Scopus)

  • Yayın Türü: Makale / Tam Makale
  • Cilt numarası: 13 Sayı: 4
  • Basım Tarihi: 2026
  • Doi Numarası: 10.17275/per.26.58.13.4
  • Dergi Adı: Participatory Educational Research
  • Derginin Tarandığı İndeksler: Scopus
  • Sayfa Sayıları: ss.251-270
  • Anahtar Kelimeler: AI-generated feedback, Automated assessment, explainable AI, large language models, student experience
  • Açık Arşiv Koleksiyonu: AVESİS Açık Erişim Koleksiyonu
  • Atatürk Üniversitesi Adresli: Evet

Özet

While current educational research on automated assessment heavily emphasizes technical validity, a significant gap remains in understanding students’ sustained, real-world experiences with these systems in authentic learning environments. This study examines students’ experiences with a large language model-based automated assessment system embedded in the regular flow of a university course. The study employed a mixed-methods design with 47 university students over a seven-week period. Quantitative data were obtained from system interaction logs and student feedback ratings, while qualitative data were collected from focus group interviews with 24 students and 175 written feedback responses. The results reveal that students perceive the LLM-based assessment system as a learning assistant, an impartial evaluator, and a self-assessment tool. The transparency of explanations was identified as a decisive factor in building trust in the algorithmic system by helping students understand the rationale behind scores and feedback. Sustained interaction with the system triggered a shift from high-frequency trial and error to more efficient and strategic participation, indicating that the assessment criteria were gradually internalized. The system appeared to create an environment free from perceived social judgment, providing favorable conditions for productive failure, repeated attempts, and self-directed revision. Overall, the study demonstrates that artificial intelligence can be positioned as a tool that scales pedagogical intent without replacing the teacher and can be effectively integrated within the framework of human–AI complementarity in higher education.