Academic Writing Analytics (AWA)

Students learn that critical, reflective writing must make their thinking visible. This is conveyed through particular linguistic forms. Can computers learn to identify these patterns, and feed them back to students to help improve their writing?

The ability to communicate and debate ideas coherently and critically is a core graduate attribute. In many disciplines, writing provides a significant window into the mind of the student, evidencing mastery of the curriculum and the ability to reflect on one’s own learning. Arguably, in the humanities and social sciences, writing is the primary source of evidence. Moreover, as dialogue and debate move from face-to-face to online in a variety of genres and digital channels, discourse shifts from being ephemeral to persistent, providing a new evidence base. Yet while all the evidence shows that timely, personalised feedback is one of the key factors impacting learning, and students consistently request quicker, better feedback, assessing writing is extremely time-consuming — whether a brief first assignment, a draft essay, a thesis chapter, or a research article in preparation for peer review.

CIC is prototyping a formative feedback app for student writing which we call Academic Writing Analytics (AWA), working in close partnership with academics from diverse faculties, HELPS and IML. This Natural Language Processing (NLP) tool, using technology from our partner at Xerox research (Xerox Incremental Parser: XIP), identifies concepts, people, places, and distinctively, the metadiscourse corresponding to rhetorical moves. These moves are important ways of using language to signal to the reader that a scholarly, knowledge-level claim is being made, but UTS practice and the wider research literature evidence how difficult this is for students to learn, and indeed, for some educators to teach and grade with confidence.

This is not automated grading, but rapid formative feedback on draft texts. AWA is designed to make visible to learners the ways in which they are using (or failing to use) language to ‘make their thinking visible’ — i.e. construct claims and argumentative reasoning for academic writing. A series of pilots is now under way with staff and students.

Key references:

Buckingham Shum, S., Á. Sándor, R. Goldsmith, X. Wang, R. Bass and M. McWilliams (2016). Reflecting on Reflective Writing Analytics: Assessment Challenges and Iterative Evaluation of a Prototype Tool. 6th International Learning Analytics & Knowledge Conference (LAK16), Edinburgh, UK, April 25 – 29 2016, ACM, New York, NY. http://dx.doi.org/10.1145/2883851.2883955 Preprint: http://bit.ly/LAK16paper

Critical Perspective on Writing Analytics. Workshop, 6th International Learning Analytics & Knowledge Conference (LAK16), Edinburgh, UK, April 25, 2016. http://wa.utscic.edu.au/events/lak16wa

Simsek, D., Á. Sándor, S. Buckingham Shum, R. Ferguson, A. D. Liddo and D. Whitelock (2015). Correlations between automated rhetorical analysis and tutors’ grades on student essays. Proceedings of the Fifth International Conference on Learning Analytics And Knowledge, Poughkeepsie, New York, ACM. http://dx.doi.org/10.1145/2723576.2723603

Simsek, D., S. Buckingham Shum, A. D. Liddo, R. Ferguson and Á. Sándor (2014). Visual analytics of academic writing. Proceedings of the Fourth International Conference on Learning Analytics And Knowledge, Indianapolis, Indiana, USA, ACM. http://dx.doi.org/10.1145/2567574.2567577

Simsek, D., S. Buckingham Shum, Á. Sándor, A. D. Liddo and R. Ferguson (2013). XIP Dashboard: Visual Analytics from Automated Rhetorical Parsing of Scientific Metadiscourse. 1st International Workshop on Discourse-Centric Learning Analytics, 3rd International Conference on Learning Analytics & Knowledge, Leuven, BE (Apr. 8-12, 2013). Open Access Eprint: http://oro.open.ac.uk/37391