Learning analytics should include measuring change
I have been mulling over the use of learning analytics as a research method to determine if and how students are learning when engaged with online learning tools. At the latest Learning Analytics and Knowledge conference in Banff, Alberta, Canada, it occurred to me that one of the most troubling concepts for presenters was to determine what to measure when applying learning analytics. I agreed when some presenters stated merely measuring the number of log-ins, hits or postings did not provide an accurate indication of whether students were learning. For instance, in online learning environments (formal and informal) I don’t log on that much. When I do I download or link readings/materials and learn on my own more rather than with others. Once I feel I have a grasp of the content or ideas I log back on and engage somewhat near the end of the timeframe alloted. I learn in social and collaborative settings, but not as much as lauded by those promoting the idea.
The point is I don’t believe we can measure learning based on online activity alone – we need to include and bridge it with assessment. More important, learning is believed to be recognizable by a change in behaviour, thinking and attitude.; in short, we change and grow through learning and this needs to be captured, measured and examined to determine the success of students.
Capturing data to measure change in learners poses some problems. Ideally having pre- and post-tests would provide one way to measure change and growth, but it’s not possible to structure all learning activities or assessments that way. Other possible ways to measure learning is the level of achievement through various assessments, and through feedback from students on perceptions of their learning (via surveys, etc.). However, perceptions and knowledge acquisition are not synonymous. Determining evidence of change in learners to use in learning analytics will be challenging but necessary in order to gather essential data.
Aside from evidence of change and growth, online learner interaction and engagement would provide data on their actions that perhaps attributed to their learning. I would suggest to move beyond using tracking data provided in a LMS and also include interactions in external learning environments and tools, such as blogs, wikis or social networks sites. I envision this data would include frequency statistics (hits, posts) and network visualizations (connections).
Lastly, it would be important to determine the resources learners are accessing. It has been realized in higher education that instructors are not the sole holder of knowledge, and the net provides endless sources of materials, resources and expertise. This might be more difficult to determine, but perhaps reviewing the cited works within students’ work might provide the resources they used outside those offered in the class.
Triangulating this data will be essential to understand how students engaged online and how that contributed to their learning and growth. Examining the correlations of this data would indicate if and which online activities and resource use affected learning and which did not.
This is a rudimentary understanding of the use of learning analytics in education. I am about to engage in a research study where I will attempt to apply this analysis method. Hopefully over time I can develop my thinking and share new ideas about this topic along with my experiences from using learning analytics as an educational research method.