This week I participated in the inaugural FLAIEC conference. It was a delightful opportunity to talk with various researchers at different levels and years of experience.
The keynote speakers, who are leaders in the field, have called for a more critical approach to examining the validity of the different indicators.
Dragan Gašević in his opening keynote, outlined the opportunities and challenges for the advancement of validity of measurement in learning analytics. While Dirk Ifenthaler called for careful processing of the existing literature in an effort to build a valid set of meaningful indicators.This topic has been central to our group's work. We have long discussions about ways to validly connect the digital traces of learning with reasonable assumptions. The most significant aspect of validity is a robust theoretical set of assumptions. A theory or framework should not be assumed to be infallible. In fact, I believe that we can use our data to confirm or challenge existing theories to develop a better understanding of human learning.
The abundance of data can lead to overfitting the data to a specific theory. For example, we are currently looking at the importance of Self Regulation (as did many of the papers at the conference). The data fits the theory rather well, with the effect of self-regulation on course achievement significant but entirely mediated by specific learning behaviors. As I reviewed our results, I started wondering about the power of alternative theories. One of our next steps will be to contrast this theory with other theories and see which one fits better with our data and which is more stable across courses and time.
The theory does not absolve us of the need to check in with more traditional (even psychometric) forms of validity. But for me, a theory is still the key to understanding the data. I am also eager to see what data from our online learners of code can tell us about the development of computer science knowledge, but that is a blog post for another time.
