Here is my hypothesis:
The more we focus on one aspect of the learned, i.e. in exams we force the expression of learning into a specific way and the closer we look at that one specific factor the less we are able to see the effects of said learning more widely.
A personal experience example
At one point during my last year in highschool I learned something about exams—which would have been helpful to learn about 10 years earlier—the teacher expects you to write down the obvious things everyone (from your teenage perspective) knows. So the teacher went through the results of the last exams with us, and was talking about the different facts that were supposed to be in the answer. I remember sitting there completely dumbfounded, thinking: oh I know this and I was thinking about that, but it never occurred to me that this stuff should have been written down in the exam. This is one of my rare detailed memories from school because I was so surprised by that insight: all the teacher wanted to know was the obvious stuff. Go figure! It, until this point, had never occurred to me to write it down. So exams in school only observed the learning I tried to second guess I ought to demonstrate not what I actually knew or had understood.
So what does this mean for our SoTL Projects?
It depends on what you want to find out. Evaluating learning is a complex art, it requires data triangulation, it requires rigor of research design and planning, and being cognisant of correlation-causation fallacies. The most attention should be paid to methodology and theoretical frameworks–particularly the theory part is often neglected in SoTL projects. For me these are not so much about learning new ideas but becoming aware of the beliefs we hold about the nature of knowledge and knowledge creation. This is important as our believes shape how we plan (methodology) our educational inquiries and the inevitable implicit biases we build into this design.*
Maybe I need to take a further step back to provide context. In the most recent #LTHEchat led by the amazing Frederica Brooksworth we were debating decolonizing the curriculum. Often when assessing SoTL project proposals or reviewing papers, I notice the main research methods used are: questionnaires, focus groups or interviews. Which in itself is not wrong, but it begs to question, what (rich) data are we missing? How do we create knowledge? How are we influencing participation? Would other approaches utilizing creative research methods, participatory approaches, and a strong focus on the impact of theoretical frameworks enable us to create meaning-making spaces that would yield not only more meaningful data but also encourage silent voices to join, to shape and shift our understanding, and inevitably our practice?
*Here are a couple of resources that offer a brief and useful overview of some of the key theories and concepts
https://www.wgu.edu/blog/five-educational-learning-theories2005.html#close https://www.sagepub.com/sites/default/files/upm-binaries/58936_Brundett_&_Rhodes.pdf
https://files.eric.ed.gov/fulltext/ED567134.pdf
Adventures in Academic Development 