Assesslog

And, yes, I’m still thinking about this 2006 video from Teachers TV: http://www.teachers.tv/video/5431…

The programme reported on a pilot of the use of PDAs for students to record the process of a design project – sketches, digital photos, notes – as they worked, effectively creating an electronic portfolio. The research was carried out by Tony Wheeler and Richard Kimbell from Goldsmiths’ Technology Education Research Unit, and you can find some more information about the project here.

The assessment of process, particularly the creative process, has been a major theme in our conversations on this unit. The use of PDAs and other mobile devices for students to contemporaneously capture their work has obvious benefits in terms of the immediacy, mobility and general convenience of the technology; it makes sense that one might receive a more valid or honest representation of the process than if the student was taking pictures, making drawings & notes, and then sitting down at a computer at some point in the future, putting everything in order and linking it together. According to Tony Wheeler, the contemporaneous recording of and reflection on students’ work also helps to guide what the students do next.

The interesting thing about this pilot was how the researchers went about assessing the students’ work; they ranked all the submissions using a judgemental pairs system.

The reason they gave for doing this was the difficulty with the meaning of numerical marks, which according to Tony Wheeler are – despite the existence of clear criteria – often awarded according to a subconscious comparison with an imaginary standard. The researchers found that the judgmental pairs system of ranking – previously quite labour-intensive – becomes much faster and simpler when the work is electronic and easily shared.

Although the researchers felt that the ranking system they’d used had benefits in terms of consistency (reliability?), they didn’t really go into much detail about the criteria that they’d based their judgements on – you got the impression that they’d looked at the students’ work as a whole and simply followed their instincts in a rather ‘organic’ way about the depth of process each student had gone through and reported. I don’t doubt that this method would result in a fair amount of consistency between markers, but it does leave me puzzled about the exact nature and content of the ‘summative judgements’ that the students would have received as the output of the assessment process, as I presume they didn’t merely receive a piece of paper with their rank order on! As the researchers said themselves; employers want flexibility, initiative and the ability to collaborate, and we need to assess those capabilities. Were these examples of the type of criteria the researchers were basing their comparative judgements on? Did they make these explicit from the start – to the students and/or to each other? Were they referred to in the ‘summative judgements’ that resulted? Did they feel that this was primarily a norm-referenced assessment, or a criterion-referenced one? I think I’d really like to have a conversation with the Goldsmiths researchers about this – I have so many unanswered questions! However, they’re probably really busy people so I guess I’ll just have to read the project report…