Assesslog

And, yes, I’m still thinking about this 2006 video from Teachers TV: http://www.teachers.tv/video/5431…

The programme reported on a pilot of the use of PDAs for students to record the process of a design project – sketches, digital photos, notes – as they worked, effectively creating an electronic portfolio. The research was carried out by Tony Wheeler and Richard Kimbell from Goldsmiths’ Technology Education Research Unit, and you can find some more information about the project here.

The assessment of process, particularly the creative process, has been a major theme in our conversations on this unit. The use of PDAs and other mobile devices for students to contemporaneously capture their work has obvious benefits in terms of the immediacy, mobility and general convenience of the technology; it makes sense that one might receive a more valid or honest representation of the process than if the student was taking pictures, making drawings & notes, and then sitting down at a computer at some point in the future, putting everything in order and linking it together. According to Tony Wheeler, the contemporaneous recording of and reflection on students’ work also helps to guide what the students do next.

The interesting thing about this pilot was how the researchers went about assessing the students’ work; they ranked all the submissions using a judgemental pairs system.

The reason they gave for doing this was the difficulty with the meaning of numerical marks, which according to Tony Wheeler are – despite the existence of clear criteria – often awarded according to a subconscious comparison with an imaginary standard. The researchers found that the judgmental pairs system of ranking – previously quite labour-intensive – becomes much faster and simpler when the work is electronic and easily shared.

Although the researchers felt that the ranking system they’d used had benefits in terms of consistency (reliability?), they didn’t really go into much detail about the criteria that they’d based their judgements on – you got the impression that they’d looked at the students’ work as a whole and simply followed their instincts in a rather ‘organic’ way about the depth of process each student had gone through and reported. I don’t doubt that this method would result in a fair amount of consistency between markers, but it does leave me puzzled about the exact nature and content of the ‘summative judgements’ that the students would have received as the output of the assessment process, as I presume they didn’t merely receive a piece of paper with their rank order on! As the researchers said themselves; employers want flexibility, initiative and the ability to collaborate, and we need to assess those capabilities. Were these examples of the type of criteria the researchers were basing their comparative judgements on? Did they make these explicit from the start – to the students and/or to each other? Were they referred to in the ‘summative judgements’ that resulted? Did they feel that this was primarily a norm-referenced assessment, or a criterion-referenced one? I think I’d really like to have a conversation with the Goldsmiths researchers about this – I have so many unanswered questions! However, they’re probably really busy people so I guess I’ll just have to read the project report…

Still thinking about this 2006 video from Teachers TV: http://www.teachers.tv/video/5431

Having already touched on the challenge of gaining stakeholders’ trust in the capability of e-assessment in my previous post, there were a few more issues raised in the programme that were highly relevant to the topics we’ve discussed so far in this unit, and also to my own experience:

One central theme of the programme was validity of assessment; the narrator begins by questioning the validity of assessing students in isolation by means of pen and paper examinations. As Hal Maclean from Ultralab states; “Society has moved on since this was a valid way to test children”. I would add adults to that as well!

One example given in the programme of how technology is enabling dramatic change in assessment practices is the use of oral assessment via mobile phones. The researchers felt that it might be more appropriate for young people to express what they’ve learned orally, through a medium that they’re familiar and at ease with (which makes sense). This form of assessment also offers the learner choice about and where to complete the activity. As an aside, it was interesting that the researchers found that students had a preference for a female robotic voice – it makes sense that the impersonal effect created by a robotic voice might reduce performance anxiety.

We’ve certainly spoken a lot about the validity and fitness-for-purpose of timed, written tests throughout this unit. One can see how being able to recall information, explain things and justify one’s actions orally is important – and, as adults in the world of work, whenever we have to do this in an instant manner it tends to be orally. Of course, being able to show understanding and justify one’s actions in writing is also important, but one rarely has to do this rapidly, in isolation and without anything to refer to.

Other alternatives to the pen-and-paper medium include electronic content submitted via PDAs and other mobile devices, and computer-based tests.

Computer-based objective tests not only offer massive gains in terms of efficiency of marking and quality of statistical output, but they also allow tests to be personalised. Different questions to test the same skill mean that students cannot copy from each other, and adaptive functionality ensures that students are appropriately challenged and tested at the right level. There is also the potential here, as with most forms of e-assessment, for the student to choose when they want to be assessed. Computer-based testing also enables the provision of instant feedback. These are key ideas that we’ve discussed within the unit to date – reliability, differentiation, personalisation & instant feedback. I liked the example given in the programme about computer games; that children & young people play them because they are challenging, and they can immediately learn from their mistakes and correct them to progress quickly. It’s very interesting that computer games generally get quite a bad press among parents and teachers – perhaps some of this could be (subconsciously) fuelled by envy of the games’ ability to capture their child’s or pupils’ attention far more effectively than they are capable of doing…?!

We currently offer objective computer-based tests on the ICM programme for our students to test their understanding of key concepts within the subject area of their study modules. Some of them I would judge to be ‘better’ than others – i.e. the ones that have adaptive capability, and provide more detailed feedback with a prompt to try again. Perhaps if/when we get more human resource allocated to online tutoring we’ll be able to make wider use of these formative assessment opportunities. The personalised assessment tasks I mentioned in my previous post could also be designed as low-stakes formative assessment with instant feedback.

More later… still have some ideas to come about ranking over numerical marking, and the assessment of process through PDAs – more of the topics covered in the programme…

So we’re looking at e-assessment this week – hurrah!!

Our ‘homework’ was to watch this programme from Teachers TV http://www.teachers.tv/video/5431, which gave several examples of innovative technology-enabled assessment practices.

Overall I felt that this gave a particularly balanced view of e-assessment; a clear picture of the benefits but also a word of warning about how challenging it is to gain credibility and trust for these new assessment practices within society – and I imagine that this, in addition to a lack of functional connections between educators and programmers/technologists, is one of the primary reasons why things have moved on so very little from when this programme was first broadcast nearly three years ago. How exactly do we overcome the natural conservatism among generations who’ve been through paper-based examination systems, and accusations of ‘dumbing down’ and ‘cheating’? I’m personally facing these very issues as I type, since creating a VLE for an electrical engineering DL programme that’s holding on to very traditional assessment methods. As expected, the students are using the discussion forums to ask for and give help on the sample test questions – a system that seems to be working beautifully, and the immediate uptake with little staff encouragement has proved that the students felt there was definitely a need here. However, staff are concerned that, as the students are now ‘talking to each other’, that they’ll be able to ‘cheat’ on the summative tests, which are effectively open-book pen-and-paper exams. In solving one big problem through technology – student isolation – we’ve created another. Whether this makes the old assessment method significantly less reliable or valid than it was before is uncertain, but if the programme team wish to keep using this assessment method, it may be worthwhile using a system that creates personalised tests, similar to the primary maths tests used as an example in the Teachers TV programme. Mark Russell at the University of Hertfordshire has been using a similar system for frequent low-stakes formative assessment in higher-level Engineering courses that he developed several years ago. I invited Mark in recently to demo his system to our programme team, who were quite interested. However, I still feel that there is a fair amount of inertia to overcome, in that the team needs to see a working product before they consider giving it a go. Am I, with my natural & social sciences background, capable of setting up a personalised assessment programme for an M-level electrical engineering unit? I guess, working in the Engineering faculty, I’m bound to be able to find someone who can give me a hand..!

There were plenty more issues raised in the programme that were relevant to what we’ve already covered in the assessment unit, and also to my own experience – see the next post!