Assesslog

In preparation for last week’s session on Assessment & Creativity, we read a range of sources related to the assessment of process.

Jill Porter’s article on promoting self-assessment with pupils with severe learning difficulties highlighted how central self-assessment can be in the assessment of process; if students are evaluating their performance, reflecting on what they know and setting targets, they are documenting the process as they go. The author suggests a number of tools and technologies for aiding the reflective process – the use of mirrors and cameras to assist self-assessment; circle-time and plan-do-review boards that introduce an element of peer-assessment, and one-to-one sessions to enable teacher assessment. The ideas presented in this article, although in the context of the primary sector, are also relevant to higher education; giving students the opportunity to video face-to-face groupwork, or using mirrors to enable students to sit outside the group and perform a peer-assessment, would both be of benefit to the self- and peer-assessment of process.

Another source we were given to read was a student assignment on the contradictory nature of assessment of the arts. It challenged the viewpoint that objective assessment is a desirable goal, suggesting that any truly objective assessment of the arts can at best only measure ‘technical proficiency’. The ideas presented here reminded me of those of Elliot Eisner, evident in The Art of Educational Evaluation (discussed in an earlier post) as he challenges the setting of educational objectives prior to the learning process.

This piece of writing explained the concept of legitimation of exclusion in a way that I found much clearer than in the Broadfoot article I read previously, noting that, in today’s society, failure is as much of a necessity as success. The author of this piece emphasised, as others have, the weakness of the correlation between exam success at school and success at university and in general employment; however, the fact remains that selection has to occur, and therefore we need something to base that selection on. But it does seem logical that we are compromising pupil and student confidence and success in favour of objectivity and comparability.

I found the author’s reference to the Mcnamara Fallacy thought-provoking. It struck a chord with me as I am currently trying to reconcile the assessment of group discussion within a framework for module assessment in the ICM programme. Not providing credit for these activities does appear to give some students the impression that they are less important that the credit-bearing activities – it’s interesting to look at the fallacy from the perspective of the person who is being measured!

The central conclusions of this text were that teaching students to reflect, evaluate and recognise creation will assist the creative process, and the assessment of that process too, but creativity itself cannot be ‘pre-ordered’. A combination of discussion, feedback, ipsative assessment and self-evaluation is required for the valid assessment of a creative process. The final assessment should draw heavily on the student’s own reflections and self-evaluation, which also helps to foster independence and self-discipline.

The third piece I read was a foreward to Anna Craft’s 2005 book, ‘Creativity in Schools’, by Tim Smit, co-founder of the Eden Project. This drew briefly on a number of ideas, such as the link between emotional development and observation and their contribution towards creativity, and the differing notions of creativity between cultures. I found particularly interesting the idea that creativity in problem-solving is evident in all cultures, but creativity in the arts is generally in inverse proportion to the power of the state and degree of atrophy within a culture. The author also touched on the concept of the ‘throwaway culture’ and suggested that this is a symptom of our constant quest for the new and innovative. Finally – I liked the definition of creativity given here as the interface between self expression (a person’s ‘unique voice’) and the outside world. This surprised me as, to be honest, the wording is far ‘fluffier’ than what I would normally feel at ease with – I might have to explore what this definition means to me in more depth later on.

The final text I read was ‘Assessing the Creative work of Groups’ by Cordelia Bryan, a chapter from the 2004 publication ‘Collaborative Creativity’ (Miell & Littleton, eds). The context of collaborative group work is highly relevant to the work I’m doing at the moment with the ICM programme, and I took some valuable ideas away from this reading. Bryan emphasises the need to prepare students for collaborative groupwork by building trust and understanding of group dynamics and techniques, and explained why this is best carried out within the context of a standard unit of study rather than a separate ’study skills’ unit. The challenges of allocating credit for group tasks were also explored, and the importance of self and peer assessment – on both achievement of the task and facilitation of the process – was emphasised. Bryan advises that students should be given the opportunity to complete the entire cycle as a learning process, and to share their perceptions of assessing and being assessed on groupwork, before a credit-bearing assessed activity takes place. She also suggests that criteria and grading methods should be discussed and agreed upon after the students have experienced the initial groupwork activity.

One of Bryan’s ideas I found most interesting was the use of ‘peer observers’ – students who observe another group working and make notes in response to key questions, and then feed back to the group on completion of the group task.

For Monday’s session, Eva suggested we read Chapter 3 of Gipps’ Beyond Testing: Towards a theory of educational assessment (1994), and reflect on the ways in which assessment, evaluation & testing impact on students’ achievement & motivation.

Reading this article caused me to think more deeply about the nature of achievement. It’s often the case that the abilities that are tested (i.e. the ones that are easiest and most economical to test), become the ones that are the most taught; there is a bias against teaching those skills that are not, or cannot easily be, measured. Therefore, when a student appears to achieve well, one has to consider the scope of the test’s validity, and the constructs or skills that it is actually measuring.

Gipps highlights Heisenberg’s uncertainty principle – that one cannot measure things without affecting them – and describes the concept of construct-irrelevant variance, or ‘test score pollution’, where test scores may rise due to teaching being focused on the test items and formats themselves, rather than on the constructs or skills they are intended to measure. This is something else to consider; whether an apparent increase in achievement is due to a greater understanding of a construct or better grasp of a skill, or largely due to students being more highly skilled at taking tests.

It is difficult to imagine how the negative effects of teaching to the test can be minimised with the existence of high-stakes testing. As Gipps points out, teachers feel that they have a professional duty to give their students the best chance to pass those tests that will have a significant effect on their lives. Gipps asks whether even a positive impact such as that created with the move from ‘O’ levels to GCSE can become corrupted over the years.

Reflecting on these points helped me to see my own experiences with a fresh perspective. I gained nine ‘A’ grades at GCSE but received a ‘D’ grade in my English Literature examination. Initially I was surprised and disappointed, and felt that the test must be at fault, but over time I began to see the D grade as an absolute and valid measure of my ‘achievement’ in this area. Gipps’ description of test results as ‘a useful but fallible indicator of achievement’ is helpful. It encourages me to examine what exactly my English Literature examination was testing. One might assume that the aim of an English Literature assessment would be to assess a student’s appreciation of good literature. However, this is not an easy quality to measure. The ability to unpick a metaphor, however, is a much easier quality to measure, and I suspect that whoever wrote my English Literature paper was thinking (hoping?) that measuring someone’s ability to deconstruct metaphor was the same thing as measuring their ability to appreciate and enjoy literature – as Gipps phrases it; generalising to other measures of the same construct. Ah well – at least they picked a measure that was resistant to pre-test coaching (not that my English teacher didn’t try). I felt thoroughly prepared (i.e. coached) for the other nine exams, so I’m not suggesting that those results are any less fallible. Society, however, will still simply see me as a ‘high achiever’ in these subjects and a ‘low achiever’ in English Literature. Gipps summarises the conclusions made by Madeus (1988) - that we as educators need to lower the stakes of tests, and also to try and persuade everyone that test results are only one piece of the puzzle.

Taking the latter point - it seems clear to me that having more detailed reporting of results is crucial in order to discourage the making of simplistic assumptions about achievement – this relates to the issues of unidimensionality and universality discussed in my earlier post on Assessment Paradigms.

Looking now at the issue of motivation (which is inextricably linked to achievement), and how it can be affected by assessment/evaluation/testing, the way forward seems a little clearer, particularly with low-stakes assessment; what Crooks (1988) terms ‘classroom evaluation activities’, as I’ll go on to describe below. With high-stakes assessment we have a difficult conflict between the external motivation experienced by students who believe they can succeed and therefore obtain the rewards, and the demotivating effect on those who know they cannot.

According to Crooks (1988), classroom evaluation activities serve to emphasise the skills, knowledge and attitudes that are valued, help to structure approaches to study and consolidate learning, and affect the development of enduring learning strategies. Crucially, when classroom evaluation is used within a framework of attainable sub-goals, each with clear criteria, it affects self-perceptions of competence (self-efficacy), which is shown to be closely related to the use of deep learning strategies and the ability to persist with challenging tasks. The term ‘classroom evaluation activities’ can also be applied to the input from the e-moderator or e-tutor in the online distance learning programmes I work with. A significant part of my role is to encourage the tutors to engage more deeply with the online activities, and I always see evidence of an instant and powerful impact on the students’ motivation when their tutor provides specific, well-timed feedback on an activity.

Norm-referencing can have a negative impact on motivation; not only can it discourage collaboration and threaten peer relations; it essentially attributes success and failure to ability rather than effort.

The GCSE was intended to emphasise positive achievement (and therefore enhance motivation) by allowing students to show what they could do rather than facing them with impossible tasks. This required the use of differentiated tasks and criteria for different levels of expected achievement, which could unfortunately have a demotivating effect on those who realised that, however hard they worked for their maths GCSE, for example, they would only be able to attain a maximum B grade. However, the changes in content and teaching brought about by the move from ‘O’ level to GCSE, particularly in subjects such as MFL, resulted in a massive increase in the number of students choosing to progress to ’A’ levels.

I’m off to the gym now – but afterwards I’m going to look into classroom evaluation practices more deeply by reading Terry Crooks’ 1988 article on the impact of classroom evaluation practices and reflecting on how the key points he makes relate to my own context of online learning environments :-)

The third text Eva asked us to read in the first week was on the social purposes of assessment – from Education, Assessment & Society: A sociological analysis, by Patricia Broadfoot (1996). This one took me a while to read – I tend to write my questions in the margins when I don’t understand what I’m reading, and it seems that Pat Broadfoot tends to write something first and explain it later, which means I used up a lot of ink.

What Eva asked us to reflect on while reading this text was: “How does the social purpose of assessment impact on the content of assessment and on the teaching content?”

I found the scope of this text very broad – looking as it does across cultures and centuries – but I felt that it provided me with a better insight of the historical and cultural forces that affect the educational system, and a further realisation of the stregth and complexity of the ties between assessment and learning.

In answer to the question of the impact of social purpose on assessment, a key theme of the paper was the differentiation of roles in what Broadfoot calls ‘complex’ societies, and the differing social status of these roles leading to competition and therefore the need for selection. Broadfoot attributes the existence of free competition to historical periods of upheaval such as the Enlightenment, which gave rise to inidvidual rights, scientific advancement and - most importantly – meritocratic ideologies, and the industrial revolution which brought about the need for formal certification of skills, and a move towards contractual employment. Selection for such roles requires the measurement of attributes against a standard; hence the need for standardised tests. The issues of status and power were also discussed in some detail; Broadfoot raises an interesting point in that the high status of certain roles permeates into the actual assessment methods through which entry to those professions is sought.

Broadfoot argues that assessment procedures may not be reinforcing the knowledge and skills that are actually required by society and the economy, but an arbitrary representation of knowledge selected by those in positions of sufficient power to determine what kinds of intellectual activity have the highest status.

Broadfoot brings many factors into explaining the pervasiveness of the written examination in our education system; the historical and cultural reasons for its dominance and the effect this has on the curriculum.  Two key ideas Broadfoot highlights are the reliability of the written examination (important for comparability when dealing with the pressure of large numbers and the need to select), and its links with the testing of innate general ability, which countries like the UK and the US still believe in as a predictor of future achievement. Broadfoot argues that the alternate view that achievement is the result of persistence is one of the reasons for the success of the Japanese education system. I’m tempted to think that perhaps neither view is wrong and that success is largely due to an innate ability (& desire) to persist – but I can see that putting the emphasis on persistence opens up the possibility of success to all. 

The second core reading of the week I’ve just thoroughly enjoyed digesting was Chapter 2 of Elliot Eisner’s The Art of Educational Evaluation, titled ‘Educational Objectives: Help or Hindrance?‘.

Before I started reading the chapter, I read a bit of background about Eisner, his life and his work - and I’m glad I did as it helped me to get an idea of the context in which he was writing. Eisner’s educational origins in the teaching of art history give some insight into his views on educational work as a form of creative artistry, and the parallels he makes between education and art.

To me, this article was quite a radical deconstruction of the assumption that the curriculum design process has to start with the statement of educational objectives, and to use a metaphor (which I generally don’t), it rocked my foundations a bit. In bish-bash-bosh format, Eisner’s argument against starting the curriculum design process with the statement of educational objectives is as follows:

1. We can’t accurately predict educational outcomes
2. Subject matter has a significant impact on precision in stating objectives.
3. Educational objectives are often assumed to be standards for measurement when in many situations they can only be used as criteria for judgement.
4. Teachers often start with means (’what am I going to do’) and then relate them to ends (’what am I going to accomplish’), and this (according to Eisner) is logical.

At first I felt that the latter point was a pretty brave thing for Eisner to stick his neck out and say. But then I thought about it some more, admitted to myself that that’s the pattern I, and my colleagues, tend to work in, and thought that maybe it’s not because we’re all lazy, incompetent bunglers – maybe it IS logical. Whenever I’ve tried doing it the other way round – identifying an objective and then thinking up an activity that will result in that objective being met - it’s not that it’s impossible, but creatively it’s a lot harder and, to be brutally honest, the activities often aren’t as much fun  

Eisner’s argument really spoke to me, although I must admit I still see the rationality behind the argument for starting with objectives – I trained as a personal fitness trainer several years ago, and the best way to get clients in shape (and, frankly, to get clients in the first place) is not by suggesting ’lets run around and lift some weights and see what happens’. But I’m willing to concede there may be subtle differences between education and physical training (on reflection, the physical training metaphor seems a very good description of the traditional approach to setting objectives – and the theory of specificity of transfer too!).

More later…

Eva suggested we have a look at the first chapter of Gipps (1993) – Beyond Testing. As expected from an introductory chapter, this text threw out a bundle of interesting concepts to grapple with.

I was startled to be confronted with the concepts of construct validity and factor analysis, a year after finishing my Research Methods assignment and feeling relieved that I’d never have to think about them again (yes, I was delusional). The paradox presented by the assumptions of universality and unidimensionality was probably the muddiest point of the chapter for me; however, I think the points that was being made were: 1) as many attributes (e.g. reading) are multidimensional, attempting to assess only one dimension of an attribute (e.g. comprehension) is illogical, and 2) if multidimensional attributes are being tested, two identical scores (for example) will not tell the same story. These points fit with the concluding paragraphs of the chapter, where it is implied that (regardless of the purpose of the assessment, I think), both the assessment and the presentation of the outcomes should ideally be in a format that gives a fuller picture of exactly what competences pupils and students have – notwithstanding the fact that a student’s learning will not necessarily fit neatly into an ordered set of performance descriptors. Gipps uses the National Curriculum as an example, which I found rather amusing – I remember playing at a friend’s house – it must have been around 1991 - and listening to my friend trying to explain to her mother what her recent maths scores meant. Her mother just wanted to know a) a percentage and b) where her daughter was in relation to the rest of the class. To be honest, that was all we wanted to know at the time as well! Recalling this made me realise just how pervasive such paradigms can be.

This first chapter focused mainly on the different purposes of assessment – e.g. monitoring, accountability, supporting learning, and the fitness for purpose of different styles of assessment. It makes sense that tests and exams used for monitoring and accountability at a class, school or national level will need to be quick, manageable and reliable. If assessment is to be used for the purpose of certification, there will need to be more of a balance struck between efficiency and depth/detail. However, assessment used on an individual level, for the support of learning, is a rather different kettle of fish – it needs to be detailed in quite a granular way so that individual strengths and weaknesses can be identified and ‘executable advice’ (Glaser 1963) can be given, and it needs to measure competence (best performance) rather than average performance. There are evidently problems with reliability in this kind of developmental assessment, and Gipps cites the work of Stiggins (1992) in suggesting that the differences between the types of assessment are so fundamental that attempts should never be made to blend the two.

Gipps highlights quite frequently that, as different assessment practices (via their effect on teaching) encourage different styles of learning, it’s vital to consider the kind of learning that we wish our students to achieve when choosing or designing a method of assessment. Gipps cites the work of Goldstein (1992) in reminding us that assessment also impacts directly on students’ learning to some extent – students never pass through the assessment process totally unscathed – it will always result in some kind of learning. Gipps uses the UK SATS as a shining example of authentic assessment (sometimes called ‘performance assessment’) that models the learning activities taking place, and therefore doesn’t lead to so much distortion of instruction. My year group was the first year to sit the Year 9 SATS. I remember feeling particularly bereft at the lack of ‘coaching’ that took place beforehand – we were so used to being given past papers for homework, practicing in exam conditions, etc – we really felt the difference and it wasn’t comfortable for us at all to not have what we saw as formal preparation for the assessment process itself. Many of my peers did rather badly, simply because they weren’t used to being asked to think rather than to recall - they assumed the questions were asking them to regurgitate some sort of prior knowledge that they didn’t have, so they left it blank and moved on.

I’d be interested to hear about anyone else’s experiences of the National Curriculum, and SATS, and how they resonate with the concepts that Gipps presents…!