Policy Development Workshop – 21st Century Assessment

17/10/2014
On 15 October AQA held the third and final policy development workshop as part of our major project The future of assessment: 2025 and beyond. The event, chaired by AQA’s Head of GQD Business Development Marc Booker, was based on the theme of this event was ‘21st Century Assessment’, discussing the role technology could play in assessment in the future.

Workshop

This theme aimed to acknowledge that although technology has altered a significant proportion of our everyday lives, its potential in terms of assessment and its ability to transform the way we assess knowledge and skills, so far remains largely unknown. We wanted our speakers and guests to explore how technology could allow us to improve the reliability and validity of assessment, and at the same time make assessments more relevant for the modern world.

Our first speaker was Professor Angela McFarlane, Chief Executive and Registrar of The College of Teachers. She spoke about the “digital generation” of young people who now spend more time online than they do watching TV. She posed some questions and challenges for our attendees to consider, such as whether or not teachers were thought of as “reliable custodians of high stakes assessment”, and talked about the idea of children often having to “power down” when they came to school.

Our second speaker was Lisa Gray, who is Assessment and Feedback Programme Manager at Jisc, a charity which champions the use of digital technologies in education and research. She explained some of the research surrounding assessment and technology, including the widespread recognition that traditional assessments have not been adequately preparing students for the world of work. She discussed the idea of “feed forward” rather than feedback, enabling learners to progress further, and finished by emphasising the role technology-enhanced assessment can play in improving the employability prospects of young people.

Following the speakers, the guests discussed some of the points raised by Angela and Lisa, and contributed some of their own thoughts about the topic. A main theme which emerged was the question of the gap between what traditional exams can reasonably assess, compared to the things industry, commerce, or universities would like to see in applicants. There also needs to be a degree of flexibility, as different destinations for students will require different skills. Additionally, there were comments regarding the purpose of assessment – if taken as part of the accountability regime, is there a way in which we can use technology to overcome the pressure placed on schools, teachers, and students and to create a new way of holding schools accountable?

Our audience, which was made up of professionals and practitioners from across the education and skills sectors, split into two groups and were joined by our speakers for smaller, more focused discussions. We chose two key areas – 2025 workforce skills, and the purpose of using technology-enhanced assessment.

One of the main conclusions reached by both groups was that increasingly, the skills which are required by employers or universities are not necessarily summative skills, and therefore cannot be measured or assessed by traditional summative exams. The process of developing these skills is ongoing, and there should be a form of assessment which reflects that; something which technology seems capable of doing. Our groups were also in agreement over the need for technology to enable assessment across the full breadth of the curriculum – with some subject specific assessment such as science practicals, and some more generic skills such as collaborative group work. Additionally there were several comments that so far, most of the technology used in education and assessment has been repurposed from other things. There are strong arguments to develop new technologies specifically for the purpose of assessing in schools. Finally, we closed by discussing the need for a systematic change regarding the nature of schools, and the teacher-student relationship, in order to truly embed technology within the assessment system. .

This was the final workshop in our series of three – read about our two previous workshops; Balancing assessment and accountability and Assessment for the real world. This marks the end of phase 2 of The future of assessment: 2025 and beyond. There will be further updates about the next stage of the project later this year, but if you would like further information or would like to comment about the project so far, please email kzsmith@aqa.org.uk.

Our thanks go to our speakers and guests who helped contribute to such an interesting discussion.

Policy Development Workshop – Balancing Assessment and Accountability

10/07/2014On 8 July AQA held the first policy development workshop of our major project The future of assessment: 2025 and beyond. The theme of this event was ‘balancing assessment and accountability’.

This theme was intended to provoke discussion about the purpose of assessment, how it is currently used and how it might instead be used in the future. Additionally, we wanted delegates to consider accountability in England, and what would be the ideal system if we were free from any current constraints.

Tom Sherrington, Headteacher at King Edward VI Grammar School in Chelmsford, was the first speaker. He talked about the difficulty in comparing similar grades across different subjects, and advocated a move to a baccalaureate model to give an equal standing to different types of qualification.

Next, Laura Dougan and Margaret Miller from the Scottish Qualifications Authority (SQA) spoke and explained the learner-centric assessment system in Scotland. They discussed some of the recent changes to assessment in Scotland, including the greater involvement of teachers and the increasing amount of internal assessment, as well as the new Insight online benchmarking tool which will be used by schools to identify areas for improvement.

Invited guests from across the education sector then discussed potential policy objectives in small groups, joined by our speakers. Each group debated the points raised by the speakers as well as contributing their own hopes and expectations for the next 10-15 years in assessment and school accountability.

Seminar One

Most of the groups spoke about a ‘triangle’ of validity, reliability and accountability, and the struggle of balancing all three of these. Some key desires which emerged were for Ofsted to be more collaborative with schools and teachers and not to focus so much on data; for assessment to follow curriculum design and not the other way around; and to see a separation of assessment and accountability.

Our main aim was to reach a consensus on the three most important issues relating to the balance of assessment and accountability in schools, and the three points below were all discussed at length by everyone at the event.

The groups all agreed that they would like teachers to be more involved in assessing, with some delegates suggesting there should be greater professionalisation of teachers as assessors, similar to other countries such as Germany. This is linked to a desire for more trust in teachers to be able to effectively teach and assess their students.

One theme which emerged was the hope that accountability would move away from being based on exam results, with some agreeing with Tom’s idea of a baccalaureate style qualification, and others thinking that expanding accountability to include destination would be a more appropriate measure.

The groups also decided accountability should be about improving and progress, with one group suggesting schools should be held accountable to their own set of broad aims, which could include assessment but crucially other measures as well.

The event was the first of a series of three events to be held over the summer and autumn. The next two events are on the themes of Assessment in Other Sectors, and 21st Century Assessment. If you wish to attend either of these events and contribute to the project, further details can be found on this blog post.

Our thanks go to the speakers and guests who attended.

Phase 2: Developing Policy

20/06/2014Over the last few months, AQA has asked UK and international experts, teachers, academics, policymakers and employers what they think the future of assessment could and should look like over the next 10-15 years. This collaborative project has so far taken the form of blogs and videos by key thinkers in the sector, as well as roundtable events and debates involving teachers and school leaders.

From these “blue skies” discussions, we have identified three themes which were repeatedly raised as being the most important areas for development and reform over the next 10-15 years. The second phase of the project will be a series of three policy development workshops. Teachers, policymakers and other stakeholders are invited to join us to discuss each of the themes and formulate policy objectives in each area.

The three areas we will be focusing on are:

Balancing assessment and accountability – Tuesday 8th July
How can we achieve good assessment and good accountability at the same time?

Speakers:
Tom Sherrington, Headteacher, King Edward VI Grammar School, Chelmsford
Margaret Miller and Laura Dougan, Policy Managers, Scottish Qualifications Authority

Assessment for the real world – Wednesday 10th September
How can vocational education and training be reliably and validly assessed?

Speakers
Professor Prue Huddleston, Fellow and formerly Director of the Centre for Education and Industry, University of Warwick
Professor Jen Cleland, John Simpson Chair of Medical Education, University of Aberdeen

21st century assessment – Wednesday 15th October
How can technology make assessment more relevant, practical and valid?

Speakers:
Professor Angela McFarlane, Assessment Co-Chair, Education Technology Action Group
Lisa Gray, Assessment and Feedback Programme Manager, Jisc

We will invite expert speakers to provoke debate, and the events will be open and interactive, with everyone having a chance to discuss their thoughts and develop more detailed thinking in smaller groups. The aim of these events is to create three long-term, achievable, policy objectives for each of the themes, which we’ll publish here following the workshops.

The three workshops will be held from 6pm to 8.30pm in central London, and a buffet supper will be provided. If you would like to register interest in attending please email jwilson@aqa.org.uk stating which workshop(s) you wish to participate in. We look forward to seeing you there – you can also follow the debate at www.aqa.org.uk/assessment2025 and on Twitter #assessment2025.

Can and should young people play a role in designing assessments?

 

09/06/2014Professor Jannette Elwood, Professor of Education, Queen’s University Belfast

At the time of writing this blog, students are once again in the thick of the examination season, having to navigate one of the most pressurized times in their educational lives. So while they are sitting up to 2 examinations a day and maybe 4 or 5 examinations a week for the next while – I think it is a very apt time to consider whether they can and should play a role in designing the assessments that they sit.

My short answer to the above question is yes! Young people can and should play a role, not only in how assessments are designed but also in how policy around assessment change is debated, consulted upon and final decisions made.

My answer to the question is based on two factors. Within international legislation – the UN Convention on the Rights of the Child (UNCRC), to which the government of the day is a signatory – children and young people are classified as rights holders and as such are recognized as being entitled to engage in processes that affect them directlyincluding the development of policies and services (in this instance educational ones) through research and consultation (Elwood and Lundy 2010). And, of course, they are key stakeholders in any future assessment systems and, as such, are as important as any other group of people (assessment experts, teachers, head teachers, parents, employers, government) to have a say about what assessment and qualifications systems will look like and entail.

We know at present the landscape of qualification reform in England, Wales and Northern Ireland is one of uncertainty and fragmentation. What is clear is that proposed changes to GCSEs and A levels across the nations of the UK will have major (and differential) ramifications for young people and their future educational and employment careers. Likewise, we are seeing major changes to assessment systems for lower secondary and primary school children and have yet to see the full impact of international testing regimes (such as PISA) and their effect on the educational experiences generally of children across all ages. Yet what is also clear is that there has been a limited history of children’s and young people’s participation in the area of assessment generally and relatively none within policy formation or qualification development. Their input is decidedly missing from any meaningful engagement about the current round of proposals and decisions concerning qualifications reform. This, I would argue, is a significantly missed opportunity to see the bigger picture – the full extent of assessment policy change and its impact.

How do we know that young people have something worthwhile to tell us about assessment reform? Well, we are beginning to know a great deal more about what they think about education generally and from national research that has engaged with young people on these issues (Elwood 2012 and 2013), we are beginning to know a great deal more about what they think, not only about qualification reform, but also about not being consulted about such significant and high level debates. This lack of participation in any decision making on these issues begins to make young people feel more like victims of assessment policy reforms rather than beneficiaries.

So what do they have to say? In talking to nearly 250 young people from across England (Elwood 2012), they told us that yes examinations do dominate their lives in school but that they enjoy them if they are well prepared and know that they won’t get far without good grades. They also thought that examinations structured through modules (and re-sits) allows for any mistakes to be made better, takes the stress off having to do everything in one sitting and that it is only fair to all young people to have a mixture of examinations and coursework – ‘we don’t all like the same things.’ However they felt there was too much confusion about the ‘worth’ of all the different types of qualifications offered and that actual grades were being devalued – with A* taking over as the status of excellence, but not everyone could achieve that. They also felt insulted at the annual circus of the ‘standards are falling’ debates feeling their achievements were degraded; obtaining good grades in whatever qualifications they sit is tough and not getting any easier. They also wanted to know why changes to examinations are introduced ‘live’, i.e. in to their examinations sessions, where their future successes might be ‘messed-up’ if these changes haven’t been piloted in advance; such changes can have considerable impact on their final grades and that is too high a price to pay.

Thus young people have a lot to tell us about both the positive and negative impact of assessments on their lives – we should be listening to them more on these matters. So how might policy makers and assessment developers engage with young people more effectively? There are a number of ways in which this can happen such as: focused policy briefings with education officials and young people in order to obtain input into current debates; examination boards actively setting up panels with young people so that their views can be fed directly into assessment design and implementation; and strategies for consultation involving social media that speak directly to students to gauge their opinions. Good practice, however, and especially rights-based approaches – would mean that we would ask young people what they think is the most effective way to engage with them directly and then change our practice accordingly.

Including young people in decision-making about the future of assessment design or implementation will not be easy; nor will it provide the answer to all the problems that they and we, as assessment professionals, face. But they are authoritative on these matters, just like us, and have a great deal to offer in terms of how they see the world, what they see as salient in their education, what will be of benefit to them and how such future assessment systems could and should be developed with a consideration of their best interests firmly centre-stage.

References

Elwood J and Lundy L (2010) Revisioning assessment through a children’s rights approach: implications for policy, process and practice’, Research Papers in Education, 25: 3, 335 — 353. http://www.tandfonline.com/doi/abs/10.1080/.U42eghb9Mtg

Elwood J (2013) The role(s) of student voice in 14-19 education policy reform: reflections from students on what theyare, and what they are not, consulted about, London Review of Education, 11(2), 97-111.
http://www.tandfonline.com/doi/abs/10.1080/.U42exRb9Mtg

Elwood J (2012) Qualifications, examinations and assessment: perspectives and views of students in the 14-19phase on policy and practice, Cambridge Journal of Education, 42: 4, 497-512.http://www.tandfonline.com/doi/abs/10.1080/.U42eUhb9Mtg

“Without Trust, We Cannot Stand”

13/05/2014Leora Cruddas, Director of Policy, ASCL

The ASCL and AQA debate on the future of teacher assessment raised a number of interesting ideas: Fiona Millar’s fascinating thoughts about what an educated 19-year old looks like; Ian Bauckham’s call for chartered assessors in schools; Patsy Kane’s testimony that teacher feedback is one of the most powerful motivators of progress in students’ learning; and Andrew Hall’s call to rebuild trust across all components of the education profession. I’d like to develop Andrew’s important theme.

It is easy – and politically expedient – to say that teachers can’t be trusted. It is an altogether much more politically courageous thing to build trust. We need political leadership of this calibre: leadership that builds strength and capacity and inspires great leadership in others. We are after all, talking about our children’s future and the economic future of the country. What our education system needs and has always needed is great leadership, both political and professional.

Interestingly, the public does trust the teaching profession. A 2013 Ipsos Mori poll showed that 86% of respondents trust teachers, while just 18% trust politicians to tell the truth. At the debate, around 90% of those present said that teachers are at least somewhat effective at formative assessment, while more than half felt that successive governments have been responsible for a breakdown in trust. But, as Charles Handy says, “we cannot wait for great visions from great people, for they are in short supply at the end of history.” There is a moment for the profession to seize the day and say what we think needs to happen to create a good assessment system.

It is now more than ten years since Onora O’Neill delivered her excellent Reith lectures on A Question of Trust. She said then: “Trust often invites reciprocal trust: and when it does, we have virtuous spirals. Equally trust can open the door to betrayal and betrayal to mistrust: there are vicious spirals.” We want to build virtuous spirals of trust across all components of the education system in order to create the very best education system for our children.

In assessment terms, as Patsy Kane said, this is likely to come from schools collaborating to build professional capital in assessment. There is probably nowhere better to start to build virtuous cycles than in the field of assessment, which is at the centre of teachers’ professional practice.

Finally, as part of the Great Education Debate, and ahead of the election in May 2015, ASCL calls on all political parties to build a culture of trust and confidence in our education system. We must create cross-party agreement on a vision for our education system that enables, motivates and inspires our children to develop as rounded and grounded (to quote the CBI) and take their place in the global society.

Trusted, professional, expert: teacher assessment in 2025

13/05/2014Dale Bassett, Head of Public Policy, AQA

Over the course of the Future of assessment project so far a number of topics have repeatedly arisen in the discussions, events and writing that have contributed to the debate. Teachers’ role in assessment has been one of these, and has also been an important theme in the Great Education Debate convened by the Association of School and College Leaders.

High-quality teacher assessment is essential, not just to improve teaching and learning but also to improve the validity and reliability of summative assessments. Yet concerns over inconsistent quality and the impact of pressures from the school accountability system have led to a decreased role for teacher assessment in qualifications.

On 10th March AQA and ASCL brought together 60 school leaders and teachers with other experts and commentators to consider the issues of trust, teacher professionalism and teacher expertise. The debate, held at Whalley Range High School in Manchester, addressed the questions of restoring trust between teachers and government, enhancing teachers’ skills in assessment, and how better teacher assessment could be a central aspect of the assessment system of the future.

One of the big challenges when discussing teacher assessment – in fact, most topics in education – is that preconceptions and assumptions can often play a bigger role in influencing thinking than facts and evidence. We decided to see what people’s views were before the discussion, and then reassess to see if the debate had changed minds, so we asked the participants a series of multiple-choice questions relating to reliability of marking, trust in teachers and government, formative assessment, and improving the assessment skills of teachers, which they answered using electronic keypads.

The video below shows some highlights of the debate. Four panellists offered thoughts to inform and provoke before the debate: journalist and campaigner Fiona Millar; Ian Bauckham, President of ASCL; Andrew Hall, AQA’s Chief Executive; and Patsy Kane, head of Whalley Range High School. So what were people’s views – and how did the debate change their initial ideas?

Chart 1

Teachers seemed more confident in their ability to reliably assess coursework and controlled assessment before the discussion, in which Fiona Millar and Andrew Hall both addressed the challenges of ensuring reliable teacher assessment in a high-stakes accountability environment. This clearly resonated with the participants, with around 10% deciding after the debate that teachers’ summative assessment was less reliable than they initially thought.

Chart 2

Teachers felt from the outside that good quality CPD was the best way to improve their assessment skills. The debate around becoming an examiner turned some of those who had thought this would be beneficial away from this and towards discrete CPD. Despite some in the room saying that they had learned a lot by being an examiner, Ian Bauckham’s reservations about the benefits clearly held sway with some of the teachers present.

Chart 3

The issue of trust was discussed by some of the panellists, with Andrew Hall saying confidence in the system was reliant on rebuilding trust between governments, teachers, exam boards, and parents. Fiona Millar also indicated trust was an important issue, and that parents tended to trust heads and teachers far more than politicians when it comes to knowing what is best for education.

Patsy Kane said that teachers wanted to understand more about the “black box” of the assessment system, particularly grading, and thought that exam boards should work closely with teachers to help them understand the GCSE and A-level reforms, and a clear majority of those present thought this was the most important way to improve trust in the system. Interestingly, very few teachers thought that stronger regulation was the key to rebuilding trust.

Watch the 12 minute video to see highlights of the event:

Can we have better descriptions of performance in our examinations?

18/03/2014Dr Chris Wheadon, Director, No More Marking Ltd.

20… 19… 18… As the British Military fitness instructor counted down my press-ups from 20 to 1, I wondered why on earth I was doing press-ups in the mud early one Saturday morning. His answer was simple: to get better at doing press-ups. Then he gave me another 20 for being cheeky.

We’ve all had this experience as teachers. Why do we have to do this sir? Is it in the test? If the answer is yes, then you can get them to do it sitting upside down in a swimming pool, if that’s how the test will be run. Because by doing whatever it is, they will get better at it, and will do better in the test. We do press-ups to get better at doing press-ups. We do tests to get better at doing tests.

Some users of tests, however, seem to want to know more than our final score or grade or, in my case, the number of press-ups I can do. What does a maths test mean in terms of how good I am at mathematics? Or at doing the calculations I need to design a bridge that won’t collapse? In my case, what does my ability to do press-ups mean in terms of my ability to haul shopping bags from supermarket to home? Already, at this point in my thinking, I can hear my fitness instructor: ‘If you want to get better at hauling shopping bags, start hauling shopping bags…’

International and national assessments tend to begin with grand statements that set up certain expectations. It seems they all aspire to tell us what students know and can do. PISA, the NAEP, GCSEs, all of them aspire to this. Well that is great news! As an employer, can you tell me at what PISA score students will be able to write a letter to a customer inviting him to participate in a research study? As I skip through the first of PISA’s latest 555 page report, I quickly realise I’m in trouble. PISA appears to be able to tell me about mathematics and reading, but not about writing. Not to mention the fact that kids don’t get a score, they get a ‘plausible value.’

Undeterred, I turn to the GCSE. Within 5 minutes I have pulled up the grade descriptors for AQA’s GCSE English grade C:

Candidates’ writing shows successful adaptation of form and style to different tasks and for various purposes. They use a range of sentence structures and varied vocabulary to create different effects and engage the reader’s interest. Paragraphing is used effectively to make the sequence of events or development of ideas coherent and clear to the reader. Sentence structures are varied and sometimes bold; punctuation and spelling are accurate.

Wow! And that is just at grade C! Adaptation! Engaging writing! Bold sentence structure! Imagine what you get at grade A! So if I employ a young person with a grade C in English, can I really expect accurate spelling and punctuation? Before you dismiss such descriptions as some form of dumbing down conspiracy, ask around any English teachers you know. A good teacher will confidently reel off the assessment objectives of the GCSE and A-level syllabus, and will tell you the relative strengths and weaknesses of a piece of writing in terms of those assessment objectives, and in terms of the grades you can expect. At some point they will say, the punctuation and spelling are not grade C level…

So, given the wealth of descriptive data we have around examinations – the criteria, the levels, the grade descriptors, the domains, the constructs – do we really need any more detail? Personally I would add very little. Firstly, returning to the grade descriptor I would simply change the first sentence:

Candidates’ writing against the GCSE English task we set them under examination conditions shows successful adaptation of form and style to different tasks and for various purposes.

I would add this qualification because I know that the candidates will have been prepared for the task in hand, have memorised strategies, mark schemes and model answers, and there is little that can be done to improve how much further we can generalise from their answers. Unfortunately, however good we get at tests, our performance is limited to some extent by the conditions under which those tests are taken.

Secondly, I would make some examination scripts available to all after the examinations. I have a feeling that we may disagree about what constitutes bold and engaging writing. Disagreement on standards is a healthy and necessary debate, which is made healthier by the presence of a good sample of evidence.

Anyway, I must get back to my press-ups. I’ve got a test coming up. Of press-ups. Just don’t ask me to haul any shopping bags, that’s not what I’m working on right now.

Dr Chris Wheadon is Director of No More Marking Ltd. (www.nomoremarking.com)