Policy Development Workshop – Assessment for the real world

12/09/2014 On 10 September AQA held the second of three policy development workshops as part of our major project The future of assessment: 2025 and beyond. This event was based on the theme of ‘assessment for the real world’, looking at vocational and practical education and assessment.

With this theme we hoped to inspire a discussion regarding how to assess skills which aren’t covered by traditional written examinations. Attitude, professionalism, ethics, and teamwork are just some examples of the skills required in many vocational professions, from medicine to beauty therapy. We wanted our guests to consider how these skills could be reliably and validly assessed in the future, and what policies might help to achieve this.

Our first speaker was Professor Prue Huddleston, Professor Emeritus at the Centre for Education and Industry at the University of Warwick. She posed some questions for our guests to consider, including wondering if there was a difference between vocational education and occupational training, and whether assessment should focus on what learners know or on what they can do.

Professor Jen Cleland, John Simpson Chair of Medical Education at the University of Aberdeen, was our next speaker. She is an expert in medical education, and took us through some of the examples of assessment used in medical training and throughout medical careers. She explained the difference between technical skills, such as taking blood pressure, and non-technical skills, including teamwork and talking to patients, and she emphasised the difficulty in measuring and assessing the latter of these skill-sets.

BOB_3218

Following the speakers, our guests discussed some of the issues relating to vocational education and assessment. One of the recurring questions asked was about investment in these qualifications – do we as a society value some vocational professions more than others, and invest more time and money into developing more rigorous qualifications in these areas? There was also consensus among the guests about the need for transferability of vocational skills assessment, particularly for young people who are likely to have several jobs across different sectors.

Our audience, which was made up of professionals and practitioners from across the education and skills sectors, split into three groups and were joined by our speakers for some smaller, more concentrated discussions. We focused on three key areas – what the standards and outcomes of these qualifications should be, who could define these standards, and how can we evidence outcomes to create confidence in these qualifications.

One of the key conclusions the groups came to was that employers should have a role in both developing and defining the outcomes of any vocational qualifications, although this should be carefully balanced with the involvement of education practitioners. However, the groups also recognised that employers are not a single homogenous group, and they have different needs and requirements according to their specialisms. Additionally, our guests thought there should be a greater effort to build more of an evidence-base around vocational qualifications, in order to change their perception and to increase the reliability. One idea suggested which could help evidence the value of the qualification was to use destination data to show the progression of students who had completed it. The use of technology could also play a role in this, as a means by which students complete assessments designed to test skills which traditional written exams cannot properly measure.

The final event in our project will be held on the 15 October on the theme of 21st Century Assessment, where we will discuss how technology can make assessment more relevant, valid and practical. If you would like to register for this event, or if you have any comments about the project, please see the details in this blog post.

Our thanks go to our speakers and guests who helped contribute to such an interesting discussion.

Policy Development Workshop – Balancing Assessment and Accountability

10/07/2014On 8 July AQA held the first policy development workshop of our major project The future of assessment: 2025 and beyond. The theme of this event was ‘balancing assessment and accountability’.

This theme was intended to provoke discussion about the purpose of assessment, how it is currently used and how it might instead be used in the future. Additionally, we wanted delegates to consider accountability in England, and what would be the ideal system if we were free from any current constraints.

Tom Sherrington, Headteacher at King Edward VI Grammar School in Chelmsford, was the first speaker. He talked about the difficulty in comparing similar grades across different subjects, and advocated a move to a baccalaureate model to give an equal standing to different types of qualification.

Next, Laura Dougan and Margaret Miller from the Scottish Qualifications Authority (SQA) spoke and explained the learner-centric assessment system in Scotland. They discussed some of the recent changes to assessment in Scotland, including the greater involvement of teachers and the increasing amount of internal assessment, as well as the new Insight online benchmarking tool which will be used by schools to identify areas for improvement.

Invited guests from across the education sector then discussed potential policy objectives in small groups, joined by our speakers. Each group debated the points raised by the speakers as well as contributing their own hopes and expectations for the next 10-15 years in assessment and school accountability.

Seminar One

Most of the groups spoke about a ‘triangle’ of validity, reliability and accountability, and the struggle of balancing all three of these. Some key desires which emerged were for Ofsted to be more collaborative with schools and teachers and not to focus so much on data; for assessment to follow curriculum design and not the other way around; and to see a separation of assessment and accountability.

Our main aim was to reach a consensus on the three most important issues relating to the balance of assessment and accountability in schools, and the three points below were all discussed at length by everyone at the event.

The groups all agreed that they would like teachers to be more involved in assessing, with some delegates suggesting there should be greater professionalisation of teachers as assessors, similar to other countries such as Germany. This is linked to a desire for more trust in teachers to be able to effectively teach and assess their students.

One theme which emerged was the hope that accountability would move away from being based on exam results, with some agreeing with Tom’s idea of a baccalaureate style qualification, and others thinking that expanding accountability to include destination would be a more appropriate measure.

The groups also decided accountability should be about improving and progress, with one group suggesting schools should be held accountable to their own set of broad aims, which could include assessment but crucially other measures as well.

The event was the first of a series of three events to be held over the summer and autumn. The next two events are on the themes of Assessment in Other Sectors, and 21st Century Assessment. If you wish to attend either of these events and contribute to the project, further details can be found on this blog post.

Our thanks go to the speakers and guests who attended.

Phase 2: Developing Policy

20/06/2014Over the last few months, AQA has asked UK and international experts, teachers, academics, policymakers and employers what they think the future of assessment could and should look like over the next 10-15 years. This collaborative project has so far taken the form of blogs and videos by key thinkers in the sector, as well as roundtable events and debates involving teachers and school leaders.

From these “blue skies” discussions, we have identified three themes which were repeatedly raised as being the most important areas for development and reform over the next 10-15 years. The second phase of the project will be a series of three policy development workshops. Teachers, policymakers and other stakeholders are invited to join us to discuss each of the themes and formulate policy objectives in each area.

The three areas we will be focusing on are:

Balancing assessment and accountability – Tuesday 8th July
How can we achieve good assessment and good accountability at the same time?

Speakers:
Tom Sherrington, Headteacher, King Edward VI Grammar School, Chelmsford
Margaret Miller and Laura Dougan, Policy Managers, Scottish Qualifications Authority

Assessment for the real world – Wednesday 10th September
How can vocational education and training be reliably and validly assessed?

Speakers
Professor Prue Huddleston, Fellow and formerly Director of the Centre for Education and Industry, University of Warwick
Professor Jen Cleland, John Simpson Chair of Medical Education, University of Aberdeen

21st century assessment – Wednesday 15th October
How can technology make assessment more relevant, practical and valid?

Speakers:
Professor Angela McFarlane, Assessment Co-Chair, Education Technology Action Group
Lisa Gray, Assessment and Feedback Programme Manager, Jisc

We will invite expert speakers to provoke debate, and the events will be open and interactive, with everyone having a chance to discuss their thoughts and develop more detailed thinking in smaller groups. The aim of these events is to create three long-term, achievable, policy objectives for each of the themes, which we’ll publish here following the workshops.

The three workshops will be held from 6pm to 8.30pm in central London, and a buffet supper will be provided. If you would like to register interest in attending please email jwilson@aqa.org.uk stating which workshop(s) you wish to participate in. We look forward to seeing you there – you can also follow the debate at www.aqa.org.uk/assessment2025 and on Twitter #assessment2025.

Can and should young people play a role in designing assessments?

 

09/06/2014Professor Jannette Elwood, Professor of Education, Queen’s University Belfast

At the time of writing this blog, students are once again in the thick of the examination season, having to navigate one of the most pressurized times in their educational lives. So while they are sitting up to 2 examinations a day and maybe 4 or 5 examinations a week for the next while – I think it is a very apt time to consider whether they can and should play a role in designing the assessments that they sit.

My short answer to the above question is yes! Young people can and should play a role, not only in how assessments are designed but also in how policy around assessment change is debated, consulted upon and final decisions made.

My answer to the question is based on two factors. Within international legislation – the UN Convention on the Rights of the Child (UNCRC), to which the government of the day is a signatory – children and young people are classified as rights holders and as such are recognized as being entitled to engage in processes that affect them directlyincluding the development of policies and services (in this instance educational ones) through research and consultation (Elwood and Lundy 2010). And, of course, they are key stakeholders in any future assessment systems and, as such, are as important as any other group of people (assessment experts, teachers, head teachers, parents, employers, government) to have a say about what assessment and qualifications systems will look like and entail.

We know at present the landscape of qualification reform in England, Wales and Northern Ireland is one of uncertainty and fragmentation. What is clear is that proposed changes to GCSEs and A levels across the nations of the UK will have major (and differential) ramifications for young people and their future educational and employment careers. Likewise, we are seeing major changes to assessment systems for lower secondary and primary school children and have yet to see the full impact of international testing regimes (such as PISA) and their effect on the educational experiences generally of children across all ages. Yet what is also clear is that there has been a limited history of children’s and young people’s participation in the area of assessment generally and relatively none within policy formation or qualification development. Their input is decidedly missing from any meaningful engagement about the current round of proposals and decisions concerning qualifications reform. This, I would argue, is a significantly missed opportunity to see the bigger picture – the full extent of assessment policy change and its impact.

How do we know that young people have something worthwhile to tell us about assessment reform? Well, we are beginning to know a great deal more about what they think about education generally and from national research that has engaged with young people on these issues (Elwood 2012 and 2013), we are beginning to know a great deal more about what they think, not only about qualification reform, but also about not being consulted about such significant and high level debates. This lack of participation in any decision making on these issues begins to make young people feel more like victims of assessment policy reforms rather than beneficiaries.

So what do they have to say? In talking to nearly 250 young people from across England (Elwood 2012), they told us that yes examinations do dominate their lives in school but that they enjoy them if they are well prepared and know that they won’t get far without good grades. They also thought that examinations structured through modules (and re-sits) allows for any mistakes to be made better, takes the stress off having to do everything in one sitting and that it is only fair to all young people to have a mixture of examinations and coursework – ‘we don’t all like the same things.’ However they felt there was too much confusion about the ‘worth’ of all the different types of qualifications offered and that actual grades were being devalued – with A* taking over as the status of excellence, but not everyone could achieve that. They also felt insulted at the annual circus of the ‘standards are falling’ debates feeling their achievements were degraded; obtaining good grades in whatever qualifications they sit is tough and not getting any easier. They also wanted to know why changes to examinations are introduced ‘live’, i.e. in to their examinations sessions, where their future successes might be ‘messed-up’ if these changes haven’t been piloted in advance; such changes can have considerable impact on their final grades and that is too high a price to pay.

Thus young people have a lot to tell us about both the positive and negative impact of assessments on their lives – we should be listening to them more on these matters. So how might policy makers and assessment developers engage with young people more effectively? There are a number of ways in which this can happen such as: focused policy briefings with education officials and young people in order to obtain input into current debates; examination boards actively setting up panels with young people so that their views can be fed directly into assessment design and implementation; and strategies for consultation involving social media that speak directly to students to gauge their opinions. Good practice, however, and especially rights-based approaches – would mean that we would ask young people what they think is the most effective way to engage with them directly and then change our practice accordingly.

Including young people in decision-making about the future of assessment design or implementation will not be easy; nor will it provide the answer to all the problems that they and we, as assessment professionals, face. But they are authoritative on these matters, just like us, and have a great deal to offer in terms of how they see the world, what they see as salient in their education, what will be of benefit to them and how such future assessment systems could and should be developed with a consideration of their best interests firmly centre-stage.

References

Elwood J and Lundy L (2010) Revisioning assessment through a children’s rights approach: implications for policy, process and practice’, Research Papers in Education, 25: 3, 335 — 353. http://www.tandfonline.com/doi/abs/10.1080/.U42eghb9Mtg

Elwood J (2013) The role(s) of student voice in 14-19 education policy reform: reflections from students on what theyare, and what they are not, consulted about, London Review of Education, 11(2), 97-111.
http://www.tandfonline.com/doi/abs/10.1080/.U42exRb9Mtg

Elwood J (2012) Qualifications, examinations and assessment: perspectives and views of students in the 14-19phase on policy and practice, Cambridge Journal of Education, 42: 4, 497-512.http://www.tandfonline.com/doi/abs/10.1080/.U42eUhb9Mtg

“Without Trust, We Cannot Stand”

13/05/2014Leora Cruddas, Director of Policy, ASCL

The ASCL and AQA debate on the future of teacher assessment raised a number of interesting ideas: Fiona Millar’s fascinating thoughts about what an educated 19-year old looks like; Ian Bauckham’s call for chartered assessors in schools; Patsy Kane’s testimony that teacher feedback is one of the most powerful motivators of progress in students’ learning; and Andrew Hall’s call to rebuild trust across all components of the education profession. I’d like to develop Andrew’s important theme.

It is easy – and politically expedient – to say that teachers can’t be trusted. It is an altogether much more politically courageous thing to build trust. We need political leadership of this calibre: leadership that builds strength and capacity and inspires great leadership in others. We are after all, talking about our children’s future and the economic future of the country. What our education system needs and has always needed is great leadership, both political and professional.

Interestingly, the public does trust the teaching profession. A 2013 Ipsos Mori poll showed that 86% of respondents trust teachers, while just 18% trust politicians to tell the truth. At the debate, around 90% of those present said that teachers are at least somewhat effective at formative assessment, while more than half felt that successive governments have been responsible for a breakdown in trust. But, as Charles Handy says, “we cannot wait for great visions from great people, for they are in short supply at the end of history.” There is a moment for the profession to seize the day and say what we think needs to happen to create a good assessment system.

It is now more than ten years since Onora O’Neill delivered her excellent Reith lectures on A Question of Trust. She said then: “Trust often invites reciprocal trust: and when it does, we have virtuous spirals. Equally trust can open the door to betrayal and betrayal to mistrust: there are vicious spirals.” We want to build virtuous spirals of trust across all components of the education system in order to create the very best education system for our children.

In assessment terms, as Patsy Kane said, this is likely to come from schools collaborating to build professional capital in assessment. There is probably nowhere better to start to build virtuous cycles than in the field of assessment, which is at the centre of teachers’ professional practice.

Finally, as part of the Great Education Debate, and ahead of the election in May 2015, ASCL calls on all political parties to build a culture of trust and confidence in our education system. We must create cross-party agreement on a vision for our education system that enables, motivates and inspires our children to develop as rounded and grounded (to quote the CBI) and take their place in the global society.

Trusted, professional, expert: teacher assessment in 2025

13/05/2014Dale Bassett, Head of Public Policy, AQA

Over the course of the Future of assessment project so far a number of topics have repeatedly arisen in the discussions, events and writing that have contributed to the debate. Teachers’ role in assessment has been one of these, and has also been an important theme in the Great Education Debate convened by the Association of School and College Leaders.

High-quality teacher assessment is essential, not just to improve teaching and learning but also to improve the validity and reliability of summative assessments. Yet concerns over inconsistent quality and the impact of pressures from the school accountability system have led to a decreased role for teacher assessment in qualifications.

On 10th March AQA and ASCL brought together 60 school leaders and teachers with other experts and commentators to consider the issues of trust, teacher professionalism and teacher expertise. The debate, held at Whalley Range High School in Manchester, addressed the questions of restoring trust between teachers and government, enhancing teachers’ skills in assessment, and how better teacher assessment could be a central aspect of the assessment system of the future.

One of the big challenges when discussing teacher assessment – in fact, most topics in education – is that preconceptions and assumptions can often play a bigger role in influencing thinking than facts and evidence. We decided to see what people’s views were before the discussion, and then reassess to see if the debate had changed minds, so we asked the participants a series of multiple-choice questions relating to reliability of marking, trust in teachers and government, formative assessment, and improving the assessment skills of teachers, which they answered using electronic keypads.

The video below shows some highlights of the debate. Four panellists offered thoughts to inform and provoke before the debate: journalist and campaigner Fiona Millar; Ian Bauckham, President of ASCL; Andrew Hall, AQA’s Chief Executive; and Patsy Kane, head of Whalley Range High School. So what were people’s views – and how did the debate change their initial ideas?

Chart 1

Teachers seemed more confident in their ability to reliably assess coursework and controlled assessment before the discussion, in which Fiona Millar and Andrew Hall both addressed the challenges of ensuring reliable teacher assessment in a high-stakes accountability environment. This clearly resonated with the participants, with around 10% deciding after the debate that teachers’ summative assessment was less reliable than they initially thought.

Chart 2

Teachers felt from the outside that good quality CPD was the best way to improve their assessment skills. The debate around becoming an examiner turned some of those who had thought this would be beneficial away from this and towards discrete CPD. Despite some in the room saying that they had learned a lot by being an examiner, Ian Bauckham’s reservations about the benefits clearly held sway with some of the teachers present.

Chart 3

The issue of trust was discussed by some of the panellists, with Andrew Hall saying confidence in the system was reliant on rebuilding trust between governments, teachers, exam boards, and parents. Fiona Millar also indicated trust was an important issue, and that parents tended to trust heads and teachers far more than politicians when it comes to knowing what is best for education.

Patsy Kane said that teachers wanted to understand more about the “black box” of the assessment system, particularly grading, and thought that exam boards should work closely with teachers to help them understand the GCSE and A-level reforms, and a clear majority of those present thought this was the most important way to improve trust in the system. Interestingly, very few teachers thought that stronger regulation was the key to rebuilding trust.

Watch the 12 minute video to see highlights of the event:

How can we improve reliability of assessment?

 

11/02/2014Alastair Pollitt, Principal Researcher, CamExam

I lost my faith in marking on 7th June, 1996, when – as a researcher recently arrived in England – I attended my first Marker Coordination Meeting. The point of this meeting was to make sure that all the markers working on one exam paper were interpreting the mark scheme in the same way, to make the marking “fair”. One of the Principal Examiners began his session by telling the markers, “Your job is to mark exactly as I would if I were marking the script. You are clones of me: it is not your job to think.”

What a chilling message. Is this how to encourage experienced and motivated professional teachers to carry on marking exam scripts? If I had been there as a marker I would have felt humiliated. School-teachers are highly educated and trained, and most of those present that day had many years of experience helping pupils develop their science ability. Their level of commitment to education was certainly higher than average (no one took on the task of marking just for the money!). Yet they were being told to stop thinking, to behave like mere automata. This cannot be the best way to use the experience and wisdom of the profession: there must be a better way.

The fundamental problem is the very notion of ‘marking’, which converts the proper process of judging how well a pupil has performed into the dubious process of counting how many things they got ‘right’. Is it even possible to assess the quality of a pupil’s science ability by counting? Are there not aspects of ‘being good at science’ that cannot be counted?

Not everything that can be counted counts, and not everything that counts can be counted. (William Bruce Cameron, 1963; often attributed to Einstein)

The simple truth is that marking reliability cannot be improved significantly, without destroying validity. Lord Bew recently reviewed the marking of National Curriculum tests for the Secretary of State, and concluded:

we feel that the criticism of the marking of writing is not principally caused by any faults in the current process, but is due to inevitable variations of interpreting the stated criteria of the mark scheme when judging a piece of writing composition. (pp 60-61)

This is true of most exams, not just of writing in English. In every question we ask markers to make a judgement: is this answer worth 0 or 1? Or 2? Or …? Trying to make these judgements reliable relentlessly drives assessment down the cul de sac of counting what can be counted, of identifying “objective” indicators of quality rather than judging quality itself. Referring to exactly this issue Donald Laming, a Cambridge psychologist, wrote:

There is no absolute judgement. All judgements are comparisons of one thing with another. (2004)

What can we do instead? Why not take Bew and Laming seriously? Stop marking: let the examiners make direct comparisons between two pieces of work; or let them rank several pieces. We have long known that teachers can rank order their pupils with high reliability and  high validity; when I began my career by creating commercial tests of reading and maths it was standard practice to report the correlations of the scores with teachers’ rankings as proof of validity. This is what it means to be an expert teacher: being able to make trustworthy judgements of how good two pupils are by comparing samples of their work.

Since most of our examiners are expert teachers, why not get them to behave like experts, instead of robots? Our exams will not only be more reliable, but more valid too.