23-T7 An introduction to international large-scale assessments (ILSAs) in education through the lens of comparative educational effectiveness research (EER)
Course coordinators
Maria Magdalena Isac, mariamagdalena.isac@kuleuven.be
Andres Sandoval-Hernandez, a.sandoval@bath.ac.uk
Course description:
International large-scale assessments (ILSAs) such as the Trends in International Mathematics and Science Study (TIMSS), the Progress in International Reading Literacy Study (PIRLS), the International Civic and Citizenship Education Study (ICCS), and the Programme for International Student Assessment (PISA) are an increasingly important part of the educational research and policy landscape internationally. The emergence of these studies is often connected with a relatively recent marked emphasis on educational accountability and educational system monitoring. Indeed, being representative studies of student populations at the educational systems level, ILSAs produce information that allows for comparing the academic performance of several participating countries around the world and generate policy interest. However, in addition to indicators of student achievement in a number of domains (e.g., reading, mathematics, science), these studies also collect a wealth of background and process information from students, their teachers and their schools. In recent years, these data are increasingly used by researchers to provide an account of the factors and the contexts within educational systems (e.g., student background, teaching practices, school management) that may affect students’ learning outcomes, which is an established area of interest for educational effectiveness research (EER).
Course objectives:
Goal:
The goal of this course is to provide an introduction to international large-scale assessments (ILSAs) in education through the lens of comparative educational effectiveness research (EER). The participants will gain theoretical and methodological knowledge of EER and ILSAs and will build on this information to provide feedback to peers and formulate own research projects.
Specific objectives:
The participants will:
- Become familiar with the discipline of (comparative) educational effectiveness research (EER);
- Gain understanding of the characteristics of ILSAs, their role in EER and, their contribution to the educational research and policy landscape internationally;
- Identify the theoretical and methodological advantages and limitations in using ILSAs;
- Gain skills in using scientific criteria for reviewing research based on ILSAs;
- Reflect on the theoretical and methodological input to setup a research project proposal that addresses research questions relevant for the topic of the course;
- Effectively communicate their research plans to course coordinators and peers;
- Engage in constructive debate and provide feedback to peers and invited guests.
Requirements/entry level:
This course is recommended for all PhD-students.
Course Programme
Meetings: The course consists of four days in total with a month in between.
During the first two days, presentations by course coordinators and invited senior speakers are alternated with discussions and group work. Invited senior speakers will be selected in such a way that they illustrate different applications of ILSAs to EER topics of interest (e.g., comparative measurement of student outcomes in different domains of learning, EER models applied to ILSA data, teacher and school effectiveness factors in ILSAs, etc.) and will be asked to actively reflect with the participants on the strengths and limitations of their work.
Day 1 – morning
- ILSAs (e.g., PISA, TIMMS, PIRLS, ICCS) and to research and educational policy questions that can be answered with these data.
Day 1 – afternoon
- Group discussion on theoretical and methodological EER approaches applied to research with ILSAs.
- Brief introduction to guidelines in reviewing research based on ILSAs;
- Invited Senior Lecture 1. For example, Cross-national comparison of school effectiveness factors: empirical application of the dynamic model of educational effectiveness by Agnes Stancel-Piatak (IEA).
- Invited Senior Lecture 2. For example, The Leaning Tower of PISA by David Rutkowski (University of Indiana)
- Invited Senior Lecture 3. For example, Grouping of students within the school and educational practices. Institutional and pedagogical devices developed by establishments with high degrees of school effectiveness in closing learning gaps by Ernesto Trevino (Universidad Catolica de Chile)
- Group reflection on strengths and limitations of ILSA-based studies;
Day 2 – afternoon
- Introducing guidelines and resources for Individual Assignment (1): Reviewing ILSA research in progress by invited (PhD students)– to be completed by Day 3.
- Introducing guidelines and resources for Group Assignment (2): Students will work in small teams (4-5 participants) to draft a research proposal on a topic of choice within the ILSA framework (max 4 pages) – to be completed by Day 4.
- (2).
- Three invited presentations from (PhD students) working with ILSA data. Participants would receive and read the papers to be presented in advance. After the presentations there would be time for clarification questions related to the theoretical and methodological approaches used by the speakers.
- General group discussion.
Day 3 – afternoon
- Individual work (supported by course coordinators) aimed to finalize (first full draft) Assignment (1): Reviewing ILSA research in progress.
Day 4 – morning
- Small group work (supported by course coordinators) aimed to finalize and prepare a presentation for Group Assignment (2): short research proposal.
Day 4 – afternoon
- Presentation of group work with feedback from peers and invited guests.
Specification of the workload:
(84 hours in total)
28 for the meetings; 56 assignments
Potential Dates:
January 27 & 28 2023
February 27 & 28 2023
Utrecht
registration deadline: 17 December 2022
Location:
Utrecht city centre (t.b.d.)
Maximum number of participants: 24
Potential to move into an online or hybrid mode (if necessary).
Assessment:
– Being present through all face-to-face meetings.
– Complete the reading materials.
– Submitting the assignments.
Feedback
– Feedback from course coordinators, peers and guests on the assignments.
– Support from teachers to improve the assignments.
Literature
Literature that will be addressed (selected book chapters and empirical articles; mandatory literature will be announced several weeks before the course).
Important (most likely) sources:
Chapman, C., Muijs, D., Reynolds, D., Sammons, P., & Teddlie, C. (Eds.). (2016). The Routledge international handbook of educational effectiveness and improvement. Routledge.
Nilsen, T., Stancel-Piątak, A., Gustafsson, J-E. (Eds.). (2022). International Handbook of Comparative Large-Scale Studies in Education. Perspectives, Methods and Findings. Springer.
Rutkowski, L., von Davier, M., & Rutkowski, D. (Eds.). (2013). Handbook of international large-scale assessment: Background, technical issues, and methods of data analysis. CRC Press.
Required reading (during the course):
We will provide reading materials at the beginning of the course.