The CEFR in non-formal education (in English courses Kampung Inggris Pare): developing descriptors of academic English

"The CEFR in non-formal education (in English courses Kampung Inggris Pare): developing descriptors of academic English"

By: Ade Teguh Ramadhan 16020230006

´Outline

•Brief introduction to EAP

•Brief introduction to 4 English courses in Kampung Inggris

•The suitability of the CEFR in the academic context

•Developing descriptors of Academic English 

´Brief introduction to EAP

´English for Academic Purposes

• EAP “refers to language research and instruction that focuses on the specific communicative needs and practices of particular groups in academic contexts” (Hyland and HampLyons, 2002)

• EAP is used in a variety of settings (in English-speaking countries or outside) and for different purposes (international or home students, at academic or secondary school level)

• Needs of mainstream students ignored by a “deficit-approach” to EAP (Wingate and Tribble, 2012)

§ Writing instruction for international students

§ Remedial writing courses, typically in learning support or study skills units

´EAP and EGP

´EAP is different from EGP because of its more focused scope

´Generally seen as a subcomponent of ESP (Johns, 2013)

´Based on a need analysis (Flowerdew & Peacock, 2001)

´Major focus is on study skills and specific tasks which are relevant to the academic context, not the language (Alexander et al., 2008)

´Bias towards written skills, perhaps because written genres are seen as “more central to professional success” (Feak, 2013)

´High stakes and limited time (Alexander, 2012)

´Testing EAP and validity claims

´Weir (2005) calls for a test to elicit authentic processing (cognitive validity) and to accurately reflect the targeted language (content validity)

´ In reality, the construct represented in tests is much more narrow and restricted than real language use:

• Length of task

• Background knowledge

• Chances to reedit responses

´ This may lead us to conclude that tasks should be relevant but they can never be authentic!

´Using EAP descriptors in teaching and testing

´In teaching, EAP descriptors could help:

• apply a systematic and standardized approach to curriculum development

• bridge the gap between teaching and assessment

´In assessment, EAP descriptors could help:

•produce more accurate test specifications to describe the content of the test

•judge performance and progress in relation to scaled learning objectives and therefore give meaning to the scores

´Back to the basics: the EAP construct

´Fulcher (1999) expressed concerns about the definition of the construct of EAP, particularly in relation to the degree of agreement between raters on what constitutes academic English

´The CEFR is used as the starting point to align tests to what is considered to be a valid standard of scaled proficiency – but how suitable are the CEFR descriptors for developing a test of academic English?

´Brief introduction to 5 English courses in Kampung Inggris

1. Mother Language English School

Mother Language English School and Test English School  are non-formal English education. It is located in Ogan street no.20, Pelem, Pare - Kediri. Mother Language English School Offering programs IGEP (Intensive General English Program), Micro Teaching Program, English Testing (IELTS, TOEFL ITP, IBT & TOEIC), and Mastery Program. The goal for a program at Mother Language English School focus on English for academic purposes.

2. Test English School

Test English School is located in Brawijaya street No.92A Tulungrejo Pare-Kediri. Test English School is an informal education that has some programs like Beginner programs (Homeschooling, preschool, afterschool & Test university), Intermediate (collaboration, elaboration, intension & Test university), and the last is Advanced programs (IELTS camp & TOEFL Camp). The goal for a program at Test English School focus on English for academic purposes

3. UNICEF English Course

UNICEF English Course is located in Jl. Pk. Bangsa Gang Ogan No.31/B Pelem Pare-KEDIRI. UNICEF English Course School is an informal education that has some programs like General English, English testing & Bali on vacation.

 4. CEC (Coupis English Course)

CEC (Coupis English Course) is located in Jl. Irian Barat. No.11, kampung Inggris Pelem, Pare, Kediri, Jawa Timur. CEC (Coupis English Course) is an informal education that has some programs like General Program, TOEFL Preperation program & IELTS Preperation program.

The suitability of the CEFR in the academic context

´Uneven distribution of descriptors across the four skills (a large part (65%) of information is about Speaking)

´ Lack of descriptors at the lower and higher levels (A1, C1, C2) and no information below A1

´ Too wide levels (A1-C2) and opaque meaning of bands: what does it mean to be at a level?

´ Very little information about professional/academic use of language

´Probabilistic interpretation of “being at a level”

´Each CEFR descriptor was originally calibrated using the Rasch model

´In the Rasch model, when a person's ability is equal to the difficulty of the item, there is by definition a 0.5 probability of a correct response

´Therefore being at a level means having a 50% probability of being able to perform the entire set of descriptors defining that level

Table: Probability of success at CEF levels for learners at different levels – given on the original theta values identified by Brian North

Developing descriptors of academic English

´Rationale

´The CEFR contextualises language activities in four domains: the public domain, the personal domain, the educational domain and the occupational domain” (CoE, 2001, p. 14).

´In creating specialist learning objectives, we are addressing the different language needs that arise in different domains.

´The GSE Learning Objectives for Academic English address the needs of learners in the educational domain, with a focus on academic study at the tertiary/post-secondary level.

´All learners need to acquire a core of proficiency in English, and for that reason, the GSE Learning Objectives for Academic English include the learning objectives developed for adult learners of general English.

´The CEFR and academic language needs

´Can understand recordings in standard dialect likely to be encountered in social, professional or academic life and identify speaker viewpoints and attitudes as well as the information content.

´Can follow the essentials of lectures, talks and reports and other forms of academic/professional presentation which are propositionally and linguistically complex.

The majority (8) are placed at level B2 and a minority (5) at level C1, none at Level C2.

In the ALTE appendix descriptors mentioning ‘academic’ are at levels 4 and 5, claimed to be at C1 and C2 respectively 

´Rating Learning Objectives

´Descriptors were distributed over 4 batches in an overlapping fashion

´Batches were evenly attributed to groups of raters

´Within batches descriptors were grouped by skill, but a) the order of the skills and b) the order of descriptors within each skill was randomised so that each batch was presented differently

´Expert raters (around 80 -100 per batch): at least 2 years teaching experience with target learner group, attended training workshops, rated on both the CEFR and the GSE

´Online raters (N=558) with at least 2 years teaching experience, attended online training, rated on the CEFR only

Research: in 4 English courses across Kampung Inggris Pare

 ´SAMPLE DESCRIPTOR

´First, decide which CEFR level you think a student would need to be at in order to have a 50% chance of successfully completing the task

• Then, decide where within that level you think the learning objective sits. Use the GSE value to indicate start, middle or end

• Answer, e.g.: B1(+) [=GSE 51-58] à 54

´Calibrating Learning Objectives

Statistical analysis and data cleaning to remove

• “Misfitting” raters, e.g. raters who rated less than 75% descriptors or with a low SD indicating they did not use the full range of scale or with an average score which was too distant from the group average

• “Poor” Learning Objectives, e.g. descriptors with an overall degree of rater agreement

The final GSE values showed a 0.99 correlation with values based on the classical analyses by North (2000)

Distribution of Academic Learning Objectives

Example Academic Descriptors

´Learning Objectives for Academic English

´In collaboration with 4 English courses in Kampung Inggris Pare

üCreated on the CEFR model (North, 2000)

üDescribe performance abilities in academic English

üCan be used to create the level-appropriate content syllabus for examinations and to identify assessment criteria in the English courses