A DESIGN OF SPEAKING ASSESSMENT RUBRIC FOR ENGLISH IMMERSION CAMP

The aim of this mix-method study is to design a speaking assessment rubric for an English Immersion Camp (EIC) program in Muhammadiyah University of North Maluku called UMMU English Immersion Camp (UEIC). In UEIC, there were seven activities applied for triggering the participants to continuously speak English during the program (see Syahidah, Umasugi, & Buamona, 2019). However, the program did not have a proper assessment form to measure the UEIC participants’ speaking development. Therefore, this study was administered to design the assessment in the form of a speaking rubric. Three instruments were used to gather the data for designing the rubric. Interviews were adminstered to two experts of English language assessment who have been having years of experince in teaching English at the university. Document analyses were done to analyze the UEIC syllabus from previous studies and speaking rubrics from various sources to make a match between the design of the assessment instrument with the UEIC achievement targets, objectives, materials, and learning techniques. Observations were conducted by four observers who were fasilitators at the UEIC to measure the participants’ speaking progress. Data from obervations were used to test the validity and reliability of the rubric. The speaking assessment rubric was designed based on the steps of designing speaking assessment proposed by O’Malley and Pierce (1996) which are identifying the purposes of speaking assessment, planning speaking assessment, developing speaking test rubric and setting standards. The rubric contained ten items divided into six aspects of speaking assessments namely (1) grammar, (2) vocabulary, (3) pronunciation, (4) fluency, and (5) comprehenesibility respectively represented into two items, and (6) accuracy measured by three items. Moreover, the validity and reliability analyses showed that the rubric containing 10 items were valid and reliable because all values needed have reached the theoretical standard value for validity and reliability of a speaking test instrument.


INTRODUCTION
Every English learning instruction needs assessment to measure learners' learning progress and the instruction effectiveness. Various experts state that assessment is important in second language learning due to its function as a reference for making decision about learners' learning progress (Bachman, 1990;Davies, 1990;Brown, 2004;Harmer, 2007;Douglas, 2010;Berry and Adamson, 2011) in all skills. In terms of English speaking skill, assessment comes to the forefront since the results of the assessment may reveal the latest level of learners' speaking ability by which the decision for next learning instruction can be predicted and designed.
There are many English instructions or programs designed to develop learners' ability in speaking. One of the program is English Immersion Camp (EIC) designed and held by Syahidah et al (2018) for students of English department at Muhammadiyah University of North Maluku. EIC is an outdoor activity where students are set to stay at a camp and English is used as the main language for the interaction of participants and facilitators during the implementation of camp. EIC is designed and implemented to create an Englishspeaking environment to help students at the university who want to learn English but have limited access to an English-speaking environment. Referring to its implementation at Muhammadiyah University of North Maluku, EIC was named the UMMU English Immersion Camp (UEIC).
Before implementing UEIC, Syahidah et al conducted a needs analysis on prospective UEIC participant students to find out their English language needs and an analysis of EIC characteristics. The results of both analyzes were used as a primary data to design the UEIC syllabus. However, despite having made a UEIC syllabus design and implementing it in a UEIC activity, the syllabus design still focuses more on developing learning materials and strategies, while the assessment and evaluation are still not described in detail. During the implementation of UEIC, the assessment was carried out by observing participants with specific assessments on speaking, listening, vocabulary mastery, and English grammar without using or referring to an assessment rubric which assessment results can be accounted for and concretely proven. Therefore, to complete the UEIC syllabus design, this study is conducted focusing on developing a speaking assessment instrument containing assessment principles in accordance with three assessment criteria, namely authenticity, validity, and reliability.

Immersion in Second Language Acquisition Theory (SLA)
Immersion is a method or approach applied in bilingual learning. As a method, immersion allows a person to learn a second language or target language where the environment and all learning instructions use the language being studied. In other words, in the application of the immersion method, a person is like being "immersed" in the environment where the target language is used (FPHLCC Language Program Handbook Development Team, 2010 language is obtained by a person from the environment in two ways, firstly by learning the language consciously (learning) and secondly by learning the language unconsciously or naturally (acquisition) because it is formed by the environment in which language interactions happened (Krashen, 1982). Based on this SLA theory, the immersion method is then developed in language learning, either by implementing it directly in the native land or in the environment surrounded by the language being studied.
The implementation of the immersion method in language learning was initially applied to immigrants in a country who had difficulty communicating daily and academically using the native language that they visited. To overcome these communication difficulties, immigrant recipient countries then create language learning programs using the immersion method for a specified period of time. Generally, the implementation of this immersion program runs in a few months, even several years, depending on the target of achieving the second language proficiency has been defined in the curriculum or learning syllabus. The immersion method can not only be implemented for learning English, but in all languages. Several countries such as Canada, the United States, and China have implemented immersion programs to help immigrants in their countries to be able to communicate in the country's language well (See Rossell, 2000;Siano, 2000;Luan and Guo, 2011).
As previously mentioned, immersion programs are generally applied with a long duration such as in months or years in the environment where the second language is used. However, in this study, the immersion program was only applied for a maximum of three days at the campsite (English Immersion Camp) as a learning supplement for English learners who are EIC participants. The results of this study provide new information about the application of the immersion program with a short day duration and adapt to the environment in which English is used and its effectiveness in helping students improve their English skills, particularly speaking skill.

The Design of English Immersion Camp Syllabus
The design of the English Immersion Camp (EIC) syllabus entitled UMMU English Immersion Camp (UEIC) has been created and implemented in previous research (Syahidah et al, 2018) by adapting the order of designing the syllabus from Hutchinson and Waters (1987), Brown (1995), Richards (2002), and Nation and Macalister (2010). The picture from the UMMU English Immersion Camp (UEIC) design shown as follows:

Evaluation
The Stages of English Language Acquisition Proposed by Goldberg (2000) This UEIC design is used in this paper by focusing on developing the design of the assessment instrument. This means that at the end of this study, there will be some changes in the assessment items of the design from observation the learners' all skills and language components to just speaking skill

Components of Speaking Assessment
There are several components of speaking skill that shoud be considered in designing a speaking assessment. Brown (2004) states that speaking assessment has to cover pronunciation, grammar, vocabulary, fluency, comprehension and task. Hughes (2005) mentions accent, grammar, vocabulary, fluency, and comprehension as the components of speaking. Meanwhile, Mazouzi (2013) summarize the components into two namely fluency and accuracy where comprehension, grammar, vocabulary, and pronunciation have been included within. To take these experts' views into consideration, the speaking assessment

Criteria for Designing Speaking Assessment Rubric
Making an English learning assessment instrument according to Bachman and Palmer (1996) requires six criteria, namely reliability, validity, authenticity, interactive, impact, and practicality (reliability, construct validity, authenticity, interactiveness, impact and practicality). In this study, three criteria are used to test the assessment instrument designed for the UMMU English Immersion Camp (UEIC) activities which are authenticity, validity, and reliability. An assessment instrument is called authentic if the it is carried out based on the proper context or the target language context (Bachman and Palmer, 1996;Brown Abeywickrama, 2010).
As for validity, an English assessment instrument is considered to be valid if the instrument used assesses what should be assessed (Harmer, 2007;Hughes, 2003). Validity has several types, but in this research content validity and construct validity are used. Content validity is the type of validation of an assessment instrument based on the content or skill being assessed (Hughes, 2003). Construct validity is the validation type of assessment instrument based on the correlation between test scores and the theoretical construction of the test items (American Psychological Association, 1954;Cronbach & Meehl, 1955). In terms of reliability, an assessment instrument is reliable seen from the accuracy and consistency in the assessment rubric to the assessment results (Douglas, 2010;Brown and Abeywickrama, 2010).

Research Design
The study employed mix-method as research design. The data were gathered through qualitative method by which interviews, questionnaire and observation were administered. Meanwhile, quantitative method was applied to analyze data from obervations of learners' speaking perfomance during the UEIC program. The UEIC assessment was designed based on data analysis of the needs

Data Collection Techniques
To obtain research data, the data collection technique used in this research is triangulation, namely: 1. Document analysis, used to analyze the UEIC syllabus from previous studies and speaking rubrics from various sources to make a match between the design of the assessment instrument with the UEIC achievement targets, objectives, materials, and learning techniques.
Furthermore, the results of the analysis of this document were functioned to compile an authentic, a valid and reliable assessment rubric for UEIC assessment.
2. Interview, used to support questionnaire data from prospective UEIC participants and to obtain their self-assessment. Interviews were also adminstered to two experts of English language assessment. These two experts are faculty members of Khairun University who have been having years of experince in teaching English at the university.
3. Observation, used to observe the speaking skill of UEIC participants as the object of direct assessment during the implementation of UEIC.
Participants were observed in all seven UEIC activities. Observations were done by four observers who were fasilitators at the UEIC who have reached advanced level of English proficiency. These fasilitators had been trained to be the observers for filling in the UEIC assessment rubrics based on the participants' speaking perfomance. The training was administered before the UEIC was conducted in order to avoid misunderstanding and bias during the assessment process.

Data Analysis
After obtaining the required data from data collection techniques, data analyses were carried out with a systematic description from the results of interviews and document analyses containing the identification of speaking assessment purposes and the planning of speaking assessment content. This description was used as considerations to design the authentic assesment rubric Validity of the rubric was confirmed through V Aiken Coefficient (Aiken, 1985).
The content validity was analyzed through Aiken's V formula as follows: If the r value is more than r critic (r > r critic), it means that the item is valid.
The value of r critic for 9 participant (df = N-2) is 0.666. Moreover, the more the value is near the value of 1, the stronger the validity is. Meanwhile, the reliability was calculated through Cronbach Alpha Coefficient test in SPSS version 26 software. The value of reliability was interpreted by comparing the value of Cronbach Alpha with the following table: The results of these data analysis procedures are then described at the following section.

RESULT AND DISCUSSION
This section is presented into two sub-sections. The first is presented by following the steps of designing an authentic speaking assessment proposed by O'Malley and Pierce (1996) which are identifying the purposes of speaking assessment, planning and developing speaking assessment rubric, and setting standards. The second is presented by describing the analysis of validity and reliability of the speaking assessment rubric.

Identifying the Purposes of EIC Speaking Assessment
At the beginning, the English Immersion Camp program was planned and administered in order to answer the needs of English students in North Maluku regarding the English environment where they have chance to practice their receptive and productive skills, particularly speaking skill. Answering the needs, the EIC program were designed with seven variations of speaking activities.
Therefore, it is written in its design that the goal of EIC is "Developing the Participants' Communicative Competence" by which the objectives are formulated into three, namely: 1. To provide English environment where participants can practice their English ability in context.

2.
To trigger participants to be more active and be confident in expressing their thought using English.
3. To create participants' togetherness and stimulate participant's ability to work in group.
Based on the goal and objectives, the purpose of the EIC assessment was determined focusing more on speaking skill in which the purpose of assessment was to identify the participants' speaking progress during the program. In other words, the assessment was not administered for initial identification and placements of the students in need of a language based program, for movement from one level to another program within a given program, and for placement out of an ESL/bilingual program in to a grade-level classroom as explained by O'Malley and Pierce (1996). Hence, the purpose of the EIC assessment was just comprehensive, and accuracy as the items in the assessment rubric. The same line also delivered by Expert 2 who said that, Assessing speaking is quite complicated since we have to combine so many aspects in one measurement like pronunciation, grammar,.. vocabulary is also important and fluency, and accuracy. So, it needs carefulness,... particularly for those who assess.
The aspects of speaking assessment revealed by these two experts are in accordance with Heaton (1991), Nunan (1999) and Brown and Abeywickrama (2010) who expained that speaking assessment has to contain the measurement of students' grammar and vocabulary mastery, pronunciation, fluency, comprehensibility, and accuracy.
Aligning the experts' inputs and analysis from related document, the writers tried to design the EIC assessment rubric by accomodating the aspects of speaking assessment stated above. The following table shows the analysis of aspects of speaking assessment in EIC assessment: Table 3. Analysis of Speaking Assessment Components of EIC

Description in the Rubric
Grammar Speaks no grammatical error (speaker self-correct without hesitation). Grammar Applies a variety of grammatical structure in the

Vocabulary
Uses sufficient but varied vocabulary to express idea clearly.

Vocabulary and Comprehensibility
Uses compund and complex sentences in speaking.

Comprehensibility
Comprehends and responds in detail of other people's talk.

Fluency
Speaks in smooth flow, quick, and continuous flow as well as natural pauses.

Pronunciation and Fluency
Is phonetically correct in speaking and awareness of accent.

Accuracy
Uses strategies as needed when she/he meets with difficulties in finding words to say.

Accuracy
Is able to deliver a presentation and to respond to the audiences with appropriate verbal cues and eye contact.

Pronunciation and Accuracy
Speaks clearly and can imitate accurate pronunciation.

Setting Standard for EIC Assessment
Once EIC assessment rubric has been designed, the writers then set the standard of participants' speaking performance. For the rubric, the writers decided to set the standard by adapting the form of analytic oral language scoring rubric (see O'Malley and Pierce, 1996: 68). The description of the rubric with its standard is seen below: Table 4. The Standard Criteria of EIC Assessment Rubric Description in the Rubric 1 2 3 4 5 Speaks no grammatical error (speaker self-correct without hesitation). Applies a variety of grammatical structure in the speaking. Uses sufficient but varied vocabulary to express idea clearly. Uses compund and complex sentences in speaking. Comprehends and responds in detail of other people's talk. Speaks in smooth flow, quick, and continuous flow as well as natural pauses. Is phonetically correct in speaking and awareness of accent. Uses strategies as needed when she/he meets with difficulties in finding words to say. Is able to deliver a presentation and to respond to the audiences with appropriate verbal cues and eye contact. Speaks clearly and can imitate accurate pronunciation.
The standard criteria for participants' speaking performance in UEIC at the rubric above was set by ranging the scoring from 1 to 5. Scores of 1 and 2 required basic level. Score of 3 indicated the intermediate level. Meanwhile, scores of 4 and 5 requires advanced level. This analytic form was used not for relocating participants of UEIC at certain level, but for gathering information about their speaking progress during participating in the UEIC. Therefore, at the end of the assessment, there was only a report about the implementation of EIC and its effect of participants' speaking development.

Validity
Analysis of validity was done through two types of validity namely content validity and construct validity.

Content Validity
As stated by Hughes (2003) that content validity is the type of validation of an assessment instrument based on the content or skill being assessed, the following table captures the result of EIC assessment based on the analysis of V Aiken Coefficient.

Valid
As seen in Table 6, the analysis of Pearson Product Moment of EIC assessment rubric reveals that the r value of all items in the rubric are bigger than the r critic which is 0.666. This interpretation means that all items are valid.
Besides, the result indicates that the scores of all 10 items of the rubric have strong correlation with the total scores since the r values are more than 0.70.
The analysis results of content validity and construct validity above concludes that all 10 items of EIC assessment rubric are valid. Therefore, the proposed rubric is acceptable to use as a way for assessing EIC participants' speaking skill.  's (1996) steps of designing authentic speaking assessment which are identifying the purposes of speaking assessment, planning speaking assessment, developing speaking test rubric and setting standards. The rubric contained ten items divided into six aspects of speaking assessments namely (1) grammar, (2) vocabulary, (3) pronunciation, (4) fluency, and (5) comprehenesibility respectively represented into two items, and (6) accuracy measured by three items. Besides, the validity and reliability analyses showed that the rubric containing 10 items were valid and reliable. At content validity test, all items in the EIC assessment rubric have reached the minimum standard of V' Aiken value where 5 rating scales and 4 raters were used