TRAINING AND IMPLEMENTATION MANUAL Tes Cepat Teachers’ Performance & Accountability (KIAT Guru) MARCH 2021 This work is a product of the staff of The World Bank. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions © 2021 The World Bank 1818 H Street NW, Washington DC 20433 Telephone: 202-473- 1000; Internet: www.worldbank.org Some rights reserved. The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for non- commercial purposes as long as full attribution to this work is given. All queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. Attribution Please cite the work as follows: Lumbanraja, S. K.; Prameswari, I.A.; Susanti, D. 2021. Training and Implementation Manual: Tes Cepat, World Bank, Jakarta. © World Bank. Photo credits Fauzan Ijazah/World Bank. TRAINING AND IMPLEMENTATION MANUAL Tes Cepat Teachers’ Performance & Accountability (KIAT Guru) Table of Contents 01 General Guideline for Administration of Both Paper and Digital Tes Cepat 1 Testing Principles 4 Protocol 6 02 Adaptive Testing Guideline for Paper Tes Cepat 9 03 Digital Tes Cepat Mobile App Administration Manual 13 04 Digital Tes Cepat Web App Administration Manual 17 04 Photo Story: Training Digital Tes Cepat to Community Members 21 Figures Figure 1 Tes Cepat Adaptive Testing 10 Figure 2 Visual Map of Student’s Competencies 12 Figure 3 Digital Tes Cepat Mobile App Features and Users 12 Figure 4 Digital Tes Cepat Mobile App Home Page 13 Figure 5 Digital Tes Cepat Web App Features and Users 18 Figure 6 Digital Tes Cepat Web App Users’ Roles and Guideline 20 Training and Implementation Manual Tes Cepat Introduction KIAT Guru developed Tes Cepat, a low-stake diagnostic test on primary grade reading and math administered by community members on a semester basis. Tes Cepat was developed in the spirit of citizen-led assessment movements that aim to increase parental involvement in education by providing them access to learning information that is easily understood and can be monitored regularly. For more information on the test, see BRIEF Tes Cepat: Diagnostic test to increase community participation in improving learning outcomes in Indonesia’ remote primary schools. Tes Cepat was initially developed as a paper-based assessment in 2016. Starting in 2019, the test was digitized into a mobile phone application to enable local administration, automated scoring and interpretation of test results, and data storage. This manual document provides details on the principal and technical implementation of the Tes Cepat for both the paper-based and android-based mobile phone app, including the differing prerequisites to each format. For more details on implementation of the paper-based test, see Tes Cepat Paper Manual Booklet. Details on operating the test Tes Cepat android-based mobile app can be found in the Tes Cepat Digital Manual. 1 CHAPTER 1 General Guideline for Administration of Both Paper and Digital Tes Cepat Training and Implementation Manual Tes Cepat ! Testing Principles 1 The test is administered to a total sample of 30 students from grade 1 through 6 per school. Sample students are selected randomly, with six students from each grade. Other students in the school may also be tested given all 30 sample students have completed the test. 2 Primary grade basic reading and math reading skills are assessed. Standards adhere to the national curriculum (KTSP 2006). 3 The test can be administered by representatives of the community, e.g. parents, community leaders, etc. Teachers can assist community members with coordinating logistics of the test and in disseminating the test results to broader stakeholders. 4 Each student is assessed by one administrator. 5 • Location of the test should be conducive and comfortable for students. This includes minimized background noise, sufficient lighting, and appropriate surface for student to sit and work on the test problems. • Time of the test should not interfere with school hours. • Parents and teachers should be informed on the location and time of the test. • If conducted by multiple administrators, location and time of the test should be done in parallel to ensure efficiency and objectivity of the testing process. • The test should last up to 15 minutes per subject. 4 Training and Implementation Manual Tes Cepat 6 Teachers and/ or parents of the child tested should not disturb the testing process. If possible, keep teachers and parents of the child tested at proximate distance from the testing location. 7 Test administrator should not give students any instruction and/ or assistance in answering the test items. 8 Test administrator should not test their own children/ family member to ensure the test’s objectivity. 5 Training and Implementation Manual Tes Cepat ! Protocol Before, during, and after the test, administrators must be cognizant and respectful of the school environment whilst maintaining integrity of the test by following this protocol: TWO DAY D-2 BEFORE THE TEST • Inform both community and school on the purpose and plan to conduct Tes Cepat. • Identify and train community members who have volunteered to become train administrators. They must be able to read, write, and count. They also must be able to operate mobile phone. • Schedule the location and time of the test that are agreeable to both parents and teachers. • Obtain a copy of student roster from the school. This roster will be used to select sample students. • Inform and obtain consent from selected sample students’ parents. 6 Training and Implementation Manual Tes Cepat ON THE DAY OF THE TEST BEFORE THE TEST • Prepare supplies: test booklets, test administration manual, and stationeries. • Arrange room and desk: Ensure student and administrator are positioned comfortably in the test location. • Inform student on the purpose of the test and obtain their consent prior to the test. Inform students that the test will begin with reading and followed by mathematics. • Fill in testing data: student data, administrator data, and date and location of the test. • Inform student on instructions to mark their answers during the test. For paper-and-pencil Tes Cepat, student may indicate their answer verbally, or in writing, or by pointing to the selected option in the test booklet. For digital Tes Cepat, student may select on the screen the answer option of their choice. DURING THE TEST • Instruct students to read thoroughly and work on the test items presented to them. For paper- and-pencil Tes Cepat, provide instructions for dictation test items. • Administrators may not at any time read aloud the content of the test items and/ or answer options. • Ensure the same test items are not given to students more than once. For paper-and-pencil Tes Cepat, follow the adaptive flow guideline. For digital Tes Cepat, monitor that the app presents each test item once. • Maintain integrity and confidentiality of the test. For paper-and-pencil Tes Cepat, ensure answer key and administrator note are not visible to students. For digital Tes Cepat, administrator may not operate the screen at any time during testing session. AFTER THE TEST • Ensure student test results are recorded. For paper-and-pencil Tes Cepat, record the last test item that students were able to answer correctly. For digital Tes Cepat, check student’s record in the “Riwayat” feature. • Thank the student and conclude the test activity. Continue with the next sample student. 7 CHAPTER 2 Adaptive Testing Guideline for Paper Tes Cepat Training and Implementation Manual Tes Cepat The test adaptivity is core to Tes Cepat’s quality. All administrators should be trained to conduct paper Tes Cepat adaptively. The following are general steps to conduct adaptive testing with paper Tes Cepat, which are summarized in Figure 1: Figure 1. Tes Cepat Adaptive Testing KELAS 5 TERAKHIR STEP 1 STEP 2 STEP 3 STEP 4 END OF TEST 30. Perhatikan gambar di bawah. Sisi Identify NO First item on the yang sama luas dengan ABCD student’s last page for one adalah ... grade level grade level below H G based on 2016/2017 First item on the last Skip further down E F Academic Year page consisting of until child is able D C NO items at child’s grade to answer (Baseline) A B level correctly a ADHE b BCGF c EFGH d EFCD Can the child answer correctly? 31. Pilih gambar yang bukan Continue to the When child merupakan jaring-jaring kubus next (higher stop adalah ... number) items a b c d Mark child’s YES Until child level based on answers the last First item on the last incorrectly (highest) test page consisting of item answered 31. Murid kelas 5 di SD Pagi Sore ada 40 items at child’s grade correctly anak. Perbandingan murid level perempuan dengan semua murid di kelas 5 adalah 5 : 8. Ada berapa Can the child answer banyak murid perempuan di kelas 5 correctly? NO di SD Pagi Sore? STOP a 5 b 25 c 8 d 13 HALAMAN 13 Continue to the next (higher number) item Example: YES Can the child answer Tono, a former 5th grader in correctly? Continue to the 2016/2017 academic year will be next (higher given item #30 on page 13 as the number) items first item on the last page for fifth Until child grade in the Math Test Booklet. YES answers incorrectly 1 STEP 1: Identify student's current grade level. This will determine the first test item for the student to work on. In the test booklet, it would be the first test item on the last page of the student’s current grade level. 2 STEP 2: If the student can answer their first test item correctly, proceed to a higher number item. However, if the student did not answer the first item correctly, proceed to a lower number item, i.e. the first item on the first page of the student’s current grade level. 10 Training and Implementation Manual Tes Cepat Example If a student is currently in second grade, number 13 would be their first reading item. If they answered the first item correctly, they would continue to number 14. However, if they answered the first item incorrectly, they would go back to number 9. Last Page of KELAS 2 TERAKHIR Perhatikan kalimat berikut! Grade 2 nia pergi ke jakarta hari senin First reading 13 Tulisan yang tepat untuk kalimat di atas adalah ... item a Nia pergi ke Jakarta hari Senin b Nia pergi ke jakarta Hari Senin c nia pergi ke Jakarta Hari Senin d Nia pergi ke jakarta hari Senin Second Perhatikan percakapan berikut! reading item if previous Tanya : ... yang memimpin senam pagi? item was Jawab : Guru olahraga answered First Page of correctly Grade 2 14 Kata yang tepat untuk kalimat tanya di atas a Siapa KELAS 2 PERTAMA b Apa Berlibur ke Kota c Mengapa Pada hari Minggu, aku pergi. d Bagaimana Bersama Bapak dan Ibu. Kami naik mobil ke kota. Hatiku sangat gembira. Ibu membawa bekal makanan sehat. HALAMAN 8 Bapak menjaga kami agar selamat. Di tengah kota, ada tanaman.. Tanaman bunga yang beraneka rupa. Bangku-bangku di sekitarnya ditata rapi. Di sana, tidak boleh membuang sampah sembarangan. Kita harus menjaga kebersihannya. Aku, Bapak dan Ibu duduk di bangku. Kami duduk memakan bekal. Kami juga menikmati pemandangan kota. Second 9 Kapan mereka berlibur ke kota? reading item if previous a di hari Minggu item was b di tengah kota answered c di mobil incorrectly d di minggu depan 10 Apa yang tidak mereka lakukan di kota? a Memakan bekal b Duduk di bangku c Membuang sampah sembarangan d Menikmati pemandangan HALAMAN 6 11 Training and Implementation Manual Tes Cepat STEP 3: If the student continues to answer the second test item correctly, 3 proceed with a higher number item. Conversely, if the student did not answer the second item correctly, move back to lower number or easier item. 4 STEP 4: Repeat STEP 2 and 3 until the highest number item that the student can answer correctly is identified. This item will be used to mark the student’s competency. 5 STEP 5: Mark student’s competency based on the last (highest number) item answered correctly. Plot student’s results into a visual map (Figure 2) to indicate where they are in the learning continuum. If a student is marked as having their grade- level competency, they will be plotted in the green box of their respective grade row. Students who are marked as having competencies below their grade level will be plotted to the left of the green box, specifically in the box of the respective grade- level competency they are marked in. Students who are marked as being illiterate or innumerate will be plotted in the red boxes. Figure 2. Visual Map of Student’s Competencies MANUAL RESULT RECAPITULATION FORM (IF THERE IS NO COMPUTER TO RECAP) BASIC COMPENTENCIES DISTRIBUTION BASED ON STUDENT’S AMOUNT AMOUNT READING NUMBER GRADE BMH BMKD KD1 KD2 KD3 KD4 KD5 KD6 OF SAMPLE STUDENT (0-1) (2-7) (8-11) (12-17) (18-23) (24-29) (30-35) (36-38) 1 2 3 4 5 MANUAL RESULT RECAPITULATION FORM (IF THERE IS NO COMPUTER TO RECAP) BASIC COMPENTENCIES DISTRIBUTION BASED ON STUDENT’S AMOUNT AMOUNT MATH NUMBER GRADE BMH BMKD KD1 KD2 KD3 KD4 KD5 KD6 OF SAMPLE STUDENT (0-1) (2-7) (8-11) (12-17) (18-23) (24-29) (30-35) (36-38) 1 2 3 4 5 12 CHAPTER 3 Digital Tes Cepat Mobile App Administration Manual Training and Implementation Manual Tes Cepat The digital Tes Cepat is administered through an android-based mobile application. There are 11 features to the digital mobile app Tes Cepat, with each catered to be used online/ offline and operated by a set of specific users (Figure 3). Figure 3. Digital Tes Cepat Mobile App Features and Users ADMINISTRATOR SCHOOL (USER COMMITTEES, TEACHER/ FEATURES NETWORK STUDENT OPERATOR COMMUNITY, PRINCIPAL PARENTS) Enter Home Page Online ✓ ✓ Data Synchronization of Online ✓ ✓ Questions and Answers Help Offline ✓ Sample Offline ✓ Input Administrator Data Offline ✓ Input Student Data Offline ✓ Tes Cepat Offline ✓ Check Student History Offline ✓ Merge Test Result Offline ✓ See Test Result Offline ✓ ✓ Send Test Result Online & ✓ Offline Figure 4. Digital Tes Cepat Mobile App Home Page button to change icon synchronization button button to do a Tes Cepat result button help button scan QR button home button profile setting button sampling button history button Require internet connection Does not require internet connection 14 Training and Implementation Manual Tes Cepat 1 STEP 1: Prepare the Mobile Application. Download the Tes Cepat app through Google Play Store. Open the Tes Cepat app on Android mobile phone, then log in to the app and select the corresponding school’s name of test subject. 2 STEP 2: Synchronize Test Item Bank. Perform this step in locations with stable internet connection. This step is done before each test period is conducted, as the item bank will differ across test periods. Remove item banks that are no longer active. In doing so, student data on these inactive items will also be automatically deleted. Ensure all recorded student data are synchronized to cloud storage before removing any old item banks from the app. 3 STEP 3: Read the Help Feature. This step can be done without internet connection. In the beginning of every training for test administrators, participants should read the list of frequently asked questions and answers (FAQs) in the Help feature before proceeding to other features in the app. 4 STEP 4: Select Sample Students. This process will be done per grade, or six times. The school will provide test administrators with student roster to identify the number of students in each grade. To identify sample students, go to “Sample” (Sampel) feature on the bottom of the home page, input the number of students in each grade, then click “Acak” (Randomize). The app will generate a list of numbers that will be used to indicate selected students from the roster. The app also generates alternative sample lists to replace initial sample students who are not available to take the test. 5 STEP 5: Perform the Test. The test should begin with reading, and continued with math. Click the “Tes Cepat” menu on the home page, then select subject. Fill in the names of both student and administrator. Make sure audio of the mobile phone can be heard clearly. Before working on the test, students will be given a trial mode for them to get used to operating the app. Administrator should inform students on the functions of the available buttons, including on how to click on the answer option of their choice. Whenever student is ready, begin the test. 6 STEP 6: Check history. Following student’s completion of the test, check their history in the “Riwayat” feature on the home page. Make sure their name is listed as it indicates a successful record of their data. 15 Training and Implementation Manual Tes Cepat 7 STEP 7: Compile Test Results. If the test is conducted in more than one device, results from every phone should be compiled. Pick a main device to collect results from the other phones. Through the “Hasil” feature, pull up QR codes of both reading and math test results from the other phones. The main device will scan these codes using the “Pindai QR” feature. When scanning, make sure the camera captures the codes completely, and that there is sufficient lighting. Check the completeness of the compiled result. This feature can only be done with devices that are registered under the same school. 8 STEP 8: Check Test Results. Check the visual map of test results by accessing the “Hasil” menu and clicking the subject of interest. Test results from different subjects or time periods can be compared by clicking the “Bandingkan” option on the bottom of the screen. 9 STEP 9: Download Test Data. To download the result in CSV & PDF format, go to the "Hasil" menu, select the subject to be downloaded, then select PDF/ CSV format to download 10 STEP 10: Report Test Results. The reporting process can be done with or without internet connection. If internet connection is available, synchronize the test results into a cloud storage through the “Sinkronisasi” (Synchronize) feature. If internet connection is not available, transfer CSV test results into a laptop using a USB cable. 16 CHAPTER 4 Digital Tes Cepat Web App Operating Manual Training and Implementation Manual Tes Cepat The digital Tes Cepat web application controls and stores overall activity and data from the digital Tes Cepat mobile app. This app can only be used online. There are 15 features in the digital Tes Cepat web app, with each operated by a specific user (Figure 5). Figure 5. Digital Tes Cepat Web App Features and Users LOCAL CENTRAL FEATURES AUTHOR OPERATOR ADMIN See test result ✓ Compare test result ✓ Upload answer data ✓ See and add test items ✓ View exam reports ✓ Download CSV data ✓ Download test reports ✓ Compare test result ✓ See Upload history ✓ Add subject and domain ✓ Generate or add school data ✓ See students’ data ✓ Generate, add, and remove the stakeholder accounts ✓ Generate, add, and remove the author accounts ✓ Generate, add, and remove the operator accounts ✓ 18 Training and Implementation Manual Tes Cepat Figure 6 maps out specific roles of each user, along with guideline to access and operate features of the Digital Tes Cepat web app. Figure 6. Digital Tes Cepat Web App Users’ Roles and Guideline LOCAL OPERATOR AUTHOR CENTRAL ADMIN Level Local National National Role Ensure all school-level Create and manage test Control the overall account data is reported to the web items and test booklets. to operate the Web App app. Check and manage Manage compiled results. system compiled results from all schools. On accessing Go to http://kiatguru. Go to http://kiatguru. Go to http://kiatguru. the web app kemdikbud.go.id , then kemdikbud.go.id , then kemdikbud.go.id , then log in using the assigned log in using the assigned log in using the assigned operator account. author account. central admin account. There is only one central admin account. On using To check and compare To add/ edit/ remove test To generate or add school the web app test results from all items, select menu “Bank data, select menu “School”. features districts, select “Laporan Soal” then the page will The school data needs Ujian” menu on the left bar. display list of test booklets to be uploaded before all stored in the system. other settings in the app. To upload results from This includes date of test school, select “Unggah booklet creation, date of To check sampled student Jawaban” menu on the left test booklet modification, data, select menu “Murid”. bar, then select the file that subject of the test This account has the want to be uploaded. booklet, title of the test authority to remove any booklet, author of the test student data. booklet, and status of the test booklet. The “Aktif” To generate, add, and status indicates the test remove local operator booklet’s availability to be accounts, select menu downloaded to the mobile “Operator” app. To generate, add, and To add Subject and remove author accounts, domains go to menu select menu “Author” “Metadata”. To generate, add, and To view reports on the remove other stakeholder test booklets, download accounts, select menu CSV data, download test “Stakeholders” reports, and compare test result. Select menu “Laporan Ujian”. To check history on uploaded test results, go to “Catatan Unggah”. This page is used to view test results that have been uploaded by local operators. 19 CHAPTER 5 Photo Story: Training Digital Tes Cepat to Community Members Training and Implementation Manual Tes Cepat STEP 1: STEP 2: Gather a group of community members In a smaller group, have community and teachers to be trained on the app. members sit around a table. Each person Explain the objective of the test, including should have a mobile device in front the role of community members as of them. They will be responsible to administrators. administer the test following the training. Teachers and the remaining participants may observe from the back of the room. STEP 3: STEP 4: Have them open the Tes Cepat app. Introduce each of the app features. Provide each person with a scrap paper for them to write their notes on Direct them to the HELP feature. throughout the training. 22 Training and Implementation Manual Tes Cepat STEP 5: STEP 6: Have them read through the content of Direct them to the SAMPLE feature. the HELP feature. STEP 7: STEP 8: Have them input a dummy number of Have them look through the list of students in the SAMPLE feature. sampled students. Make sure they understand how to identify sample students, both from the main and alternative lists, by the generated numbers. Have them write down these steps in the scrap paper, which will serve as their personal handbook during the test administration. 23 Training and Implementation Manual Tes Cepat STEP 9: STEP 10: Direct them to the TEST feature. Again, Have them fill name and gender of tested ensure they take notes on steps required student, and the constituency of the to go through this feature. administrator (themselves). Have them write this information on their note. STEP 11: STEP 12: Have them go through the “Trial” mode and experiment with the different buttons. Ensure they write these steps on Before proceeding to the test, their notes. demonstrate how to operate the preliminary “trial” mode or test items. Explain where the test problem is, where the answer options are, how to select an answer, and how to pause, resume, and quit the test. 24 Training and Implementation Manual Tes Cepat STEP 13: STEP 14: Whenever they are ready, proceed to the actual test mode. Have them go through the test as best as they can When everyone has gone through the test, demonstrate how to access and read the results map. STEP 15: STEP 16: Have them access and read through Take the time to make sure they have a the results map. Ensure they write the thorough understanding of the map, and required steps and information on their how to interpret it. notes. 25 Training and Implementation Manual Tes Cepat STEP 17: STEP 18: Demonstrate how to access and scan QR Present the merged results from their code of the results. Explain the function devices. of the code, and how to use it in merging results from all available devices. STEP 19: STEP 20: Regroup community members with In pairs of 2-3 people, have a community teachers. Inform them that they will be member assumes the administrator assigned into pairs to go through rounds role, while a teacher/ other community of test simulation. member plays the role of a student. Have them do the simulation for reading and math tests. In the end of the test, instruct them to pull up the results. 26 Training and Implementation Manual Tes Cepat STEP 21: STEP 22: Throughout the simulation rounds, Following the simulation, discuss and ensure that each pair is monitored. reflect on the training. Have community Without disturbing the simulation members and teachers to agree on the process, record any mistakes that the testing schedule for their school. administrator makes. These notes should be shared during the discussion part or the end of the training. 27