如果你将要设计一个培训项目，并且测评是一个必不可少的部分，那么以下这10个问题，可以有助于你分析和制定一个最恰当的方案：

1. 能用于培训和测评的时间分别是多少？
2. 测评的场地是否有特殊要求？
3. 你是否需要为培训和测评的全程做一个非常准确的记录以留底备案？
4. 你的客户单位是否有特许要求？
5. 该评估是否需要按照有关政策、法律法规和行业标准来制定和执行？
6. 该评估是否需要反映行业或者工作场所的文化特征？
7. 是否存在部分学员的工作时间与其他学员存在较大差别而导致他们无法在同一时间参加统一的测评？
8. 该培训项目和测评是否需要镶嵌到学员工作的日程当中去？
9. 你是否需要其他未参与培训的员工的协助才能完成测评？
10. 你的学员是否在语言、残疾、疾病等方面存在需要你予以特殊照顾？

对上述问题进行分析后，我们可以在以下几种常用的测评方法中，选择其中一种或者多种组合，制定最终的方案：

1. 模拟活动或实操活动: 学员在一个模拟或者实际的环境里完成指定的任务，测评者从中观察
2. 结构性活动: 学员通过回答结构性的问卷以展示其能力，或者通过参加系统性的考试
3. 提问和作业: 学员通过口头或者书面的形式提交作业
4. 档案: 学员收集自己的学习材料、学习成果或者第三方的报告以编制自己的培训档案
5. 先前学习认证或者当前能力认证: 若学员在接受培训之前已参加过类似培训，或者目前已经在从事相关工作，那么他们可以通过先前的测评结果或目前的雇主证明直接获得相应的测评结果，而无需重复培训或参加测评

在企事业单位高层上评估用于支持教学的新技术的核心要素

何文超

E-mail: w.r.he@unsw.edu.au

澳大利亚新南威尔士大学

这是一个约5分钟的关于评估教育技术产品的简要演讲展示。我采用我创建的DOAMI模型——Decisions（决定）、Objectives（目标）、Approaches（途径）、Management（管理）、Interpretation（解析），来讲解从企事业单位高层的角度，如何评估用于教学的新技术。

Evaluation Proposal for an Online

IELTS Writing Tutoring System

Wenchao He

Faculty of Education, University of Wollongong, Australia

1    Introduction

This plan describes the background, purposes, limitations, stakeholders, decisions, questions, methods, participants, instruments, procedures, logistics, and timeline for the evaluation of Chao Yue IELTS Online Tutoring System, which is an e-learning program for IELTS candidates who have purchased IELTS preparation books on www.cyielts.com.au. The methodology, procedures, and instrumentation included in this plan are based on several meetings between members of the development team and the evaluation team, as well as a review of several draft design documents. The primary objective of this evaluation is to provide timely and accurate information to support decisions regarding the enhancement and marketing of the e-learning program. The evaluation project will involve students and teachers who are to go through the standard instructional process and complete tests and/or questionnaire and/or interview. The system log will also be used for data analysis. The evaluation project will start on 1 June 2009 and finish by the end of the month. The estimated cost will be $27,200. This plan is to be reviewed, revised and approved in the first week of June.
 

2    Background

Chao Yue IELTS Online Tutoring System (http://e-learning.cyielts.com.au) is a Moodle-based learning management system providing IELTS candidates with tutoring service for their IELTS preparation. IELTS stands for International English Language Testing System. People from non-English speaking countries should take IELTS test and use the test result to apply for courses and/or jobs offered in English speaking countries (e.g., Australia). IELTS result is also a requirement for applying for a permanent visa and/or immigration to English speaking countries. Chao Yue IELTS sells IELTS preparation books online and provides the customers with tutoring services. The operation is that the students firstly purchase IELTS preparation books online, then receive and use the books for self-study. Once they have any difficulties in studying the materials, they can ask the online tutors for help on the e-learning system. When they complete an IELTS writing task, they are to submit the essays to the e-learning system and the tutors will provide them with feedback and a web-based customised small course with various learning activities. The tutoring service for the first essay that the student submits is free of charge. After the first round, the student can decide to pay for extra writing tutoring service. So basically, Chao Yue IELTS’s e-learning system offers two kinds of services – general questions answering and IELTS writing tutoring. This evaluation will focus on the IELTS writing tutoring part of the system. A detailed tutoring process is illustrated in Figure 1.

Besides the one-on-one tutoring mode in the IELTS Writing Tutoring Section, once the tutors identify the students’ common problems in IELTS writing, they will organise some instructional materials and make them available to the students. The tutors also moderate an online learning community where students discuss about their learning and help each other.
 

•    Figure 1    Chao Yue IELTS Writing Tutoring Process

 
3    Purposes

The overall purpose of this evaluation is to provide the directors and tutors at Chao Yue IELTS with the timely, accurate information required to support decisions regarding the enhancement and marketing of the IELTS Writing Tutoring program. This evaluation is primarily an effectiveness evaluation in the sense that the information collected will be used as the basis for determining whether the program accomplishes its objectives within the immediate or short-term context of its implementation. As a result of this evaluation and the decisions and actions stemming from it, the e-learning program should be ready for being further promoted in the IELTS preparation education market, or should be enhanced before being further promoted.
Specific sub-purposes of this evaluation are:

*    to collect information for promoting the e-learning program,
*    to investigate the reasons why the students subscribe to or not subscribe to extra tutoring service after they receive the first round of service for free,
*    to measure to what extent the program helps the students improve their English writing skills for IELTS, and
*    to decide if the program needs to be modified for students’ better learning outcome.

 
4    Stakeholders

The clients for this evaluation are the managers of Chao Yue IELTS. Important primary stakeholders include the tutors and marketing personnel involved in the development effort and the students of the e-learning system. The technical staff who is in charge of maintaining the e-learning system and the customer service staff may also be viewed as secondary stakeholders in this evaluation, but dissemination of the evaluation plans and results will be controlled by the clients.

 
5    Decisions

To empower this evaluation to provide timely and accurate information to support decision making, we have identified a set of anticipated decisions that will be made about the e-learning program. Most of these decisions must be made regardless of the quantity and quality of information available to the decision makers, but the evaluation will ensure that the decisions are informed by the best possible information. The following decisions are anticipated:

*    Advertising and marketing decisions will be made about the e-learning program by the managers of Chao Yue IELTS.
*    Modifications will be made in the e-learning program to improve its effectiveness and appeal.
*    Strategies will be made for persuading students subscribe to extra IELTS Writing tutoring service after the free-trial.
 

6    Questions

The following questions will be addressed during this effectiveness evaluation:

*    How are the knowledge and skills of the students changed by the program?
*    What are the students’ reactions to free trial of the program?
*    What are the factors in students’ decision to subscribe or not to subscribe to the program after free trial?
*    What are the fee-paying students’ reactions to the program?
*    What corrections must be made to the program?
*    What enhancements can be made to the program?
 

7    Methods

7.1    Data Collection Matrix

No single evaluation design can encompass the six major questions specified for the evaluation of the IELTS Writing Tutoring e-learning program. Therefore, a mixed methods evaluation design and multiple data collection strategies will be utilised to collect the information required to address these questions. The data collection strategies include questionnaire, interview, automatic data collection and tests. Table 1 specifies the data collection methods and their corresponding evaluation questions.

•    Table 1    Data Collection Matrix

7.2    Learner Questionnaire

7.2.1    Questionnaire Design

Questionnaires are one of the most frequently used methods of collecting data for effectiveness evaluation (Reeves and Hedberg 2008). The Learner Questionnaire for this evaluation will be addressing all the six research questions. The current enrolled and past non-paying and fee-paying students will be invited to complete the questionnaire. Although the questionnaire may be anonymous, the two groups of students will be labelled so that we can differentiate their responses. The objectives of the questionnaire are as followed:

*    to clarify what kinds of benefit in terms of knowledge and skills, if any, the students have got from the program,
*    to let the students recall and specify their reactions to the free trial,
*    to know about the subscribers and non-subscribers’ decision making process in terms of subscription,
*    to let the fee-paying students recall and specify their reactions to the program, and
*    to obtain the data about the weakness and strengths of the program from the students’ perspective.

Generally, the questionnaire will predefine a series of variables represented as a list of 5-point Likert scaling questions. The questionnaire will be web-based so it can be completed on the Internet. The respondents will be given an option to address if they are willing to be contacted for a follow up e-mail or telephone interview.

7.2.2    Rationale of the Questionnaire

A questionnaire is an effective tool to gathered responses in a standardised and objective way. It does not require too much time and effort for evaluators to prepare and analyse, and for students to complete (see Table 2). So this method is relatively quick to collect information. In this evaluation context, a web-based questionnaire is easy to develop in Moodle because this is an existing function built in the system.

•    Table 2    Requirement of Questionnaire (Harvey 1998)

However, it is believed that questionnaire’s response rate could be low, specially the questionnaire can not be delivered and responded to in face-to-face class time (e.g., in an e-learning context). This is because students may not be willing to answer the questions. They might not wish to reveal the information or they might think that they will not benefit from responding. Furthermore, the students may answer superficially especially if the questionnaire takes a long time to complete (Harvey 1998). Thus, we should simplify the questionnaire and most of the questions designed are of 5-point Likert scaling type. Where possible, every respondent should have an opportunity to enter a draw to prizes.

7.3    Teacher Interview

7.3.1    Interview Design

In the teacher interview session, the teachers who are offering IELTS writing tutoring service at Chao Yue IELTS will be asked semi-structured questions regarding the students’ progress and the quality of the program. The objectives of the interview are:

*    to obtain the information about the teachers’ impression and observation of the students’ progress, and
*    to let the teachers address their concerns about the design of the program including the instructional process detailed in Figure 1, the current configuration of the Moodle system, and the instructional materials they are using for developing customised small courses for the students.
The semi-structured interview will start with broad questions and then the teachers are free to add extra information to help achieve the objectives of the interview.

7.3.2    Rationale of the Interview

Generally, there are three types of interview design: (1) standardised interview, where questions are asked in the same way across a sample population by different interviewers, (2) semi-structured interview, where the interaction is kept focused on the same ground while allowing individual experience to emerge, and (3) conversational interview, where the evaluator responds quickly to individual differences and situational changes. According to Harvey (1998), all the three types of interview require much time and effort for preparation, conducting the interview and analysis afterwards (see Table 3). However, as the sample for the interview is not large – less than 5, this part of the evaluation would not be extremely time-consuming. Amongst the three types of interviews, the semi-structured one is adopted because the anticipated information from the interview is based on the teachers’ tacit impression, if we choose standardised interview, some important information may be missed; if we choose conversational interview, the teachers may provide too much information which is irrelevant to the predefined evaluation questions.

•    Table 3    Requirement of Interview (Harvey 1998)

7.4    Automated Data Collection

7.4.1    Automated Data Collection Design

In the Moodle system, users’ every single activity will be logged. The e-learning program includes a number of discrete modules, with each module containing several different types of instructional options, e.g., tutorials, quizzes, practice exercises, and mastery tests. The students’ paths through the modules, their choices among the instructional options, and their quiz and test scores, will all be recorded. However, not all the logged data will be used for the evaluation. Our focus is on the following aspects:

*    the frequency of usage,
*    the time spent on each activity and/or resource,
*    exit page (on which page the users are likely to exit the system), and
*    the comparison between expected activity paths and actual activity paths.

The primary objective of collecting the information about the above aspects is to presume the users’ attitude towards the program (e.g., high frequency and moderate time of usage may imply positive attitude while unexpected and sudden exit could be a sign of dislikes). Another objective is to identify any technical problems within the program which may be indicated by unexpected exit and/or unexpected activity paths.

7.4.2    Rationale of Automated Data Collection

Automated data collection is a step-by-step recording of user interaction with an e-learning program. The level of detail recorded is determined by a purpose-built program and can include mouse clicks, menu calls, and all operations on objects. Modelled on a “record” facility, log data is a useful way of tracking user preferences and navigational choices. This approach does not require any attention from the students and administration because once the logging function is turned on, everything is done automatically. However, the preparation time could be long because not all the data is useful for evaluation purpose (see Table 4). Evaluators need to determine what data needs to be logged, how it should be recorded and summarised and what level of detail of activities should be logged (Harvey 1998).

•    Table 4    Requirement of Automated Data Collection (Harvey 1998)

In the current evaluation context, the logged data is a useful resource for presuming users’ reaction and help refine the questionnaire and interview protocol. The reason that the logged data should be linked to the questionnaire and the interview protocol is that the data does not answer “why” questions and further evaluation is necessary to find explanations for occurring phenomena (Harvey 1998). We acknowledge that this approach has such nature so the logged data will not form a strong evidence for the evaluation result.

7.5    Pre- and Post-Testing

7.5.1    Pre- and Post-Testing Design

As the e-learning program being evaluated is an exam preparation program, to determine its effectiveness, a pre- and post-testing approach should be applied and the result can demonstrate if students are making progress or not. The tests are to solely answer the first evaluation question – “How are the knowledge and skills of the students changed by the program?” The subject of the tests will be the fee-paying students who have paid for extra five rounds of IELTS writing tutoring process (see Figure 1). The objective of the tests is to measure to what extent the five rounds of IELTS writing tutoring service has helped the students raise their IELTS writing scores.

7.5.2    Rationale of the Testing

Tests are widely accepted as an effectiveness evaluation strategy, but we should be wary of tests unless you have evidence of their reliability and validity (Reeves and Hedberg 2008). Pre- and post-testing’s result could be easily affected by external factors which influence student learning. It would be very wrong to assume that we can discount a variety of external factors which will affect student performance when considering the reasons for variation between pre and post test responses (Harvey 1998). The reason why this approach requires long time on preparation (see Table 5) is not only about the test questions preparation but also about organising the students (e.g., trying every effort to exclude external factors, recruiting at least 60 students in order to allow us to make reliable statistical conclusions from the data, etc.).

•    Table 5    Requirement of Pre- and Post-Testing (Harvey 1998)

Despite that, we still need to include this approach in the evaluation because this is the only method that can objectively and quantitatively demonstrate the effectiveness of the e-learning program. One of the evaluation purposes is to collect information for promoting the e-learning program. The result of the tests will become an important resource to response to this predefined purpose.

7.6    Procedure of the Evaluation

The evaluation process can be divided into the following phrases:

*    The evaluators recruit students and teachers for the evaluation. The students should include non-paying students and fee-paying students.
*    The administrator turns on automatic logging function in the Moodle system.
*    The evaluators conduct pre-test for fee-paying students who have paid for five rounds of IELTS writing tutoring process.
*    The students go through their study process.
*    The evaluators analyse the logged data of the students’ activities in the learning system.
*    Based on the analysis of the logged data, the evaluators revise the questionnaire and the interview protocol.
*    When the non-paying students finish their first round free trial, they are given the questionnaire to complete.
*    When the fee-paying students finish their five rounds of study process, they are to take the post test and complete the questionnaire.
*    The evaluators conduct the interview with the teachers.
*    The evaluators analyse all the data.
*    The evaluators conduct follow-up interview with students and teachers, where necessary.
*    The evaluators compose the evaluation report.

 
8    Participants

The participants in this effectiveness evaluation will be:

*    2 evaluators
*    60 fee-paying students
*    60 non-paying students
*    4 teachers

 
9    Sample Instruments

9.1    Student Questionnaire

Appendix A includes the Student Questionnaire that the students participating in this evaluation will complete after they finish the free trial (for non-paying students) or after they finish the five rounds of tutoring process (for fee-paying students).

9.2    Teacher Interview Protocol

Appendix B includes the protocol for teacher interview that will complete with the four teachers who are currently offer the tutoring service in the e-learning program.

9.3    Automatic Data Collection

Appendix C includes a list of all the data to be collected by the computer while users are studying within the e-learning program. This data includes both navigation paths and response choices.

9.4    Pre-Test and Post-Test

Appendix D includes a sample of pre-test. All the recruited students should take the pre-test before they enrol in the e-learning program. The pre-test will take one hour to complete. It includes two writing tasks which are standard IELTS Writing tasks.

Appendix E includes a sample of post-test. It is also a standard IELTS Writing test. The difficulty of the pre- and post-tests are the same.
 

10    Data Analysis

10.1    Student Questionnaire

As for the Likert Scale questions, the evaluators are to check the answers for values out with expected ranges and inconsistencies. The results will be presented in bar charts, histograms, and/or scatterplots.
As for the open ended questions, the evaluators are to check the answers to see if they have common responses that can be coded up in some way. The coding technique will be used to transfer the qualitative data into quantitative data.

10.2    Teacher Interview

The evaluators are to check the interview transcripts to see if different teachers have common responses that can be coded up. As for their differences, the evaluators are to summarise them and present the result in a concise format.

10.3    Automatically Collected Data

The automatically collected data will be analysed and will form the basis for the evaluators to presume the users’ thoughts as they were going through the tutoring process. The assumption will be reflected on the revision of the questionnaire and the interview protocol.

10.4    Pre-Test and Post-Test

The test scores will be input into an SPSS database and will be calculated out whether there is significant difference between the pre-test and post-test.
 

11    Deliverables

Below is an outline of how the evaluation findings will be reported:

11.1    Progress of Students
11.1.1    Self-Rated Progress
11.1.2    Pre- and Post-Tests
11.1.3    Teachers’ Observation
11.2    Free Trial
11.2.1    Students’ Reaction to the Free Trial
11.2.2    Factors that Influence Students’ Decision to Subscribe to Extra Tutoring Service after Free Trial
11.2.3    Students’ Performance Described by Teachers
11.3    Paid Tutoring Service
11.3.1    Students’ Reaction to the Program
11.2.2    Students’ Performance Described by Teachers
11.4    Corrections to Be Made
11.4.1    Instructional Process
11.4.2    Instructional Materials
11.5    Enhancements to Be Made
11.5.1    Instructional Process
11.5.2     Instructional Materials
 

12    Limitations

12.1    Individual Learners’ Learning Experiences Vary

Although the trained teachers are to provide the service in a standard process using the same instructional strategy, individual learners’ learning experience will not be the same. This is because the writing tasks each student does are not the same, either as the teachers’ comments and the customised small courses. This may influence the reliability of the evaluation, especially for the student questionnaire part.

12.2    Difficulty in Controlling the Pre- and Post-Tests

The pre- and post-tests require the students complete the writing tasks online individually in one hour. The students are supposed not to look up words in dictionary or have any reference materials. However, this is not controllable because the evaluators can not monitor the testing process directly (e.g., seeing the students doing the tasks without any form of assistance). If any students do not follow the instruction, the reliability of the tests could be doubtful.
 

13    Logistics

Two evaluators will coordinate the implementation of this evaluation plan, including scheduling, data collection, and data transmission, with the course manager for this project. All data will be processed, analysed, interpreted, and reported by the two evaluators. All reports will be provided to managers and the teachers at Chao Yue IELTS. Further dissemination of the evaluation findings will be determined by the managers. Additional details about the logistics are found in the timeline section of this plan. The due dates for various deliverables are also specified in the timeline.

 
14    Timeline

 
15    Budget

 
16    References

Harvey, J 1998, Evaluation Cookbook, Learning Technology Dissemination Initiative, Edinburgh.
Reeves, T C & Hedberg, J G 2008, Evaluating e-Learning: a user-friendly guide (in press).

 
Appendix A    Student Questionnaire

Dear Student,

We are currently running an evaluation project for our e-learning program. We need your feedback to help us improve the IELTS writing tutoring service. Your responses are anonymous. Once you have finished the questionnaire, you will be entered into a draw to win one of the five $100 Coles Gift Cards. It will approximately take you 10 minutes to complete the questionnaire. Thank you for your time!

1. Please circle your responses to the following statements using the scale below.

1 = Strongly Disagree    2 = Disagree        3 = Neither Agree or Disagree
4 = Agree            5 = Strongly Agree

a. The program helped you identify your weakness in IELTS writing. 1 2 3 4 5

b. The program helped you identify your strengths in IELTS writing. 1 2 3 4 5

c. The tutor’s comment helped you understand how IELTS writing is scored. 1 2 3 4 5

d. The customised small course helps you overcome your weakness in IELTS writing. 1 2 3 4 5

e. Your IELTS writing skills have been improved at the completion of the program. 1 2 3 4 5

d. You satisfied with the free trial. 1 2 3 4 5

e. You decided whether to subscribe to extra tutoring service based on money for value. 1 2 3 4 5

f. You decided whether to subscribe to extra tutoring service based on your experience in the free trial. 1 2 3 4 5

g. You decided whether to subscribe to extra tutoring service based on your availability. 1 2 3 4 5

For Fee-Paying Students:

h. You think your decision to subscribe to extra tutoring service was correct. 1 2 3 4 5

i. You satisfied with the quality of the extra tutoring service. 1 2 3 4 5

j. You actively involved in the learning activities in the paid tutoring process. 1 2 3 4 5

2. What is the best aspect of the e-learning program?

3. How would you improve the e-learning program?

4. Please write any other comments about this e-learning program.

 
Appendix B    Teacher Interview Protocol

*    From your impression and observation, what progress did the students make during the studying this e-learning program?
*    What do you think of the current instructional process?
*    What do you think of the current configuration of the Moodle system?
*    Do you find it difficult or easy to organise relevant instructional materials for developing customised small course for the students?

 
Appendix C    Automated Data Collection

Appendix D    Pre-Test

Task 1

You should spend about 20 minutes on this task. The table below gives information about changes in modes of travel in England between 1985 and 2000. Summarise the information by selecting and reporting the main features, and make comparisons where relevant. Write at least 150 words.

Average distance in miles travelled per person per year, by mode of travel

Task 2

You should spend about 40 minutes on this task. Write about the following topic: Successful sports professionals can earn a great deal more money than people in other important professions. Some people think this is fully justified while others think it is unfair. Discuss both these views and give your own opinion. Give reasons for your answer and include any relevant examples from your own knowledge or experience. Write at least 250 words.

 
Appendix E    Post-Test

Task 1

You should spend about 20 minutes on this task. The graph and table below give information about water use world wide and water consumption in two different countries. Summarise the information by selecting and reporting the main features, and make comparisons where relevant. Write at least 150 words.

Global water use by sector

Water consumption in Brazil and Congo in 2000

Task 2

You should spend about 40 minutes on this task. Write about the following topic: Today, the high sales of popular consumer goods reflect the power of advertising and not the real needs of the society in which they are sold. To what extent do you agree or disagree? Give reasons for your answer and include any relevant examples from your own knowledge or experience. Write at least 250 words.

Usability Evaluation of E-Learning Environment