Abstract Information for Language Symposium
April 25-26, 2008

Assessing Students’ Cognitive and Affective Backgrounds in Culture Study at the Entry Level of Foreign Language Programs

Language: All

Level: Beginning

 One critical goal of foreign-language education is the development of intercultural understanding, fostering positive cross-cultural attitudes. Past studies (e.g., Pica, 1994; Damen, 2003) indicate that American college students are generally ill-prepared to meet culture-learning requirements, keeping experiential and psychological distance when immersed in the foreign culture. The present study investigates the cognitive and affective backgrounds brought by college students to culture study at the entry level of foreign-language programs. By assessing students’ pre-existing conditions the study identifies goal areas that must be addressed by beginning foreign-language programs to improve students’ cross-cultural attitudes. A culture survey has been developed, asking 1) what culture means to students, 2) what students know about the language-specific culture, and 3) how important culture is to students as a foreign-language learning objective. The cultural framework of the National Standards of Foreign-Language Learning (1999) serves as the theoretical framework. Preliminary findings indicate that the students construct the meaning of culture along the themes of “difference,” “uniqueness,” and “heritage”, all of which give “identity” to individuals. The students’ cultural knowledge centers around cultural products (e.g., food and clothing) and cultural practices (e.g., customs and manners) with language being cited as one example of cultural product and cultural practice. An overwhelming majority of the students considers culture study to constitute an important part of foreign-language study while expressing strong interest in learning about the “life-style” of other cultures. Findings suggest the needs for deliberate inclusion of cultural perspectives (e.g., beliefs and values) in cultural instruction.

Making the Assessment Mandate Work

Language: English as a Second Language

Level: Basic-Bridge Level Program Assessment

Whether at the course, program, or institutional level, assessment plays a major role in U.S. higher education, particularly when accreditation or re-accreditation is involved. The purpose of this presentation is to describe how one ESOL program took advantage of the institutional assessment mandated by an accrediting agency to improve placement testing, track student progress, update curricula, and provide opportunities for greater program involvement of instructional staff, including adjunct instructors.

The presenters are faculty administrators in a credit bearing intensive English program at a small private university.  They discovered that what had at first appeared to be a one-size-fits-all assessment directive could be modified to serve the more immediate needs of the ESOL program.  By identifying program goals and then incorporating procedures and outcomes expected in the university-wide initiative with assessment practices already in place within their unit, the presenters developed a plan to systematically analyze curricula and examine student learning and 2) record student satisfaction and look at the academic success of students admitted for degree work after they transitioned from ESOL to their major fields of study.  While the assessment process has presented numerous challenges and put a strain on ESOL program resources, it immediately resulted in positive change.  To demonstrate how the plan was implemented, this presentation will focus on how the rubric developed to quantify the assessment of student writing during the placement process was used to assess the development of students’ academic writing skills as they progressed through the program writing curriculum.

The role of e-portfolios in program assessment: one undergraduate program's experience

Language:French, German, Russian, Spanish

Level: All

In this presentation we will discuss the process by which one foreign language department has developed its mechanism for program assessment for undergraduate majors.  Our department began planning for program assessment in 2002, and implementation of its system commenced in 2003.  The electronic portfolio represents a key tool in the department's assessment plan, and all majors are required to develop such a portfolio for graduation.  The portfolios permit the department to assess itself by measuring undergraduates' progress throughout their careers, while providing degree candidates with a valuable tool that demonstrates their language proficiency and academic performance when applying for employment and graduate school.

We will offer written rubrics developed by the department for the evaluation of e-portfolios, as well as information on the course in which all majors enroll for the purpose of portfolio development.  Our system of program assessment also depends on student "buy-in," so we will include information about how we have secured their cooperation.  At the same time, our initiative remains a work in progress, so we will share our ideas for future revisions for our assessment initiative.  All programs with a commitment to excellence in undergraduate foreign language education will be interested in learning about this initiative.

Online language placement at Northwestern University; Challenges, Successes, Lessons Learned

Language: All

Level: All

Accurate placement is critical to students’ success in language programs and can also be used to evaluate effectiveness of curricula over time. The logistics imposed by conventional paper-based assessments place considerable stress on faculty since all testing usually occurs all at once. This leads many to consider different testing models such as online placement.

Northwestern University first developed online placement tests in 1996 for French, German, and Spanish, and in 2002 redeveloped the existing tests while adding eight additional languages. The development of effective online tests presented both technical and pedagogical challenges; it required careful, creative innovations, particularly for languages using non-Roman scripts. 

Following changes in the test and its technology, the overall language placement process evolved from an annual burst of frenetic activity to one of continually revolving placement, requiring paradigm shifts in the culture and practice of language placement. Some aspects, such as the forecasting of class enrollments and students’ access to the tests, have improved tremendously while others, such as the aggregate placement work effort, have not improved as much.

In this session, we will share an anecdotal history of the test development and the evolution of Northwestern’s language placement process by offering both a qualitative and quantitative assessment of the changes that have taken place. Based on these results we hope to share meaningful observations and suggestions to others who have recently implemented or are considering implementing an online mechanism for placement testing. 

This session will complement a series of poster sessions given by individual language faculty from five different languages concerning the specific challenges presented by online testing in their languages.

Quantitative Tracking of Oral Proficiency

Language: English

Level: All

Since building oral proficiency is now commonly a central goal for language programs, the demand for accurate, detailed oral assessment continues to grow. Specifically, there is benefit both for program development and feedback to students in implementing procedures to track oral proficiency through assessment measures fully independent of course grades and other scores students obtain from oral communication courses. We describe such a curriculum-external oral proficiency assessment procedure implemented at George Mason University’s English Language Institute (GMU ELI). Although valuable information about overall proficiency can be gained from tests requiring integration of skills in test tasks, such results can also prove difficult to convert into accurate ratings of oral proficiency as a distinct factor, making feedback on oral proficiency less transparent. Therefore, to the extent possible, GMU ELI’s assessment procedure aims at evaluating oral proficiency alone. The assessment attempts to cultivate student utterances that depict authentic language use. Generation of authentic language is fostered by assessing students in person in real time rather than from recordings and by assessing students in pairs rather than in isolation, allowing for natural student-to-student interaction. From this assessment, it has been possible to provide a quantitative measure of oral proficiency for any student. In addition, with data gathered over consecutive semesters, an accumulation of statistics has permitted the ELI to track oral proficiency over time in quantitative terms. Numerical measures have been informative for individual students, classroom instructors, and for evaluation of the program’s success in building oral proficiency as an instructional goal.

Testing for Proficiency: How and why we do it

Language:Spanish, but can apply to other languages

Level: Intermediate

Is testing for proficiency a valid assessment tool?  What are some of the problems that occur when testing for proficiency in a second language?  The University of Minnesota has been using proficiency tests as a way for students to fulfill the second language requirement in the College of Liberal Arts since 1986.   In most cases, the test determines whether the student has intermediate-level  (high, mid or low, depending on the language and modality being evaluated) proficiency in reading, writing, listening and speaking.  The reading and listening sections of the Spanish Language Proficiency Exam (LPE) have been delivered on the computer using Authorware.  Currently, we are changing the system so that the tests can be delivered online.  The writing test is given on paper, although we will have the option to test via the computer as well.  The speaking test is a one-on-one oral interview, based on the ACTFL OPI.  We hope that the new system for delivering the reading and listening (and writing) tests will allow us to create more test items.  We hope to continue the one-on-one oral interviews even though they are labor intensive.  Some of the issues that we have addressed and continue to look at are: 1) how can we keep the test secure?  2)  is this a valid test?  3) is it easy for students to cheat?  4) are graders adequately trained?   5) does this test continue to fulfill its purpose of measuring functional language ability? 6)  is there something better out there?

Language Placement of Advanced L2 Readers: The Effects of Criterion-Referenced Self-Assessment

Language: Spanish

Level:Advanced Learners

Brantmeier (2006) found that self-assessment (SA) of second language (L2) reading ability is not an accurate predictor for computer-based testing or subsequent classroom performance. With 359 advanced L2 university-level learners, the present study attempts to provide new empirical evidence concerning the use of SA, when measured via criterion-referenced items, as a predictor of reading scores on a computer-based placement test and subsequent reading achievement. Before reading, participants completed a contextualized, criterion-referenced instrument for SA that was created from the European Language Portfolio. A four-item contextualized SA questionnaire was also completed after reading (Tobias & Everson, 1998). Participants completed the following: self-assessment questionnaire taken prior to computer-based test, computer-based test of 8 readings, 2 in-class readings, written recall, multiple-choice, topic familiarity questionnaire, and post-reading SA questionnaire. Findings indicated that a descriptive and criterion-referenced questionnaire for SA may be a reliable predictor of both reading scores on a computer-based test and subsequent classroom performance when comprehension is measured via sentence completion and multiple choice items. The post-reading SA instrument proved reliable for all three comprehension tasks. Results hold important implications for both language placement directors and test constructors. Certain SA items could be used in combination with the computer-based test for placement of advanced learners. These SA criteria could aid in the placement of readers with marginal reading scores (readers who score on the border between intermediate and advanced). Moreover, SA can help instructors diagnose strengths and weaknesses of individual readers. Assessing reading ability accurately may also lead to greater self-awareness and could contribute to goals to produce lifelong L2 readers.

Evaluating academic summary writing: An analysis of paraphrases and holistic rating

Language: English as a Second Language

Level:College-level (low intermediate and low advanced)

The importance of paragraphing strategies in academic writing is well established in the literature.  A recent study by Keck (2006) further examined the typology of paraphrases and the distribution of different types of paraphrases in summaries written by L1 and L2 writers.  Using the typology developed in Keck (2006), the present study analyzed academic summary paragraphs written by college-level English as a Second Language writers at a U.S. university (n=90).  The present study explored two aspects of evaluating academic summaries.  First, does the distribution of paraphrase types between two different levels of L2 proficiency resemble the distribution pattern observed between L1 and L2 writers in Keck(2006)?; In other words, what is the nature of developmental changes in L2 writing with respect to paraphrasing skills?  Second, what is the relationship between two different measurements of academic summary paragraphs; namely, evaluation based on the typology of paraphrases and evaluation based on instructors’ holistic rating of the summaries?  The data in the present study came from authentic classroom assessment materials, and the findings from the analysis have serious implications for the practice of both teaching and assessment in the same kind of classes from which the data were made available; Furthermore, the findings also have direct application to training future teachers in an MA-TESL program.

Reference
Keck, C. (2006). The use of paraphrase in summary writing: A comparison of L1 and L2 writers. Journal of Second Language Writing, 15, 261-278.

Do online and in-class oral assessment tasks provide complementary information about oral achievement? 

Language: German

Level: Intermediate

The development of oral skills is an integral part of communicative language programs. Naturally, oral achievement should be periodically tested to reflect programmatic goals and provide students feedback on their progress. Yet, language instructors’ anecdotal descriptions point to two major problems with oral tests: practical issues, such as the time necessary to administer an oral test, and lacking task and evaluation criteria. Recent research on oral language skill development has mainly focused on the refinement and validation of standardized oral proficiency tests (e.g. OPI, VOCI, COPI). Not much attention has been paid to aspects of oral achievement tests. In turn, recent pedagogical literature advocates the use of online-testing tools (Rowekamp & Treece, 2007; Trubiano & Heller, 2007). However, there has been no assessment of which aspects of speaking abilities can be tested with on-line tools.

The incentive of the current investigation was to explore the effect of an on-line and an in-class oral test on students’ ability to demonstrate their L2 speaking abilities. In addition, the study assessed learners’ perception to be able to demonstrate adequately their strengths in the respective test format. 49 intermediate learners of German participated in two oral achieving tests: on-line responding individually to oral voice prompts or in-class in a small group discussion. In addition, students filled out a questionnaire that asked about factors that influenced their test performance as well as their perception of the test format. The transcription of the recordings will be assessed for the following criteria: quantity of use of collocations and situation specific expressions, length and complexity of responses, holistic communicative comprehensibility, and accuracy (verb endings, tense markers, word order). The discussion of the results will focus on how the two test formats elicit different speaking abilities and how they can be implemented in the basic foreign language curriculum.

A process-oriented online pragmatics test module

Language: Chinese

Level: Intermediate

Assessing interlanguage pragmatic knowledge has been a topic of recent research interest, but the effect of multimedia elements on learners' behaviors in assessment instruments has not been studied. Currently-used pragmatics assessment measures make use of writing tasks, multiple-choice tests, audio recording, role-play, and self-assessment measures. Web-based pragmatics tests, according to Roever (2006), represent psychological reality that is most similar to what learners encounter in real-time communication. However, a common limitation of these tests is that they normally present prompts in written form. Thus, the validity of the test is likely to be affected by learners’ reading proficiency.

This study used a Chinese pragmatics test that consists of a multiple-choice task and a multiple rejoinder DCT. In both tasks, learners had access to audio stimuli in addition to written prompts. The research questions are: 1) To what extent do learners take advantage of audio stimuli during the test? 2) Does task type affect the extent to which learners use audio stimuli? 3) Will those who access the audio stimuli perform better than those who don’t?

A computer program was developed specifically to capture participants’ performance, including the frequency of audio selection. Every action that learners made during the test was recorded by the computer for analysis. However, simply recording what learners do does not indicate why they do it. Therefore, in addition to the quantitative analysis, interviews were conducted to collect students’ perceptions of the audio stimuli to provide a qualitative analysis. The results of this study further our understanding of the utility of including audio in online tests.

Understanding Assessment: A Guide for Foreign Language Educators

Language: English

Level: All

Because of increased focus on assessment, it is critical that language educators understand testing. However, there is often a divide between the formal field of language testing and teachers’ understanding of assessment. To address this concern, the Center for Applied Linguistics (CAL), with funding from the U.S. Department of Education, is developing an online assessment literacy tutorial for language educators. The term “assessment literacy” refers to educators’ knowledge and understanding of assessment (Stiggins, 1997; Stoynoff and Chapelle, 2005; Boyles, 2005). CAL’s tutorial is a free, web-based resource to provide language educators with information about testing.

This presentation will discuss opportunities and challenges in promoting assessment literacy among language educators, using the online tutorial as an example. The presentation will include information on the tutorial’s development and content, including previous research on instructors’ assessment knowledge. The tutorial’s development included focus group research, interviews, and online surveys of foreign language and test professionals. The information gathered from these methods resulted in tutorial revisions reflective of needs expressed by educators.

The tutorial, Understanding Assesment, consists of six modules that define and discuss essential testing terms. Users explore different aspects of the language assessment process and put this knowledge to use to address a real-life assessment scenario at the end of each module.

The end of the tutorial gives users a chance to define their own testing needs and provides them with links to further resources about assessment.

References
Boyles, P (2005) “Assessment Literacy.” In Rosenbusch, M. (ed) National Assessment Summit Papers. Ames, Iowa:
Iowa State University, 11-15.
Stiggins, G. (1997) Student Centered Classroom Assessment. Upper Saddle River, NJ: Prentice Hall.
Stoynoff, S., & Chapelle, C. A. (2005). ESOL tests and testing: A resource for teachers and program administrators.
Alexandria, VA: TESOL Publications.