文字與圖像之跨媒介研究：以 EFL 大專生自編雜誌為例｜國立臺灣科技大學博碩士論文系統

簡易檢索 / 詳目顯示

回結果列表

研究生：	劉孟婷 Meng-Ding Liu (Michelle Liu)
論文名稱：	文字與圖像之跨媒介研究：以 EFL 大專生自編雜誌為例 Investigating Inter-Semiosis Between Text and Image: An Analysis of EFL College Students’ Editorial Works
指導教授：	李思穎 Sy-Ying Lee 駱藝瑄 Yi-Hsuan Gloria Lo
口試委員:	李思穎 Sy-Ying Lee 駱藝瑄 Yi-Hsuan Gloria Lo 廖美玲 Meei-Ling Liaw
學位類別：	碩士 Master
系所名稱：	人文社會學院 - 應用外語系 Department of Applied Foreign Languages
論文出版年：	2021
畢業學年度：	109
語文別：	英文
論文頁數：	138
中文關鍵詞：	英文外語識能教育、多模態性、多元識能、文字與圖像之關係、系統功能多模態言談分析
外文關鍵詞：	EFL literacy education, Multimodality, Multiliteracies, Text and image relations, Systemic Functional Linguistics Discourse Analysis (SFMDA)
相關次數：	點閱：319 下載：2
分享至:	分享至facebook 分享至twitter

查詢本校圖書館目錄查詢臺灣博碩士論文知識加值系統勘誤回報

近年來，各種不同多模態教學的應用，例如數位故事創作，簡報，影片製作等在第一語言以及第二語言教育中有日漸增加的趨勢。這個從傳統語言教學導向到多模態語言教學導向的轉變是經由當前不斷進步的科技技術而有所推動的。這也證明了傳統上以語言為本之教學法已不足以發展本世紀所需要的新識讀技能。
由於使用固定架構來分析英文外語學習者結合傳統與新語文素養之自編雜誌的研究數量不多，因此本研究採用社會符號學的觀點，並選擇了系統功能多模態言談分析（SFMDA）來查證：（1）外語學習者使用後設功能的頻率和多樣性；及（2）舉出外語學習者所可能表現出之有效和無效溝通意圖的範例。這項研究的目的為向老師和外語學習者介紹如何詮釋多模態文本，且希望能夠為此多模態文本提出一個全新及有效的架構。
本研究使用了兩個架構（Unsworth，2006； Wu，2014）對學生在自編雜誌中所表現之溝通意圖進行編碼，並且透過讀者的觀點來分系學生的溝通意圖。研究結果表明學生在文本中利用了多種有意義的後設功能元來吸引目標讀者，且學生的表現也可以從三種面向來檢視：（1）自編雜誌中後設功能的使用次數，（2）多種後設功能的組合，以及（3）每頁平均使用之圖像。由於自編雜誌的性質是以文字為主，圖像為輔，學生較著重於利用圖像來擴展雜誌中的文字敘述，而不是利用其餘的後設功能來連結文字與圖像。此研究也進一步推論了架構中「冗餘」的兩種變化。
此外，由於學生在多模態文本製作裡缺乏訓練，且對跨媒介的各種意義形成關係缺乏理解，因此在自編雜誌中時常看到文字圖像組合無法被分析的情況以及與文字和圖像相關之錯誤。這進一步的顯示學生對多模態的理解仍然不完善，而語言教師有責任及義務來教育和增強學生對多模態文本的認識以及運用多模態資源以達饒富意義之溝通目的。此外，無論是所接收之資訊輸入或欲表現之作品產出，老師與學習者皆須具備更好的工具以詮釋與評估多模態文本之使用。通過在語言教室裡加入修訂過後的文本圖像關係架構，學生能夠探索如何呈現，解釋和評估文本及圖像的互動，以及了解如何有意義的表達個人思想。

The increasing use of multimodality in L1 and L2 literacy classrooms has been observed in recent years through various means, such as digital storytelling projects, PowerPoint presentations, videos making, etc. This shift from traditional to multimodal literacy practices is prompted by the ongoing technological advancement of the current era, which reveals the inadequacy of conventional language-based pedagogy to develop the new literacy practices that are happening in this millennium.
With the scarce use of fixed framework to analyze EFL learners’ multimodal compositions that combines traditional and new literacies performances, this study adopts a social semiotic approach and selects Systemic Functional Multimodal Discourse Analysis (SFMDA) to (1) uncover the variety and the frequency of meta-functions students used in their editorial booklets, and (2) to demonstrate that both effective and ineffective communicative intent could be made by EFL learners. This study aims to inform EFL instructors and learners on how to interpret multimodal compositions, and hope to propose a revised framework for this specific type of compositional genre.
Two frameworks (Unsworth, 2006; Wu, 2014) were utilized in this study to support the coding of learners’ communicative intents in their booklets which were later interpreted through readers’ perspectives. The findings reveal that learners utilized diverse meaning-making meta-functions in their compositions to appeal to their target audiences, and learners’ performances can also be examined by three aspects: (1) total meta-functions used in each booklet, (2) the diverse combinations of multiple meta-functions, and (3) the average image used per page. With the nature of the booklets, where text was the primary semiotic mode and image was supplementary, learners focused more on expanding their text with the use of image rather than employing other meaning-making relations. Two variations of ‘redundancy’ were also further deduced in this study.
In addition, due to the lack of training in crafting multimodal compositions, and not being well-informed of the various meaning-making relations between semiotic modes, un-codable instances and common text-image related mistakes were observed in learners’ booklets. This finding also shows that learners’ understanding towards multimodality is still very much under-developed and literacy instructors now have the responsibility to educate and enhance learners’ awareness of and skills/abilities in maneuvering multimodal semiotic resources to achieve meaningful communicative purposes. Simultaneously, both teachers and learners also have to be equipped with a better instrument to help them interpret and evaluate multimodal texts as input (e.g., incoming information) or output (e.g., performance and production). By introducing the revised text-image relation framework in literacy classrooms, learners are able to explore how text-image interactions can be presented, interpreted, and evaluated, as well as crafting meaningful personal expressions.

TABLE OF CONTENTS
摘要    i
ABSTRACT    iii
ACKNOWLEDGEMENTS    v
CHAPTER ONE: INTRODUCTION    1
1 Background and Motivation    1
2 Purpose of the Study and Research Questions    5
3 Definitions of Terms    5
CHAPTER TWO: LITERATURE REVIEW    9
1 Systemic Functional Multimodal Discourse Analysis (SFMDA)    9
2 Ideational Meaning at the Intersections of Language and Image    11
2.1 Unsworth’s Framework (2006)    12
2.2 Wu’s Framework (2014)    19
3 Text and Image Relations    23
4 The Practice of Multimodality in L1 English Language Arts Classrooms    26
5 The Practice of Multimodality in English as a Second Language (ESL) Classroom    29
6 The Practice of Multiliteracies in Classrooms    32
CHAPTER THREE: METHODOLOGY    35
1 Context and Participants    35
2 Instruments    36
2.1 Editorial Booklets    36
2.1.1 Group 1: Deep into the Eddies    36
2.1.2 Group 2: Dadaocheng - Where Preservation Meets Innovation    38
2.1.3 Group 3: Something you Don’t Know about your Second Hometown    39
2.1.4 Group 4: Unique Gods in Bangka    40
2.2 Analytical Frameworks    42
3 Data Collection Procedure    45
3.1 The Curriculum    45
4 Data Analysis Procedure    46
4.1 Pre-Analysis    47
4.2 Systemic Functional Multimodal Discourse Analysis (SFMDA)    48
CHAPTER FOUR: RESULTS    49
1 RQ1: Which meta-functions are used by learners to achieve communicative goal through text-image compositions?    49
2 RQ2: How does Systemic Functional Multimodal Discourse Analysis (SFMDA) help us assess students’ meaning-making products as a result of the multiliteracies curriculum?    52
2.1 Meta-functions under ‘expansion’    52
2.1.1 Effective and ineffective examples of meta-functions under ‘elaboration’    53
2.1.2 Effective and ineffective examples of meta-functions under ‘complementarity’    62
2.2 Meta-functions under ‘connection’    68
2.2.1 Effective and ineffective examples of meta-functions under ‘projection’    68
2.2.2 Effective and ineffective examples of meta-functions under ‘enhancement’    71
2.3 Un-codable instances    76
2.3.1 Ineffective use of image    76
2.3.2 Lack of relation between text and image    81
2.3.3 Invalid and ineffective use of image    83
2.3.4 Problematic layout    84
2.4 Combinations of multiple meta-functions    85
2.4.1 Combination of four meta-functions    85
2.4.2 Combinations of three meta-functions    87
2.4.3 Combinations of two meta-functions    90
3 Conclusion    96
CHAPTER FIVE: DISCUSSION AND CONCLUSION    97
1 RQ1: Which meta-functions are used by learners to achieve communicative goal through text-image compositions?    97
1.1 Patterns of meta-functions used in respective analytical layers    98
1.2 Learners’ performances    99
1.3 ‘Expansion’ over ‘connection’    101
2 RQ2: How does Systemic Functional Multimodal Discourse Analysis (SFMDA) help us to assess students’ meaning-making processes and outcome as a result of the multiliteracies curriculum?    102
2.1 The principle of ‘redundancy’    102
2.2 The simultaneous occurrence of two variations of meta-functions    104
2.3 The ambiguity between ‘exposition,’ ‘exemplification’ and ‘augmentation’    104
2.4 The occurrence of un-codable instances    106
2.5 Common mistakes seen in EFL learners’ multimodal ensemble    107
2.5.1 The ineffective use of image    107
2.5.2 Lack of relation between text and image    108
2.5.3 Invalid use of text and/or image    108
2.5.4 Problematic layout    109
3 Concluding remarks    111
3.1 The newly revised framework    111
3.2 Pedagogical implications    112
3.3 Limitations and suggestions for future studies    114
REFERENCES    115
                                

Abdel-Hack, E. M., & Helwa, H. S. A. (2014). Using Digital Storytelling and Weblogs Instruction to Enhance EFL Narrative Writing and Critical Thinking Skills Among EFL Majors at Faculty of Education. International Research Journals, 5(1).
Arola, K. L., Ball, C. E., & Sheppard, J. (2014). Writer/designer: A guide to making multi- modal projects. New York, NY: Macmillan Higher Education.
Bateman, J. A. (2014). Text and Image: A critical introduction to the visual/verbal divide. New York: Routledge.
Bateman, J. A., & Schmidt, K. (2012). Multimodal Film Analysis: How Films Mean. New York: Routledge.
Benveniste, E. (1986). “The Semiology of Language, translated by Genette Ashby & Adelaide Russo”. In Semiotics: An Introductory Anthology, Edited by: Innis, Robert. 228–245. London: Hutchinson.
Black, R. W. (2007). Digital design: English language learners and reader feedback in online fanfiction. In Michele Knobel, & Colin Lankshear (Eds.), A new literacies sampler (pp. 115–136). New York: Peter Lang.
Black, R. W. (2008). Adolescents and online fan fiction. New York: Peter Lang.
Bloch, J. (2007). Abdullah’s blogging: A generation 1.5 student enters the blogosphere. Language Learning & Technology, 11(2), 128–141.
Browne, A. (1983). Gorilla. London: Julia MacRae.
Browne, A. (2001) My Dad. UK: Random House.
Burningham, J. (1977). Come away from the water, Shirley. London: Cape.
Burningham, J. (1978). Time to get out of the bath, Shirley. London: Cape.
Canagarajah, A. S. (2006). TESOL at forty: What are the issues? TESOL Quarterly, 40, 9-34. doi:10.2307/40264509
Dresang, E. (1999). Radical change: Books for youth in a digital age. New York: Wilson.
Ducate, L. C., & Lomicka, L. L. (2008). Adventures in the blogosphere: From blog readers to blog writers. Computer Assisted Language Learning, 21(1), 9–28.
Exley, B., & Mills, A. K. (2012). Parsing the Australian curriculum English: grammar, multimodality and cross-cultural texts. Australian Journal of Language and Literacy, 35(2), 192-205.
Feng, D., & O’Halloran, K. L. (2012). Representing emotive meaning in visual images: A social semiotic approach. Journal of Pragmatics, 44, 2067-2084.
Fernández-Fontecha, A., O’Halloran, K. L., Tan, S., & Wignell, P. (2018). A multimodal approach to visual thinking: The scientific sketch note. Visual Communication, 1-25.
Gee, J. P. (2000). Teenagers in new times: A new literacy studies perspective. Journal of Adolescent & Adult Literacy, 43(5), 412-420.
Gill, T. (2002). Visual and verbal playmates: An exploration of visual and verbal modalities in children’s picture books. Unpublished B.A. (Honours), University of Sydney.
Gilman, P. (1992). Something From Nothing. Markham, Ontario: Scholastic.
Grapin, S. E., & Llosa, L. (2020). Toward an integrative framework for understanding multimodal L2 writing in the content areas. Journal of Second Language Writing, 47, 100711.
Hafner, C. A. (2014). Embedding digital literacies in English language teaching: Students’ digital video projects as multimodal ensembles. TESOL Quarterly, 48(4).
Hafner, C. A., & Miller, L. (2011). Fostering learner autonomy in English for Science: a collaborative digital video project in a technological learning environment. Language Learning & Technology, 15(3), 68–86.
Halliday, M. A. K. (1994). An Introduction to functional grammar (2nd ed.). London: Edward Arnold.
Halliday, M. A. K., & Matthiessen, C. M. I. M. (2004). An introduction to functional grammar (3rd ed.). London: Arnold.
Huang, S. Y. (2015). The intersection of multimodality and critical perspective: Multimodality as subversion. Language Learning & Technology, 19(2), 21–37.
Huang, S. Y. (2019). EFL learners’ critical multimodal reflections on the politics of English. TESOL Journal, 1-17.
Hughes, J. M., & Morrison, L. (2014). The impact of social networking and a multiliteracies pedagogy on English language learners' writer identities. Writing & Pedagogy, 6(3), 607–631. http://doi.org/10.1558/wap.v6i3.607
Hull, G. A., & Katz, M. (2006). Crafting an Agentive Self: Case Studies of Digital Storytelling. Research in the Teaching of English, 41(1).
Jewitt, C. (2008). Multimodality and Literacy in School Classrooms. Review of Research in Education, 32(1), 241–267. doi:10.3102/0091732x07310586
Jewitt, C. (2014). The Routledge Handbook of Multimodal Analysis (2nd ed.) London: Routledge.
Jewitt, C., Bezemer, J., & O’Halloran, K. (2016). Introducing Multimodality. London and New York: Routledge.
Jiang, L., & Luk, J. (2016). Multimodal composing as a learning activity in English classrooms: Inquiring into the sources of its motivational capacity. System, 59, 1-11.
Kajder, S., & Swenson, J. A. (2004). Digital Images in the Language Arts Classroom. Learning & Leading with Technology, 31(8).
Kamil, M. L., Intrator, S. M., & Kim, H. S. (2000). The effects of other technologies on literacy and literacy learning. In Kamil, M. L., Mosenthal, P. B., Pearson, P. D., Barr, R. (Eds.), Handbook of reading research (Vol. 3, pp. 771–788). Mahwah, NJ: Erlbaum.
Kress, G. (2000). Multimodality: Challenges to Thinking About Language. TESOL Quarterly, 34(2), 337-340. https://doi.org/10.2307/3587959
Kress, G. (2003). Literacy in the new media age. London: Routledge.
Kress, G. (2010). Multimodality: A Social Semiotic Approach to Contemporary Communication. London: Routledge.
Kress, G., & Jewitt, C. (2003). Introduction. In C. Jewitt & G. Kress (Eds.), Multi-modal literacy (pp. 1–18). New York: Peter Lang.
Kress, G., & Van Leeuwen, T. (1996). Reading images: The grammar of visual design. London, England: Routledge.
Kress, G., & Van Leeuwen, T. (2001). Multimodal discourse: The modes and media of contemporary communication. London: Arnold.
Kress, G., & Van Leeuwen, T. (2006). Reading images: The grammar of visual design (2nd ed.). London, England: Routledge.
Lee, S. Y., Lo, Y. H. G., & Chin, T. C. (2019). Practicing multiliteracies to enhance EFL learners’ meaning making process and language development: a multimodal problem-based approach. Computer Assisted Language Learning. doi:10.1080/09588221.2019.1614959
Lemke, J. L. (1998). Multiplying meaning: Visual and verbal semiotics in scientific text. In J. R. Martin and R. Veel (Eds.), Reading science: critical and functional perspectives on discourses of science (pp. 87-113). London: Routledge.
Lim, V. F. (2004). Developing an integrative multi-semiotic model. In K. O'Halloran (Ed.), Multimodal discourse analysis: Systemic functional perspectives (pp. 220-246). London and New York: Continuum.
Lim, V. F., O’Halloran, K. L., Tan, S., & E, M. K. L. (2015) Teaching visual texts with the multimodal analysis software. Education Tech Research Dev, 63, 915–935. https://doi.org/10.1007/s11423-015-9395-4
Machin, D. (2016). The need for a social and affordance-driven multimodal critical discourse studies. Discourse & Society, 27(3), 322–334. doi: 10.1177/0957926516630903
MacLeod, J. (2017). A Paris year: My day-to-day adventures in the most Romantic City in the World. New York, NY: St. Martin’s Press.
Marsh, E. E. & White, M. D. (2003). A Taxonomy of Relationships between Images and Text. Journal of Documentation, 59(6), 647-672.
Martinec, R., & Salway, A. (2005). A system for image - text relations in new (and old) media. Visual Communication, 4, 337-371. doi:10.1177/1470357205055928
McCarthy, J. P., & Anderson, L. (2000). Active learning techniques versus traditional teaching styles: Two experiments from history and political science. Innovative Higher Education, 24(4), 279–294. doi:10.1023/B:IHIE.0000047415.48495.05
McCloud, S. (1994). Understanding comics: The invisible art. New York: Harper Collins.
McGinnis, T., Goodstein-Stolzenberg, A., & Saliani, E. C. (2007). “indnpride”: Online spaces of transnational youth as sites of creative and sophisticated literacy and identity work. Linguistics and Education, 18, 283–304.
Nelson, M. E. (2006). Mode, meaning, and synaesthesia in multimedia L2 writing. Language Learning and Technology, 10(2), 56–76.
Nelson, M. E. (2012). Multimodality and literacy. In the encyclopedia of applied linguistics. Blackwell Publishing Ltd.
Nelson, M. E., & Hull, G. A. (2009). Self-presentation through multimedia: A Bakhtinian perspective on digital storytelling. In Knut Lundby (Ed.), Digital storytelling, mediatized stories: Self-representation in new media (pp. 123–142). New York: Peter Lang.
New London Group. (1996). A pedagogy of multiliteracies: Designing social futures. Harvard Educational Review, 66(1), 60-92.
New London Group. (2000). Multiliteracies: Literacy Learning and the Design of Social Futures. London: Routledge.
O'Halloran, K. L. (2000). Classroom discourse in Mathematics: A multisemiotic analysis. Linguistics and Education, 10(3), 359-388.
O'Toole, M. (2011). The language of displayed art. Abington, Oxon: Routledge.
O’Halloran, K. L., & Lim, V. F. (2011). Dimensioner af multimodal literacy. Viden om Læsning, 10, 14-21.
Pai, H. Y. (2000). Taipei people. Hong Kong: Chinese University Press.
Prior, P., & Smith, A. (2020). Editorial: Writing across: Tracing transliteracies as becoming across time, space, and settings. Learning, Culture and Social Interaction, 24, 100246.
Royce, T. (2002). Multimodality in the TESOL Classroom: Exploring Visual‐Verbal Synergy. TESOL Quarterly, 36(2), 191-205.
Ryan, S. Y. (2016). Green Island. New York: Penguin Random House Inc.
Schriver, K. (1997). Dynamics in document design. New York: Wiley.
Schultz, J. (2006). How can we develop SOSE literacy? A practical approach. Social Educator, 24(2), 8-12.
Selfe, R. J., & Selfe, C. L. (2008). “Convince me!” valuing multimodal literacies and composing public service announcements. Theory Into Practice, 47(2), 83-92.
Sendak, M. (1962). Where the wild things are. London: The Bodley Head.
Shin, D. S., & Cimasko, T. (2008). Multimodal composition in a college ESL class: New tools, traditional norms. Computers and Composition, 25(4), 376-395.
Shin, D.S., Cimasko, T., & Yi, Y. (2020). Development of metalanguage for multimodal composing: A case study of an L2 writer’s design of multimedia texts. Journal of Second Language Writing, 47, 100714.
Tardy, C. M. (2005). Expressions of disciplinarity and individuality in a multimodal genre. Computers and Composition, 22, 319–336.
Twiner, A, Coffin, C, Littleton, K, & Whitelock, D. (2010). Multimodality, orchestration and participation in the context of classroom use of the interactive whiteboard: A discussion. Technology, Pedagogy and Education, 19(2), 211–223.
Unsworth, L. (2001). Teaching multiliteracies across the curriculum: Changing contexts of text and image in classroom practice. Buckingham: Open University Press.
Unsworth, L. (2002). Changing dimensions of school literacies. Australian Journal of Language and Literacy, 25(1), 62-79.
Unsworth, L. (2004). Comparing school science explanations in books and computer-based formats: The role of images, image/text relations and hyperlinks. International Journal of Instructional Media, 31(3), 283-301.
Unsworth, L. (2006). Towards a metalanguage for multiliteracies education: Describing the meaning-making resources of language-image interaction. English Teaching: Practice and Critique, 5(1), 55-76.
Unsworth, L. (2008). Multiliteracies and meta-language: Describing image/text relations as a resource for negotiating multimodal texts. In J. Coiro, M. Knobel, C. Lankshear, & D.J. Leu (Eds.), Handbook of research on new literacies (pp. 377–405). Mahwah, NJ: Erlbaum.
Unsworth, L., Thomas, A., & Bush, R. (2004). The role of images and image–text relations in group ‘basic skills tests’ of literacy for children in the primary years. Australian Journal of Language and Literacy.
Vasudevan, L., Schultz, K., & Bateman, J. (2010). Rethinking Composing in a Digital Age: Authoring Literate Identities Through Multimodal Storytelling. Written Communication, 27(4), 442–468.
Wignell, P., O’Halloran, K. L., & Tan, S. (2018). Semiotic space invasion: The case of Donald Trump’s US presidential campaign. Semiotica, 226, 185-208. doi: https://doi.org/10.1515/sem-2017-0109.
Wignell, P., Tan, S., & O’Halloran, K. L. (2017a). Under the shade of AK47s: a multimodal approach to violent extremist recruitment strategies for foreign fighters. Critical Studies on Terrorism, 10(3), 429-452. doi:10.1080/17539153.2017.1319319
Wignell, P., Tan, S., O’Halloran, K. L., & Lange, R. (2017b). A mixed methods empirical examination of changes in emphasis and style in the extremist magazines Dabiq and Rumiyah. Perspectives on Terrorism, 11(2).
Wu, S. X. (2014). A multimodal analysis of image-text relations in picture books. Theory and Practice in Language Studies, 4(7), 1415-1420.
Yang, Y. F. (2012). Multimodal composing in digital storytelling. Computers and Composition, 29, 221–238.
Yasin, M. S. M., Hamid, B. A., Othman, Z., Bakar, K. A., Hashim, F., & Azmah Mohti. A. (2012). A visual analysis of a Malaysian English school textbook: Gender matters. Procedia - Social and Behavioral Sciences, 69, 1871–1880.
Yeh, H. C. (2018). Exploring the perceived benefits of the process of multimodal video making in developing multiliteracies. Language Learning & Technology, 22(2), 28-37. https://doi.org/10125/44642
Zhang, Y., & O’Halloran, K. L. (2019). Empowering the point: Pains and gains of a writer’s traversals between print-based writing and multimodal composing. Linguistics and Education, 51, 1–11.

全文公開日期 2024/02/01 (校內網路)
全文公開日期本全文未授權公開 (校外網路)
全文公開日期 2024/02/01 (國家圖書館：臺灣博碩士論文系統)

簡易檢索 / 詳目顯示

相關論文