The following chapter discusses conceptualisation of the Discussion Boards as a formative assessment with reference to content and language integration and analyses how pedagogy used in teaching and assessment of the Advanced Business Russian course at Regent’s University London defines assessment methods and criteria. Focusing on the dual task of assessing both the content and the language elements, the chapter evaluates assessment criteria and objectives with reference to the module intended learning outcomes (ILO) in relation to both elements. It further suggests a conceptual framework of reference, based on the principles of J. Biggs and Collis’ SOLO Taxonomy and integrating Coyle’s principles of 4 Cs: content, communication, cognition and culture. This framework explicates the way cognitive skills at different levels can be expressed through language, appropriate for a level of cognition.

Keywords: content and language integration, assessment criteria and objectives, module intended learning outcomes, conceptual frame of reference, level of cognition, cognitive skills

Speaking about the Discussion Boards Assessment, it is important to conceptualise this as a formative assessment ‘as its intention is to be directly diagnostic’.¹ with reference to content and language integration. In other words, it is crucial to define the kind of pedagogy employed both in teaching and assessment and, consequently, what assessment methods and assessment criteria are best applicable to this kind of pedagogy. For this purpose, it is useful to refer to Bernstein’s concepts of classification and framing and visible and invisible pedagogy. As Hoadley explains, ‘Bernstein provides a language for the description of pedagogic discourse through the concepts of classification and framing’.² According to Bernstein, ‘classification’ measures the strength of boundaries between various discourses or contents whereas ‘frame’ or ‘framing’ refers to the style or method of knowledge transmission and acquisition.³ In other words, classification is about the degree of separation between the boundaries of different subjects (power), and ‘framing’ is about ‘the degree of control teacher and pupil possess over the selection, sequencing, pacing and evaluation of the knowledge transmitted and received in the pedagogical relationship’.^2,4 Developing these ideas, Bernstein introduces the concepts of ‘visible’ and ‘invisible’ pedagogy. And although he uses these concepts in relation to the nursery education, they can be applied to any educational level. According to Bernstein, invisible pedagogy ‘is realised through weak classification and weak frames. Visible pedagogies are realised through strong classification and strong frames; the more implicit the manner of transmission and the more diffused the criteria, the more invisible the pedagogy; the more specific the criteria, the more explicit the manner of their transmission, the more visible the pedagogy’.⁴ ‘Visible pedagogy’ fits the first quadrant of the Leung and Morton Matrix on content and language integration.⁵ This catalogues four modes of language integration described by Nikula et al.⁵ in their volume on content and language integration. According to this quadrant of higher disciplinary orientation to language and more visible pedagogy.⁵ linguistic literacy is achieved through content mastering with language firmly imbedded into curriculum. Leung and Morton develop this idea further by saying, that in this quadrant, development of language competence is closely connected with ‘the construction of meaning, associated with specific disciplinary content, skills and thinking processes’.⁵ Lorenzo & Dalton⁶ -Puffer conceptualise language and content integration in this quadrant as ‘language in terms of subject-specific literacy’, in their case, ‘historical literacy’.⁵ They see language competence as a merger with subject – specific competence or as ‘a linguistic switchboard activated by content’.⁵ Radicalising even further the above concept of language and content integration, one can say that content could be understood in terms of language to such a degree that language, in fact, becomes content itself, being at the same time an instrument for content creation. A good example of the above is the famous artificial phrase constructed by the Russian linguist Lev Sherba: Glosse kzdra shteko budlanula bokra and kurdyachit bokrenka.⁷ A loose English translation could be: ‘The glocky kuzdra shteckly budled the bocker and is kurdyaking the bockerling’. All the words in this phrase are obviously meaningless, but all affixes are real, used in a grammatically correct way, and they provide enough semantics to create content which could be ‘a creature of a feminine gender performed rather a strong action on the ‘bocker’ and then continued her act by ‘kurdyaking’ ‘the bockerling’ where ‘bockerling’, judging by its suffix, is the cub of ‘bocker’. Another good example of language becoming content is the quotation from Carroll Lewis who, when asked about the meaning of his poem ‘The Hunting of the Snark’, said: ‘As to the meaning of the Snark, I'm very much afraid I didn't mean anything but nonsense! Still, you know, words mean more than we mean to express when we use them; so, a whole book ought to mean a great deal more than the writer means. So, whatever good meanings are in the book, I'm glad to accept as the meaning of the book’.⁸

Returning to the Advanced Business Russian course delivered at Regent’s University London, one can say that it fits the first quadrant of the Leung and Morton Matrix on content and language integration of higher disciplinary orientation to language and more visible pedagogy.⁵ In that its main intended learning outcome (ILO) would be the acquiring of ‘International Business literacy’ by way of merging language competence with the specific disciplinary literacy. The question of how this model works and how it is assessed will be discussed in Section 1. Nevertheless, it is important to mention that there is a strong emphasis on acquiring the discipline content, which is International Business, in the Advanced Business Russian course ILO while the language is regarded in terms of a ‘subject literacy’.⁵ So, it is not surprising that, out of necessity, the pedagogy of the advanced Business Russian course at Regent’s University London, reflected in its assessment, sometimes crosses the boundaries of the above quadrant entering the quadrant with a less visible language pedagogy and higher disciplinary orientation to language.

As stated earlier, the Discussion Board Assessment is a formative assessment in that ‘its results are used for feedback during learning [because] students and teachers both need to know how learning is proceeding’.⁹ Biggs and Tang have defined the aim of a formative assessment: ‘Assessment practices must have sent the right signals to students about what they should be learning and how they should be learning’.⁹ But in the Advanced Business Russian course assessment there is an additional problem in assessing both language and content. What this means in practice is that both elements, language and content, should be assessed equally according to the following ILO of the Advanced Business Russian course presented here:

At the end of the module, learners will be expected to A13: be able demonstrate critical knowledge and understanding of the contemporary economic, socio-political, and cultural dynamics of the country or countries where the foreign language is spoken, and their relation to wider international developments 

Cognitive skills: At the end of the module, learners will be expected to B13: be able to critically analyse and evaluate models and concepts within international business B15: be able to integrate and evaluate complex information and data from a variety of sources.

Practical and professional skills: At the end of the module, learners will be expected to C14: be able to use the target language(s) competently and effectively as a medium for understanding, expression and communication in professional and social situations. 

Key transferable skills: At the end of the module, learners will be expected to: D 13: be able to show evidence of learner autonomy, continuing professional development and commitment to lifelong learning in a self-reflective manner.

But, despite the clearly defined ILO for the Advanced Business Russian course, assessment in the context of teaching Advanced Business Russian is a complicated matter for many reasons. First, as mentioned earlier, there is a dual focus, namely, language and content, meaning that both elements should be assessed. This situation creates a dilemma for the assessor who needs to decide ‘what to assess’ and ‘how to assess’, in other words, it is necessary to understand clearly to what extent language and content are integrated and, if they are, whether it means that they should be assessed through the same tasks and activities or separately. Then there is a second question: how can an assessor interpret the integrated assessment outcomes? For example, if a learner performs poorly, is it because of her weak understanding of the subject or task or because of her inadequate language skills? Teachers may not be sure whether a student is simply unable to demonstrate knowledge because of a language barrier or whether, indeed, the student does not know the content material being assessed. Yet, a distinction needs to be drawn, especially if a student is not succeeding in a course.¹⁰

A concrete example of the above-mentioned dilemma: An Italian student who was given 48% for her Discussion Boards Assessment complained that her grade was unfair because she was being punished for not knowing Russian well enough. She argued that, as a foreigner, she should not be expected to know the Russian language at native-speaker level. However, the problem was that her level of language attainment did not reach the required level of ‘subject literacy’, so she failed to create content in her work. In the context of teaching Advanced Business Russian, the conceptualization of the Discussion Boards Assessment is primarily based on the assumption that it assesses both the subject learning activities and the target language, with the focus on ‘subject literacies’.⁵ Language, therefore, is used as a means of content construction and communication within a domain of a specific subject – International Business. Mohan and van Naerssen outline the subtle relation between language and content in content and language integrated learning as follows: ‘Discourse does not just express meaning. Discourse creates meaning. As we acquire new areas of knowledge, we acquire new areas of language and meaning.¹

It is clear, therefore, that it is not easy for a teacher to find the right balance between language and content while assessing a student’s work. Nevertheless, conceptualising the Discussion Boards Assessment makes it clear that it should, as with any other assessment, have the explicitly defined criteria and objectives based on the module’s intended learning outcomes (ILO) in relation to both its elements together with a useful assessment tool, a conceptual framework of reference, which would define the ways in which cognitive skills at different levels can be expressed through the language, appropriate for this level of cognition. The framework should also reflect integration of Coyle’s 4Cs, namely, content (subject matter), communication (language learning and using), cognition (learning and thinking process) and culture (developing intercultural understanding and global citizenship).¹

The proposed Framework below, (Table 1) is based on the principles of J. Biggs and Collis’ SOLO Taxonomy which defines students’ level of understanding, i.e. their ability to construct knowledge. Biggs and Tang also speak about two main changes in how students learn: ‘quantitative- as the amount of detail in the student’s response increases; and qualitative, as that detail becomes integrated into a structural pattern.⁸ Biggs and Collis’ SOLO Taxonomy (Figure 1) includes five levels of understanding represented by a typical set of verbs. J. Biggs and Collis’ SOLO Taxonomy.⁸ Table 1 represents the Discussion Boards Assessment Framework based on Biggs and Collis’ SOLO Taxonomy, where knowledge is looked upon in its relation to language at 5 Levels of Biggs and Collis’ Taxonomy: prestructural, unistructural, multistructural, relational and extended abstract. The framework also includes cognitive dimensions according to the 5 levels of understanding that reflect integration of Coyle’s 4Cs – content, communication, cognition and culture.¹ 

Figure 1 Biggs and collis’ SOLO taxonomy.

This leads naturally to a second conceptual postulate of the Discussion Boards Assessment, namely, that it should be built on constructivist learning principles, meaning that ‘students must be active discoverers and constructors of their own knowledge.¹¹ So, it might be argued that the principle of learner-centeredness is a core concept of the Discussion Boards Assessment in the Advanced Business Russian course at Regents University London. The third conceptual postulate of the Advanced Business Russian course assessment concerns its role in the teaching and learning process. Wewer calls assessment an integral part of the curriculum and a crucial component in every teacher’s pedagogy.¹² Coyle et al.¹ define this type of assessment as ‘Assessment for Leaning’ referring to the concepts of assessment developed by Black and William.¹ Underlying the main principle of the Assessment for Learning, Barbero says that Assessment for Learning ‘involves the teacher and the learner in a continual review of the progress achieved’.⁹ Having defined Assessment for Learning, it is easy to see that the Discussion Boards Assessment perfectly fits this description since ‘it has the aim of informing the planning of future learning and teaching’.⁹

As has been established in the previous section, the Discussion Boards Assessment assesses both elements - language and content, according to the ILO of the Advanced Business Russian course. In this way, the Discussion Boards Assessment is what Biggs and Tang refer to as ‘authentic and performance assessment.⁸ Conceptualising this term, Biggs and Tang say that in this type, it is necessary that students demonstrate ‘some sort of ‘performance of understanding’ that reflects the kind of understanding that requires an active demonstration of the knowledge in question’.⁸ In the Discussion Boards Assessment Framework, (Table 1), we demonstrated the correlation between two essential elements of the Discussion Boards Assessment: content and language and 5 levels of understanding (cognition) of J. Biggs ‘and Collis’ SOLO Taxonomy, that reflect integration of Coyle’s 4Cs – content, communication, cognition and culture.¹

Learners of the Advanced Business Russian should be able, according to their ILO, to demonstrate an Extended Abstract Level of understanding of Biggs and Collis’ SOLO taxonomy⁸ which corresponds to the ‘language literacy’ described in Table 1 as the ability ‘to use any structure on a native speaker level and extended, varied and comprehensive vocabulary, including expression of shades of meaning, style and register’. But it hardly needs to be said that not all learners of the Advanced Business Russian can demonstrate this level of understanding; therefore, we should instigate specific guidelines and criteria to assess different levels of learners’ content and language attainment. In other words, we need to answer the question ‘What to assess?’ from a more practical point of view, bearing in mind that the above Framework is a theoretical ‘point’ of reference’ designed to help teachers to better interpret the integrated assessment outcomes. Below is a list of the guidelines/criteria that reflect the course ILO and define the learners’ level of ‘integrated content and language literacy’, explaining what will be assessed by the Discussion Board Assessment.

Please comment, both on the teacher’s two statements about small businesses in Russia and on franchising, and on at least three student entries.

1. Level of understanding criteria: your posts should demonstrate that
1. You have read both the articles that were discussed in class and that you understood them
2. Have not merely repeated the content of the articles but that you have worked with this material, analysing and interpreting it. 

2. Content criteria: your posts should demonstrate
1. Critical thinking
2. Originality
3. Ability to develop an argument
4. Ability to provide evidence to support your conclusions
5. Ability to integrate and evaluate information and data from a variety of sources.

3. Language criteria: your language should:
1. Express the content as it is described in the content guidelines
2. Be discipline-specific, i.e. it should demonstrate knowledge of the vocabulary, specific to the topics discussed
3. Demonstrate coherence and cohesion to convey meaning and construct content
4. Be grammatically correct to be able to construct content.

Also, an answer to the question of ‘How to assess?’ is required. Thus, a tool for evaluating integrated competence in content and language with reference to the above guidelines and criteria is needed. Barbero says that ‘the most appropriate tools to evaluate integrated competences in authentic disciplinary tasks are rubrics’ (Barbero, in Assessment and Evaluation in CLIL, 2012, p.45). Table 2 shows a specially constructed rubric to evaluate content and language in the Discussion Boards Assessment according to the above guidelines and criteria.

According to Coyle et al, performance evidence ‘is a vital component of any evaluation’.¹ In the Discussion Boards Assessment, students’ performance evidence is vital for determining whether this assessment is appropriate and contributes to the students’ learning experience. According to Coyle et al., a traditional evaluation of bilingual education concentrates mainly on language outcomes. However, this will not work for the Advanced Business Russian course assessment because the nature of its content and language integration needs different methods of measuring performance evidence, bearing in mind that ‘language may be a limiting factor’, which can have an impact on the assessment results and thus, on the performance evidence. Indeed, as it has been stated earlier, though students may well understand the content well, they may not possess the necessary language proficiency to construct it. As Coyle et al. point out, ‘understanding a concept is a different capability from expressing that understanding to somebody else. Speaking of the Discussion Boards Assessment based on the first quadrant of Leung and Morton’s Matrix of higher disciplinary orientation to language and more visible language pedagogy with the focus on ‘subject literacies.⁵ it is important to note that it assesses content literacy and language proficiency in equal degree. As explained earlier, the Discussion Boards Assessment is therefore a criterion and performance – based assessment; however, the question of performance evidence is a different matter in that it includes not only evidence of ILO and criteria fulfilment but also other factors such as students’ opinion about the Discussion Boards Assessment, their motivation, interest and dedication. Taking the above into account, two instruments – i.e. two questionnaires were used to investigate the Advanced Business Russian learners’ opinion about the Discussion Boards Assessment, their motivation, interest and dedication. These were offered to the students before they had participated in the Discussion Board Assessment, and after they had completed it. The students’ opinion about the Discussion Boards Assessment was extremely valuable, as often ‘teachers and students have rather different positions and points of view with regard to subject and language learning as well as with regard to teaching and the way it is structured.^13‒15 

Questionnaire 1: Do you think that participation in the Discussion Boards Assessment will enlarge your Business Russian vocabulary, consolidate your grammatical knowledge and strengthen your writing skills, resulting in your stronger ability to express your thoughts (i.e. to construct content) on two topics offered to you - ‘Problems of small businesses in Russia’ and ‘Franchising in Russia’?

Some responses to this question, asked to three randomly chosen students:

First student:

I am not sure whether Discussion Boards can help me with my Business Russian vocabulary, strengthen my grammar and improve my writing because to start writing something useful, you already need to know a lot of vocabulary and grammar. I would prefer simpler exercises.

Second student:

Maybe, but my writing is poor and I don’t know anything about small businesses or franchising in Russia. I don’t understand the Russian grammar either, especially declensions. How can I express my opinion on these topics?

Third student:

Yes, I think so. I would like to try my hand at writing something in Russian but I need to learn some vocabulary. I am interested in small businesses in Russia as I was thinking of starting a small business there or in my native country, Bulgaria. But it is still difficult for me to express my opinion on these subjects because I know about them too little. The only thing that can help me is that there are the same problems with the small businesses in Bulgaria and franchising is a popular business in my country. 

Questionnaire 2: Do you think that participation in the Discussion Boards Assessment has enlarged your Business Russian vocabulary, consolidated your grammatical knowledge and strengthened your writing skills, resulting in your stronger ability to express your thoughts (i.e. to construct content) on the two topics offered to you - ‘Problems of small businesses in Russia’ and ‘Franchising in Russia’?

The same students’ responses:

First student:

To some extent, yes, because we studied both topics before participating in the Discussion Boards. The teacher also helped us to compile a vocabulary list for these two topics. I also liked the way we worked together composing a mind map which included the essential vocabulary and our thoughts about problems of small businesses and why franchising is not a popular business for Russians. I also liked how other students responded to my entries and I liked commenting on theirs. It was great fun!

Second student:

Yes, definitely! The best thing was that I could return to my work and correct it later after I looked up some words in the online dictionary, so I feel that my business vocabulary and my writing skills have improved. It was great working together on a mind map and suggesting different ideas for these topics that we included in our mind map!

Third student:

Yes, it was useful for me as I had never participated in the Discussion Boards Forums. I used an online dictionary which was very convenient and read some articles, unfortunately, not in Russian but in English on the internet connected with this topic. The articles helped me to understand the problems of the Russian small businesses such as corruption and high taxes. When I couldn’t express my thoughts, I asked my Russian friend to help me and it was great. Besides, it was a collective work - we created a mind map where we exchanged opinions and shared information. I liked this work!

The final word:

The questionnaires above demonstrate that before participating in the Discussion Boards Assessment students were apprehensive about its usefulness and their ability to cope with it but we can see from the students’ responses in the second questionnaire how their opinion about the Discussion Boards Assessment has changed and their motivation, interest, dedication, and satisfaction with the results of their work have grown as a result of their participation in the Discussion Boards Assessment. Drawing on the results of the questionnaires, it may be possible to conclude that the students’ personal perspectives of learning have changed during their work on the Discussion Boards Assessment because it

1. Contributed to the collaborative learning
2. Helped students to realise their potential
3. Gave them the incentive to construct their own knowledge
4. Helped them to strengthen their cognitive and writing skills
5. Stimulated their interest in, and their awareness of, developments in Russia.

None.

The author declares that there is no conflicts of interest.

1. Coyle D, Hood Ph, Marsh D. Content and Language Integrated Learning.Cambridge: Cambridge University Press. 2010.
2. Hoadley U. Analysing pedagogy: the problem of framing. Journal of Education. 2006;40(1):1‒24.
3. Bernstein B. Class, Codes and Control, Towards a Theory of Educational Transmission. London, UK: Routledge and Kegan Paul Ltd. 1975.
4. Bernstein B. Class and Pedagogies: Visible and Invisible. ERIC. 1975.
5. Leung C, Morton T. Conclusion: Language competence, learning and pedagogy in CLIL ‒ Deepening and broadening integration. In: Nikula T, et al, Editors. Conceptualising integration in CLIL and multilingual education. Bristol, UK: Multilingual Matters. 2016;235–248.
6. Lorenzo F, Dalton‒Puffer C. Historical Literacy in CLIL: Telling the Past in a Second language. In: Nikula T, et al, Editors. Conceptualising integration in CLIL and multilingual education. Bristol, UK: Multilingual Matters. 2016;55‒72.
7. Shalyapina ZM. Understanding and Generation of Texts from the Standpoint of the Three‒Dimensional Stratificational Linguistic Framework. In: Borisova E, & Souleimanova, Editors. Understanding by Communication. Cambridge, UK: Cambridge Scholars Publishing. 2013;6‒32.
8. Dodgson Collingwood S. The Life and Letters of Lewis Carroll (Rev CL Dodgson). 2008.
9. Biggs J, Tang C. Teaching for Quality Learning at University. What the student Does, 4th Edition. UK: Maidenhead, Berkshire, Open University Press, McGraw‒Hill Education. 2011.
10. Barbero T. Assessment Tools and Practices in CLIL. In: Quartapelle, editor. Assessment and Evaluation in CLIL. 2012;40‒61.
11. Barr RB, Tagg J. From Teaching to Learning – A New Paradigm for Undergraduate Education. Heldref Publications. 1995;1‒12.
12. Wewer T. Assessment of Young Learners’ English Proficiency in Bilingual Content Instruction CLIL. University of Turku. 2014.
13. Wegner A. Seeing the Bigger Picture: What Students and Teachers Think About CLIL. International CLIL Research Journal. 2012;1(4).
14. Kampen van E, Admiraal W, Berry A. Content and language integrated learning in the Netherlands: teachers’ self‒reported pedagogical practices. International Journal of Bilingual Education and Bilingualism. 2018;21(2):222‒236.
15. Kiely R. CLIL – The question of assessment. Devoloping Teachers.