Introduction

The role of translation in a crisis has gained attention among TS scholars (Federici et al., 2019; O’Brien and Federici, 2019). Previous studies had concerned MT in assisting health communication Dew et al., 2018). Dreisbach and Dreisbach (2021) illustrated the doctor-patient communication; they recognize that communication failures are due to linguistic barriers for the patients are unable to convey their symptoms in a foreign or second language. Li et al. (2020) underscored a need to go beyond English-mediated knowledge production to contend with prompt Covid-19 publicity in non-English-speaking countries and areas. Therefore, it is urgent and necessary to control the spread of covid-19 cases in international and multilingual communities by employing MT tools.

In order to meet the needs of the international community to combat COVID-19, linguists, translators and translation companies continue to develop the corpus of the corona crisis at the monolingual, bilingual and multilingual levels. For example, together with Systran, Taus has contributed to producing Corona Crisis translation models in twelve languages including English, French, Spanish, German, Italian, Chinese, and Russian (TAUS and Systran).[1] The EMEA corpus (comprising extensive documents from the European Medicines Agency) focuses on more comprehensive health (EMEA).[2] The sketch engine corpus consists of texts published as part of the covid-19 open research dataset (Sketch Engine).[3] Coronavirus Corpus from English-Corpora. The coronavirus corpus is developed based on records of the coronavirus’s social, cultural, and economic impact in 2020 and beyond (English-Corpora).[4] The public can access machine translation tools to consult information. However, such corpus is relatively less known to the public. The Corpus from Systran, can be used in the same way as Google Translate or Bing. Systran, Google Translate, and Bing from Microsoft are parallel corpora offering multilingual translations. Therefore, they can be used widely across the globe in a variety of languages. Their interface is familiar to users and easy to handle (Bowker, Ciro, 2019).

Despite the development of MT, specifically during the pandemic, there are some thorny problems in the use of MT tools (Koehn and Knowles, 2017). Several studies found that MT still failed to distinguish from synonyms, collocations, or word meanings. MT error occur frequently, such as word missing, wrong word order, incorrect punctuation, and unknown words (Chon, Shin, and Kim 2021; Stankevičiūtė, Kasperė, and Horbačauskienė, 2017). Furthermore, MT is also considered as ignoring the complexity of language, and calls for cross-disciplinary research (Vieira, O’Hagan, and O’Sullivan, 2020). The use of MT tools has also been alerted in terms of accuracy in translating safety guidelines, which may pose a potential threat to child health (Das et al., 2018). Therefore, it is significant to evaluate the product of MT tools (Asscher, Glikson, 2021; Guzmán et al., 2017).

Main part

Purpose of the Paper

This study attempts to evaluate the accuracy and adequacy of English-Indonesian translations of COVID-19 terms by using MT tools, namely, Systran, Google Translate and Bing. The evaluation can be carried out in two folds: manual evaluation and automatic evaluation (Koehn, 2010: 217; Maurya et al., 2020). Manual evaluation is evaluating the translation product which is carried out by human without machine or application support whereas automatic evaluation assesses the translation product by using a machine or an application. This present study is crucial because the result can give a shade of light on practical and theoretical probing in MT studies.

The research questions include:

1. How is the meaning of Covid-19 related terms in the target language at the word level generated by the three machine translation tools?
2. How is the meaning of Covid-19 related terms in the target language at the sentence level generated by the three machine translation tools?
3. How is the meaning of Covid-19 related terms in the target language at the paragraph level generated by the three machine translation tools?

Materials and Methods

This paper used the evaluation translation method (Williams and Chesterman, 2011) to answer one main research question. This study aims at comparing three Machine translations through human evaluation (Koehn, 2010). At the word level, the output of source language generated by that Machine Translation is assessed manually based on correctness standards to answer the research questions. The establishment of criteria for the translation rubric is furthermore carried out to evaluate translation products (Samir, 2020). The word level is described by Non-Native Indonesia (2) if the word is not available in KBBI (The Great Indonesian Dictionary) but it can still be understood under two reasons; the issue of loanword and calque. The word is categorized Good Indonesia (3) if the word is available in KBBI database and the words are still attributed to loanword and calque. The word is categorized as flawless (4) if it is available in KBBI and dismisses the two issues. If the word scores 1, it is then categorized as Not Recognized or disfluent. The following rubric is depicted to figure out the scoring system.

Table 1. Scoring Rubric Quality of the word-level

Fluency and accuracy are further measured. The translations at the sentence and paragraph levels are scored based on the criteria shown in Table 2.  This evaluation involved two raters to avoid sensitization on internal threats (Saldanha and O’Brien, 2014: 31). The raters were effective in both the source language and the target language. They were coded as Rater 1 (R1) and Rater 2 (R2). Both R1 and R2 used the following rubric proposed by Koehn (2010).

Table 2. The scoring rubric of adequacy and fluency (Phillip Koehn, 2010)

Sample and Data Collection

The data were taken purposively from Google Trend for 12 past months dealing with the word query of Covid-19 in Indonesia. Google Trend provides statistics to show a particular trend with a word query on Google. The authors used the keyword "covid-19", then the Google Trend displayed the words query that most people used in the last 12 months. It was done on 9 July 2021. Three sentences and one paragraph taken from texts on Covid-19 were used as samples. In addition, the samples to evaluate the sentence and paragraph levels were taken from English-Corpora, specifically The Coronavirus Corpus. The samples were taken in different ways and depended on the context the authors chose. The authors selected samples based on words that appeared in previous samples at the word level. Three words from 20 words in Google Trend were used as a lemma and then there were searches for the lemma to get the context of the lemma in The Coronavirus Corpus.

Data Analysis

Data were analyzed by comparing the source text and target text in the info metric. The data were then described based on the accuracy and fluency in the target language and the availability of word equivalency. The score was recorded into the rubric. After the scores were recorded, the data was presented in percentage. Based on the scores, the accuracy and the fluency of the target language were then illustrated concisely.

Furthermore, to provide the credibility of the finding, specifically in examining the sentence and paragraph levels, two raters were asked to evaluate the text. If the scores were different, the raters were to share their analysis and discuss it. If the two raters still disagreed and did not reach a consensus, the authors decided to take the lowest score.

Finding and Discussion

Based on the search from Google Trend, the result shows a considerable number of words related to Covid-19. The authors limited the number of words to 10 top words with the keywords “Covid-19” and “coronavirus diseases” cut of data as of 7/9/20-7/9/21 located in Indonesia. Therefore, there are 20 words retrieved from Google Trends as data sampling. However, the authors also used words and phrases based on the Google Trend retrieval, as shown in
Table 3.

Table 3. Comparison of words translation

After importing the data in the form of an excel worksheet, the words were then translated automatically using three machine translation tools, as shown in Table 3. The meaning of each word was not significantly different from one MT with another MT; most MT generated similar syntax in the target language. Somehow several words were rendered differently in terms of a word class. For example, ‘patient’ in English was transformed into Indonesian as ‘pasien’ by Google Translate and Systran. The word category denotes a noun category. On the other hand, Bing translated the word ‘Patient’ as ‘Sabar’, in which the word category attributes to adjectives in Indonesian. However, Bing also translates the word Pasien alternatively as Google Translate and Systran do. That transformation was not considered an error since pasien and sabar are both correct in Indonesian.  In this case, Bing emphasizes the word patient more adjectival than nominal.

Another different result produced by MT was the word order in a phrase. Skin harsh, for instance, was translated uniquely by Google Translate and Bing as kulit kasar and kulit keras respectively. The difference is that the word harsh was translated into kasar and keras. The meanings of these two words in Indonesian are different although Keras and Kasar have similarities in terms of their characteristics. Both keras and kasar denote something rough if they are correlated to the word kulit (skin). Systran, otherwise, translated it as keras kulit, in which the head of the word is different from the other two. Systran produces keras as the headword which leads to a different meaning. Therefore, the word does not meet the Indonesian “grammalities."

Furthermore, Systran translated the word test in a different form, but the meaning was the same. Google Translate converted the word lockdown differently from the other two. It was transformed to Kuncitara in Indonesian, which is still not available in the Indonesian dictionary (KBBI V), while Bing and Systran translated lockdown into Indonesian as penguncian. However, neither kuncitara nor penguncian is a popular term to show the act of limiting movement of society in the Indonesian language. People in Indonesia are more familiar with the word lockdown, in this case the most common ways of solving problems caused by the absolute lexical gap is applying the loan process in translation (Raeisi, Dastjerdi and Raeisi, 2019).

Evaluation of the word level

After translating the words and phrases into Indonesian, the words were further examined to classify the word quality based on the rubric in Table 1. The quality of words was characterized as Flawless, Good, Non-native, and disfluent/incomprehensible.  The following table shows the category of word-level quality.

Based on Table 4., eight words were categorized as flawless (40%) and good (40%), three words were qualified as disfluent (15%), and only one word was classified as non-native (5%), as can be seen in Figure 1 to summarize the data.

The result indicated that the accuracy of Machine Translation in configuring a source text into a target text at the word level was not the issue. Covid-19, for example, was translated in the same way as the source text, since the word arises due to a disaster setting and denotes a pronominal word. All languages around the globe will deserve it equally the same as the source text. However, Covid-19 is still not available in KBBI V (The Great Indonesian Dictionary V).  Other simple phrases and words were still unavailable in KBBI V such as gugus tugas, pendidikan jarak jauh, and kuncitara. Another issue that needs to be discussed is spelling. All machine translation tools translated the phrase “a sense of smell” as indera penciuman. However, the word indera was mistakenly spelled; the word should be spelled indra according to KBBI. The phrase tes diagnosticcepat and agen infeksi virus was uncommon in the Indonesian language. Most Indonesians are familiar with the phrases rapid test rather than test diagnostic cepat and virus carrier rather than agen infeksi virus.

In this case, the authority that maintains the vocabulary database must update incoming new terms. The government also needs to release a policy to support the authority together with a linguist in developing a word-sized database (corpus).

Evaluation of the sentence level

In measuring the sentence level, the data were taken from The Coronavirus corpus available in English-Corpora. The 20 top words were selected as a lemma. They were used to search for the context of the word in the corpus. When the context was found, the authors browsed the original text and then selected one sentence for each lemma. The lemma consists of three; therefore, three sentences (S3) were retrieved from the coronavirus corpus (English-Corpora).[5] Table 5 shows the comparison of the sentence level in Three Machine Translation.

Based on the measurement rating, Google Translates rendered the three English sentences into Indonesian without any mistakes. The sentences were grammatically constructed without losing, adding, and distorting the message from the ST.  The meaning could be retrieved in the fullest acceptable native language. In addition, Google Translate could justify word choice; for example, the word recommended in ST1, Google Translate produces the word merekomendasikan instead of menyarankan. The sentence was constructed in the context of academic writing. Therefore, the word merekomendasikan was more appropriate.

Bing, on the other hand, had consistently translated the two sentences with good fluency. There were some reasons; firstly, the word recommended was translated into menyarankan in ST1. It did not sound academic since the context of the sentence discussed a research situation. Secondly, inappropriate word equivalency in a phrase of ST2 occurred. The phrase “my sense of smell and taste go away” was translated as “indera penciuman dan rasanya hilang.” This Indonesian phrase could not be easily understood because the suffix -nya did not refer to a particular definite pronoun. However, the meaning conveyed in the sentence could still be understood after necessary editing. Furthermore, in ST3, Bing had successfully translated the sentence correctly to deliver meaning without distortion.

Meanwhile, Systran had an inconsistent result of measurement rating. Based on the scoring rubric, data showed that each sentence had different fluency. The ST1, for example, had been categorized as Flawless, ST2 as Non-native sentence, and ST3 categorized as Good. In ST2, Systran had yet unsuccessfully translated the phrase my sense of smell and taste go away. They were converted into saya merasakan bau dan rasa hilang that did not bring an acceptable meaning in Indonesian.

Table 5. Comparison of Machine Translation results *(ST = Source text

Even though readers could still grasp the meaning, they still need effort to understand the texts. Additionally, compared to the two other MTs, Systran still required improvement when translating ST3; the issue was word choice; the word moderate was translated into moderate instead of sedang.  The word sedang was more appropriate to make the flow of the word sound more natural in Indonesian.

Figure 2. Lemma from Coronavirus Corpus

Category of the paragraph level

In assessing the Machine Translation at the paragraph level, the authors took the context of a lemma from the Coronavirus Corpus as shown in Figure 2. Each lemma linked to the web page provided in the corpus to capture the context from the text. The text is taken from the websites Centers for Disease Control and Prevention (English-Corpora).[6]

Table 6. Sample Paragraph

They were translated using Machine Translations to compare the result of its translation. Table 6 shows the fluency of paragraphs in each machine translation.

The finding reveals that the three Machine Translations translated the languages correctly. They perform good fluency that make readers understand the text easily. However, there is an issue with the translation of the subject pronoun "they" in that paragraph. Indonesians are not familiar with using "mereka" to refer to an inanimate subject. The example is shown in the sentences below.