Shared Task at SIGTURK 2026 Workshop

Terminology-Aware Machine Translation for English–Turkish Scientific Texts

Today, English serves as the default language of science, leading to the creation of a vast body of scientific texts with specialized technical terminology in English. Domain experts play a crucial role—and ideally should be the primary decision-makers—in determining how scientific terms are translated into their native languages. In this work, we explore whether models can be developed to follow experts’ translation choices and automatically correct or post-edit translations accordingly.

Subtask 1 – Term Detection

Before correcting the translation of technical terms, we must first be able to detect them. The initial task, therefore, is to accurately identify the boundaries of these terms. Given the paragraphs and the sentences, the model should fill the term_pairs with the detected terms. To support contextual understanding, we also provide the corresponding source and target paragraphs.

Data Format

Example JSON instance:

{
    "paragraph_id": 1,
    "sentence_id": 1,
    "source_paragraph": "Randomness is one of the most important parts of cryptography because key generation and the key itself depend on random values. In literature, there exist statistical randomness tests and test suites to evaluate the randomness of the cryptographic algorithm. Although there exist randomness tests, there is no mathematical evidence to prove that a sequence or a number is random. Therefore, it is vital to choose tests in the test suites due to independency and coverage of the tests used in the suites. Sensitivity of these tests to nonnandom data is also important. The tests should be classified to determine that tests are independent and wide.",
    "target_paragraph": "Rasgelelik kriptografinin en önemli kısımlarından biridir çünkü, anahtar üretimi ve anahtarın kendisi rastgele değerlere bağlıdır. Literatürde birçok istatistiksel rastgelelik testi ve bu testleri içeren test paketleri yer almaktadır. Buna rağmen bir dizinin veya bir sayının rastgele olduğunu gösterecek hiçbir matematiksel kanıt yoktur. Bundan dolayı bir istatistiksel test paketi oluştururken bu testlerin seçimi hayati bir önem taşımaktadır. Ayrıca bu testlerin rastgele olmayan verilere karşı duyarlılığı da çok önemlidir. İstatistiksel testlerin birbirinden bağımsız olduğunu ve kapsamının geniş olduğunu belirlemek için sınıflandırılması gerekmektedir.",
    "source_sentence": "Randomness is one of the most important parts of cryptography because key generation and the key itself depend on random values.",
    "target_sentence": "Rastgelelik kriptografinin en önemli kısımlarından biridir çünkü, anahtar üretimi ve anahtarın kendisi rastgele değerlere bağlıdır.",
    "term_pairs": [
        {
            "en": "Randomness",
            "en_start": 0,
            "en_end": 10,
            "tr": "Rastgelelik",
            "tr_start": 0,
            "tr_end": 11
        },
        {
            "en": "cryptography",
            "en_start": 49,
            "en_end": 61,
            "tr": "kriptografinin",
            "tr_start": 12,
            "tr_end": 26
        },
        {
            "en": "key generation",
            "en_start": 70,
            "en_end": 84,
            "tr": "anahtar üretimi",
            "tr_start": 66,
            "tr_end": 81
        },
        {
            "en": "key",
            "en_start": 70,
            "en_end": 73,
            "tr": "anahtar",
            "tr_start": 66,
            "tr_end": 73
        },
        {
            "en": "random values",
            "en_start": 114,
            "en_end": 127,
            "tr": "rastgele değerlere",
            "tr_start": 103,
            "tr_end": 121
        }
    ]
}

Evaluation

We use Precision, Recall, and micro/macro F1 to evaluate each direction separately. Evaluation is token-based: partial matches (e.g., detecting “key” instead of “key generation”) receive proportional credit. See the provided evaluation script for exact scoring details.

Subtask 2 – Term Correction with Expert Input

In this subtask, the goal is to post-edit the translation of technical terms using expert-provided hints. Given the source and target paragraphs, corresponding sentences, detected term boundaries, and expert input (if available), the model must correct the translation of each term to conform to the expert’s preferred terminology.

The system should fill the "correction" field for each term:

  • The field must contain only the corrected Turkish term (with appropriate suffixes if required).

This task focuses on evaluating the instruction-following capability of pretrained language models. Note that:

  • The provided hints may already represent the correct translation.

  • In some cases, the hint may appear without Turkish suffixes or as a partial translation.

  • Models should use the hint contextually and output a fluent, morphologically correct correction.

Data Format

An example JSON entry with required corrections is shown below:

    {
        "paragraph_id": 1,
        "sentence_id": 3,
        "source_paragraph": "The simplest port scanners use the operating system's network functions and are generally the next option to go to when syn is not a feasible option (described next). Nmap calls this mode connect scan, named after the unix connect() system call. If a port is open, the operating system completes the tcp three-way handshake, and the port scanner immediately closes the connection to avoid performing a denial-of-service attack. Otherwise an error code is returned. This scan mode has the advantage that the user does not require special privileges. However, using the os network functions prevents low-level control, so this scan type is less common.",
        "target_paragraph": "En basit port tarayıcıları işletim sisteminin ağ işlevlerini kullanır ve ve genellikle uygulanabilir bir seçenek olmadığında syn gidilebilecek sonraki seçenektir (sonraki bölümde açıklanmaktadır). Nmap , unix connect () sistem çağrısından sonra adlandırılan bu mod bağlantı taramasını çağırır. Eğer bağlantı açıksa işletim sistemi tcp 3 yollu el sıkışmasını tamamlar ve bağlantı noktası dos saldırısı yapılmasını önlemek amacıyla bağlantıyı hemen kapatır. Aksi halde bir hata kodu döndürülür. Bu tarama modu sayesinde, kullanıcının özel ayrıcalıklara sahip olmasına gerek yoktur. Buna rağmen, işletim sistemi ağ fonksiyonlarını kullanmak düşük seviye kontrolünü önler, bu nedenle bu tarama türü daha az yaygındır.",
        "source_sentence": "If a port is open, the operating system completes the tcp three-way handshake, and the port scanner immediately closes the connection to avoid performing a denial-of-service attack.",
        "target_sentence": "Eğer bağlantı açıksa işletim sistemi tcp 3 yollu el sıkışmasını tamamlar ve bağlantı noktası dos saldırısı yapılmasını önlemek amacıyla bağlantıyı hemen kapatır.",
        "term_pairs": [
            {
                "en": "port",
                "en_start": 5,
                "en_end": 9,
                "tr": "bağlantı",
                "tr_start": 5,
                "tr_end": 13,
                "hint": "bağlantı noktası",
                "correction": "bağlantı noktası"
            },
            {
                "en": "tcp three-way handshake",
                "en_start": 54,
                "en_end": 77,
                "tr": "tcp 3 yollu el sıkışmasını",
                "tr_start": 37,
                "tr_end": 63,
                "hint": "üç yönlü tokalaşma",
                "correction": "tcp 3 yönlü tokalaşmasını"
            },
            {
                "en": "port scanner",
                "en_start": 87,
                "en_end": 99,
                "tr": "bağlantı noktası",
                "tr_start": 76,
                "tr_end": 92,
                "hint": "bağlantı noktaları tarayıcısı",
                "correction": "bağlantı noktaları tarayıcısı"
            }
        ]
    }

Evaluation

Accuracy is measured on the correction field using Exact Match.

Subtask 3 – End-to-End Post-Edit

Here, we are curious how the models would perform end-to-end when given access to terimler.org. The models need not query terimler.org instantly; the offline glossary will be provided in a dictionary format. Here, we will not provide the term boundaries or hints, but only the final translation. Given the source and target paragraphs and sentences; the task is to post-edit the target sentence. The model should fill the edited_target_sentence correctly.

Example

{
  "source_sentence": "We discuss p-branes, plane waves, Kaluza-Klein monopoles, and time-dependent S-brane solutions.",
  "target_sentence": "Dalgaları, p-branları, Kaluza-Klein monopollerini ve zamana bağlı S-brane çözümlerini tartışacağız.",
  "edited_target_sentence": "Düzlem dalgaları, p-zarları, Kaluza-Klein monopollerini ve zamana bağlı S-brane çözümlerini tartışacağız."
}

Evaluation

We evaluate post-edit outputs using chrF and COMET scores (subject to change).

Evaluation Script and Development Data

We provide an evaluation script along with development data for all subtasks on our GitHub repository: https://github.com/aligebesce/sigturk2026_sharedtask

Submission of Model Predictions

TBA

Submission of Papers

TBA

Important Note on Models

Participants may use only pretrained models and resources whose weights are openly available for download at evaluation time. Use of models with closed or restricted weights (e.g., API‐only, gated by manual approval, paywalled checkpoints, or proprietary services) is not permitted. All model architectures are allowed.

Participants must clearly document all models and resources used in their system description papers, including:

  • model name and version/commit,

  • where the weights can be obtained (URL) and the license,

  • any additional resources, training or fine‐tuning data, and prompts.

Important Dates

EventDate (AoE)

Task details and dev data release

October 27, 2025

Test data release / submissions open

November 8, 2025

Submission deadline

December 15, 2025

Evaluation completed

January 2, 2026

System paper deadline

January 2, 2026

Notification of acceptance

January 23, 2026

Camera-ready papers due

February 3, 2026

Organizers

Asst. Prof. Gözde Gül Şahin, Koç University
Ali Gebeşçe, Koç University
Ege Uğur Amasya , Koç University

Contact

For any questions regarding the shared task, please contact: sigturk2026.sharedtask@gmail.com

Acknowledgements

This research is supported by the Wikimedia Foundation Research Fund (Grant No. G‐RS‐2402‐15231). We thank Zafer Batık and Başak Tosun of the Wikimedia Community User Group Turkey for introductions to the Turkish Wikipedia community and assistance with our inquiries regarding the Wikimedia Foundation and community; Kızıl of the Wikipedia Turkey Translators Group for connecting us with translators and demonstrating the translation workflow within Turkish Wikipedia; Prof. Bülent Sankur of terimler.org for insights on technical translations and for facilitating connections with academics who contributed to terminology decisions; and Gizem Ekiz for invaluable help organizing project events and coordinating communication among academics and Wikipedians.