Shared Task at SIGTURK 2027 Workshop
Culture and Understanding across Turkic Languages
The SIGTURK 2027 shared task evaluates cultural and linguistic understanding in five Turkic languages written in three scripts. It has two tracks, Text / LLM and Vision / VLM. Teams may enter either track or both.
| Language | Code |
|---|---|
Turkish |
|
Azerbaijani |
|
Uzbek |
|
Kazakh |
|
Uyghur |
|
Tracks and Tasks
Text / LLM Track
Use one system for all text tasks (T1–T4).
T1 – Cultural Knowledge and References: recognition, attribution, and association of cultural references.
T2 – Idioms and Conventional Expressions: non-literal meaning, contextual interpretation, and appropriate use.
T3 – Turkic Reading Comprehension: explicit and inferential comprehension supported by a passage.
T4 – Cross-Lingual Cultural Understanding: stability of cultural knowledge and pragmatics across prompt languages.
Vision / VLM Track
Use one system for the vision task (V1).
V1 – Visual Cultural Grounding: visual recognition, attribution and association, function, and cultural context.
The same system may serve both tracks, but each track requires a separate submission and is evaluated separately.
Participation Rules
One system per track. Use one system for all text tasks (T1–T4) and one for the vision task (V1).
Training data. Organizers will not provide a training dataset. Synthetic data, self-created data, and data collected from the web may be used.
Model-size bands. Each model-size band has a separate ranking within each track (see Leaderboards and Model-Size Bands).
Compatible models. Any model architecture is allowed if it meets the parameter limits of its band and is compatible with the Hugging Face Transformers library.
Transformers-compatible submission. The model must load and generate outputs locally with the Hugging Face Transformers library. Include model weights, required tokenizer and configuration files, any required processor files, and run instructions.
Local inference. During evaluation, all outputs must be generated by the submitted model running locally through the Hugging Face Transformers library. External APIs, remotely hosted models, and online services may not be used to generate outputs.
Validation and test data. A validation dataset will be shared for model development and hyperparameter selection. Final evaluation uses a private test set that will not be shared with participants; official rankings are based on those results.
Submission platform. Platform details, technical submission requirements, and the submission schedule will be announced separately.
Leaderboards and Model-Size Bands
Each track has its own leaderboard, with a separate ranking for each model-size band. P is the total number of model parameters. For mixture-of-experts (MoE) models, all expert parameters count, including experts that are not active for a given input. For vision-language models, the vision encoder parameters are included in P. Boundary values belong to the lower band.
On the Text leaderboard, the Culture Score combines T1 and T2; T3 and T4 are reported separately.
| Band | Text / LLM | Vision / VLM |
|---|---|---|
Tiny | P ≤ 1B | – |
Small | 1B < P ≤ 8B | P ≤ 8B |
Medium | 8B < P ≤ 30B | 8B < P ≤ 30B |
Large | P > 30B | P > 30B |
Task examples and the leaderboards are at sigturk2027.github.io.
How to Participate
Choose a track or both.
Validate your submission with the validation data and public evaluator after their release on October 22, 2026.
Register and submit. Registration, platform, and submission-format details will be announced before submissions open.
Describe your system. System-paper requirements and the paper schedule will be published with the final participation rules.
Registration is not open yet.
Important Dates
The dates below are the current estimated schedule. Remaining dates will be added when confirmed.
| Event | Date |
|---|---|
Validation data and public evaluator release | October 22, 2026 |
Public leaderboards open | October 27, 2026 |
Model submission deadline | TBA |
Final evaluation results announced | TBA |
System paper submissions open | TBA |
System paper submission deadline | TBA |
Notification of acceptance | TBA |
Camera-ready deadline | TBA |
SIGTURK 2027 Workshop | TBA |
Organizers
Asst. Prof. Çağrı Toraman, Middle East Technical University
Ruşen Birben, Independent Researcher
Evren Ayberk Munis, Politecnico di Torino
Contact
For any questions regarding the shared task, please contact: nlp@ceng.metu.edu.tr
Task specifications, evaluation tools, and release updates are published in the project repository.