Despite the revolution caused by deep NLP models, they remain black boxes, necessitating research to understand their decision-making processes. A recent work by Dalvi et al. (2022) carried out representation analysis through the lens of clustering latent spaces within pre-trained models (PLMs), but that approach is limited to small scale due to the high cost of running Agglomerative hierarchical clustering. This paper studies clustering algorithms in order to scale the discovery of encoded concepts in PLM representations to larger datasets and models. We propose metrics for assessing the quality of discovered latent concepts and use them to compare the studied clustering algorithms. We found that K-Means-based concept discovery significantly enhances efficiency while maintaining the quality of the obtained concepts. Furthermore, we demonstrate the practicality of this newfound efficiency by scaling latent concept discovery to LLMs and phrasal concepts.
@inproceedings{hawasly-etal-2024-scaling,title={Scaling up Discovery of Latent Concepts in Deep {NLP} Models},author={Hawasly, Majd and Dalvi, Fahim and Durrani, Nadir},editor={Graham, Yvette and Purver, Matthew},booktitle={Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},month=mar,year={2024},address={St. Julian{'}s, Malta},publisher={Association for Computational Linguistics},url={https://aclanthology.org/2024.eacl-long.48},pages={793--806},area={Latent Concepts}}
LAraBench: Benchmarking Arabic AI with Large Language Models
Ahmed Abdelali, Hamdy Mubarak, Shammur Chowdhury, and
14 more authors
In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) Mar 2024
Recent advancements in Large Language Models (LLMs) have significantly influenced the landscape of language and speech research. Despite this progress, these models lack specific benchmarking against state-of-the-art (SOTA) models tailored to particular languages and tasks. LAraBench addresses this gap for Arabic Natural Language Processing (NLP) and Speech Processing tasks, including sequence tagging and content classification across different domains. We utilized models such as GPT-3.5-turbo, GPT-4, BLOOMZ, Jais-13b-chat, Whisper, and USM, employing zero and few-shot learning techniques to tackle 33 distinct tasks across 61 publicly available datasets. This involved 98 experimental setups, encompassing ~296K data points, ~46 hours of speech, and 30 sentences for Text-to-Speech (TTS). This effort resulted in 330+ sets of experiments. Our analysis focused on measuring the performance gap between SOTA models and LLMs. The overarching trend observed was that SOTA models generally outperformed LLMs in zero-shot learning, with a few exceptions. Notably, larger computational models with few-shot learning techniques managed to reduce these performance gaps. Our findings provide valuable insights into the applicability of LLMs for Arabic NLP and speech processing tasks.
@inproceedings{abdelali-etal-2024-larabench,title={{LA}ra{B}ench: Benchmarking {A}rabic {AI} with Large Language Models},author={Abdelali, Ahmed and Mubarak, Hamdy and Chowdhury, Shammur and Hasanain, Maram and Mousi, Basel and Boughorbel, Sabri and Abdaljalil, Samir and El Kheir, Yassine and Izham, Daniel and Dalvi, Fahim and Hawasly, Majd and Nazar, Nizi and Elshahawy, Youssef and Ali, Ahmed and Durrani, Nadir and Milic-Frayling, Natasa and Alam, Firoj},editor={Graham, Yvette and Purver, Matthew},booktitle={Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},month=mar,year={2024},address={St. Julian{'}s, Malta},publisher={Association for Computational Linguistics},url={https://aclanthology.org/2024.eacl-long.30},pages={487--520},area={Large Language Models}}
LLMeBench: A Flexible Framework for Accelerating LLMs Benchmarking
Fahim Dalvi, Maram Hasanain, Sabri Boughorbel, and
10 more authors
In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations Mar 2024
The recent development and success of Large Language Models (LLMs) necessitate an evaluation of their performance across diverse NLP tasks in different languages. Although several frameworks have been developed and made publicly available, their customization capabilities for specific tasks and datasets are often complex for different users. In this study, we introduce the LLMeBench framework, which can be seamlessly customized to evaluate LLMs for any NLP task, regardless of language. The framework features generic dataset loaders, several model providers, and pre-implements most standard evaluation metrics. It supports in-context learning with zero- and few-shot settings. A specific dataset and task can be evaluated for a given LLM in less than 20 lines of code while allowing full flexibility to extend the framework for custom datasets, models, or tasks. The framework has been tested on 31 unique NLP tasks using 53 publicly available datasets within 90 experimental setups, involving approximately 296K data points. We open-sourced LLMeBench for the community (https://github.com/qcri/LLMeBench/) and a video demonstrating the framework is available online (https://youtu.be/9cC2m_abk3A).
@inproceedings{dalvi-etal-2024-llmebench,title={{LLM}e{B}ench: A Flexible Framework for Accelerating {LLM}s Benchmarking},author={Dalvi, Fahim and Hasanain, Maram and Boughorbel, Sabri and Mousi, Basel and Abdaljalil, Samir and Nazar, Nizi and Abdelali, Ahmed and Chowdhury, Shammur Absar and Mubarak, Hamdy and Ali, Ahmed and Hawasly, Majd and Durrani, Nadir and Alam, Firoj},editor={Aletras, Nikolaos and De Clercq, Orphee},booktitle={Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations},month=mar,year={2024},address={St. Julians, Malta},publisher={Association for Computational Linguistics},url={https://aclanthology.org/2024.eacl-demo.23},pages={214--222},area={Demos and Tools}}
What do end-to-end speech models learn about speaker, language and channel information? A layer-wise and neuron-level analysis
Shammur Absar Chowdhury, Nadir Durrani, and Ahmed Ali
Deep neural networks are inherently opaque and challenging to interpret. Unlike hand-crafted feature-based models, we struggle to comprehend the concepts learned and how they interact within these models. This understanding is crucial not only for debugging purposes but also for ensuring fairness in ethical decision-making. In our study, we conduct a post-hoc functional interpretability analysis of pretrained speech models using the probing framework (Hupkes et al., 2018). Specifically, we analyze utterance-level representations of speech models trained for various tasks such as speaker recognition and dialect identification. We conduct layer and neuron-wise analyses, probing for speaker, language, and channel properties. Our study aims to answer the following questions: (i) what information is captured within the representations? (ii) how is it represented and distributed? and (iii) can we identify a minimal subset of the network that possesses this information? Our results reveal several novel findings, including: (i) channel and gender information are distributed across the network, (ii) the information is redundantly available in neurons with respect to a task, (iii) complex properties such as dialectal information are encoded only in the task-oriented pretrained network, (iv) and is localized in the upper layers, (v) we can extract a minimal subset of neurons encoding the pre-defined property, (vi) salient neurons are sometimes shared between properties, (vii) our analysis highlights the presence of biases (for example gender) in the network. Our cross-architectural comparison indicates that: (i) the pretrained models capture speaker-invariant information, and (ii) CNN models are competitive with Transformer models in encoding various understudied properties.
@article{chowdhury2024:csl,title={What do end-to-end speech models learn about speaker, language and channel information? A layer-wise and neuron-level analysis},journal={Computer Speech and Language},address={London, UK, UK},volume={83},pages={101539},year={2024},issn={0885-2308},doi={https://doi.org/10.1016/j.csl.2023.101539},url={https://www.sciencedirect.com/science/article/pii/S088523082300058X},author={Chowdhury, Shammur Absar and Durrani, Nadir and Ali, Ahmed},keywords={Speech, Neuron-level analysis, Interpretability, Diagnostic classifier, AI explainability, End-to-end architecture},area={Representation Analysis}}
While a lot of work has been done in understanding representations learned within deep NLP models and what knowledge they capture, work done towards analyzing individual neurons is relatively sparse. We present a technique called Linguistic Correlation Analysis to extract salient neurons in the model, with respect to any extrinsic property, with the goal of understanding how such knowledge is preserved within neurons. We carry out a fine-grained analysis to answer the following questions: (i) can we identify subsets of neurons in the network that learn a specific linguistic property? (ii) is a certain linguistic phenomenon in a given model localized (encoded in few individual neurons) or distributed across many neurons? (iii) how redundantly is the information preserved? (iv) how does fine-tuning pre-trained models towards downstream NLP tasks impact the learned linguistic knowledge? (v) how do models vary in learning different linguistic properties? Our data-driven, quantitative analysis illuminates interesting findings: (i) we found small subsets of neurons that can predict different linguistic tasks; (ii) neurons capturing basic lexical information, such as suffixation, are localized in the lowermost layers; (iii) neurons learning complex concepts, such as syntactic role, are predominantly found in middle and higher layers; (iv) salient linguistic neurons are relocated from higher to lower layers during transfer learning, as the network preserves the higher layers for task-specific information; (v) we found interesting differences across pre-trained models regarding how linguistic information is preserved within them; and (vi) we found that concepts exhibit similar neuron distribution across different languages in the multilingual transformer models. Our code is publicly available as part of the NeuroX toolkit (Dalvi et al., 2023).
@article{JMLR:v24:23-0074,author={Durrani, Nadir and Dalvi, Fahim and Sajjad, Hassan},title={Discovering Salient Neurons in deep NLP models},journal={Journal of Machine Learning Research},year={2023},volume={24},number={362},pages={1--40},area={Neuron Analysis}}
Evaluating Neuron Interpretation Methods of NLP Models
Yimin Fan, Fahin Dalvi, Nadir Durrani, and
1 more author
In Advances in Neural Information Processing Systems Mar 2023
Neuron interpretation offers valuable insights into how knowledge is structured within a deep neural network model. While a number of neuron interpretation methods have been proposed in the literature, the field lacks a comprehensive comparison among these methods. This gap hampers progress due to the absence of standardized metrics and benchmarks. The commonly used evaluation metric has limitations, and creating ground truth annotations for neurons is impractical. Addressing these challenges, we propose an evaluation framework1 based on voting theory. Our hypothesis posits that neurons consistently identified by different methods carry more significant information. We rigorously assess our framework across a diverse array of neuron interpretation methods. Notable findings include: i) despite the theoretical differences among the methods, neuron ranking methods share over 60% of their rankings when identifying salient neurons, ii) the neuron interpretation methods are most sensitive to the last layer representations, iii) Probeless neuron ranking emerges as the most consistent method.
@inproceedings{fan-et-al-2023-neuron-eval,author={Fan, Yimin and Dalvi, Fahin and Durrani, Nadir and Sajjad, Hassan},booktitle={Advances in Neural Information Processing Systems},editor={Koyejo, S. and Mohamed, S. and Agarwal, A. and Belgrave, D. and Cho, K. and Oh, A.},pages={1--13},publisher={Curran Associates, Inc.},title={Evaluating Neuron Interpretation Methods of NLP Models},volume={36},year={2023},area={Neuron Analysis}}
Can LLMs Facilitate Interpretation of Pre-trained Language Models?
Basel Mousi, Nadir Durrani, and Fahim Dalvi
In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing Dec 2023
Work done to uncover the knowledge encoded within pre-trained language models rely on annotated corpora or human-in-the-loop methods. However, these approaches are limited in terms of scalability and the scope of interpretation. We propose using a large language model, ChatGPT, as an annotator to enable fine-grained interpretation analysis of pre-trained language models. We discover latent concepts within pre-trained language models by applying agglomerative hierarchical clustering over contextualized representations and then annotate these concepts using ChatGPT. Our findings demonstrate that ChatGPT produces accurate and semantically richer annotations compared to human-annotated concepts. Additionally, we showcase how GPT-based annotations empower interpretation analysis methodologies of which we demonstrate two: probing frameworks and neuron interpretation. To facilitate further exploration and experimentation in the field, we make available a substantial ConceptNet dataset (TCN) comprising 39,000 annotated concepts.
@inproceedings{mousi-etal-2023-llms,title={Can {LLM}s Facilitate Interpretation of Pre-trained Language Models?},author={Mousi, Basel and Durrani, Nadir and Dalvi, Fahim},editor={Bouamor, Houda and Pino, Juan and Bali, Kalika},booktitle={Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},month=dec,year={2023},address={Singapore},publisher={Association for Computational Linguistics},url={https://aclanthology.org/2023.emnlp-main.196},doi={10.18653/v1/2023.emnlp-main.196},pages={3248--3268},area={Large Language Models}}
On the Effect of Dropping Layers of Pre-trained Transformer Models
Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and
1 more author
Transformer-based NLP models are trained using hundreds of millions or even billions of parameters, limiting their applicability in computationally constrained environments. While the number of parameters generally correlates with performance, it is not clear whether the entire network is required for a downstream task. Motivated by the recent work on pruning and distilling pre-trained models, we explore strategies to drop layers in pre-trained models, and observe the effect of pruning on downstream GLUE tasks. We were able to prune BERT, RoBERTa and XLNet models up to 40%, while maintaining up to 98% of their original performance. Additionally we show that our pruned models are on par with those built using knowledge distillation, both in terms of size and performance. Our experiments yield interesting observations such as: (i) the lower layers are most critical to maintain downstream task performance, (ii) some tasks such as paraphrase detection and sentence similarity are more robust to the dropping of layers, and (iii) models trained using different objective function exhibit different learning patterns and w.r.t the layer dropping.
@article{sajjad2023:csl,address={London, UK, UK},author={Sajjad, Hassan and Dalvi, Fahim and Durrani, Nadir and Nakov, Preslav},issn={0885-2308},doi={https://doi.org/10.1016/j.csl.2022.101429},url={https://www.sciencedirect.com/science/article/pii/S0885230822000596},issue_date={January 2023},journal={Computer Speech and Language},number={C},publisher={Academic Press Ltd.},title={On the Effect of Dropping Layers of Pre-trained Transformer Models},volume={77},pages={101429},year={2023},area={Transfer Learning}}
NeuroX Library for Neuron Analysis of Deep NLP Models
Fahim Dalvi, Nadir Durrani, and Hassan Sajjad
In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics: System Demonstrations Jul 2023
Neuron analysis provides insights into how knowledge is structured in representations and discovers the role of neurons in the network. In addition to developing an understanding of our models, neuron analysis enables various applications such as debiasing, domain adaptation and architectural search. We present NeuroX, a comprehensive open-source toolkit to conduct neuron analysis of natural language processing models. It implements various interpretation methods under a unified API, and provides a framework for data processing and evaluation, thus making it easier for researchers and practitioners to perform neuron analysis. The Python toolkit is available at https://www.github.com/fdalvi/NeuroX.
@inproceedings{dalvi-etal-2023-neurox,title={NeuroX Library for Neuron Analysis of Deep NLP Models},author={Dalvi, Fahim and Durrani, Nadir and Sajjad, Hassan},booktitle={Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics: System Demonstrations},month=jul,year={2023},address={Toronto, Canada},publisher={Association for Computational Linguistics},pages={75--83},area={Demos and Tools}}
NxPlain: A Web-based Tool for Discovery of Latent Concepts
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, and
3 more authors
In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations May 2023
The proliferation of deep neural networks in various domains has seen an increased need for the interpretability of these models, especially in scenarios where fairness and trust are as important as model performance. A lot of independent work is being carried out to: i) analyze what linguistic and non-linguistic knowledge is learned within these models, and ii) highlight the salient parts of the input. We present NxPlain, a web-app that provides an explanation of a model’s prediction using latent concepts. NxPlain discovers latent concepts learned in a deep NLP model, provides an interpretation of the knowledge learned in the model, and explains its predictions based on the used concepts. The application allows users to browse through the latent concepts in an intuitive order, letting them efficiently scan through the most salient concepts with a global corpus-level view and a local sentence-level view. Our tool is useful for debugging, unraveling model bias, and for highlighting spurious correlations in a model. A hosted demo is available here: https://nxplain.qcri.org
@inproceedings{dalvi-etal-2023-nxplain,title={{N}x{P}lain: A Web-based Tool for Discovery of Latent Concepts},author={Dalvi, Fahim and Durrani, Nadir and Sajjad, Hassan and Jaban, Tamim and Husaini, Mus{'}ab and Abbas, Ummar},booktitle={Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations},month=may,year={2023},address={Dubrovnik, Croatia},publisher={Association for Computational Linguistics},pages={75--83},area={Demos and Tools}}
ConceptX: A Framework for Latent Concept Analysis
Firoj Alam, Fahim Dalvi, Nadir Durrani, and
4 more authors
In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence (AAAI, Poster presentation) Feb 2023
The opacity of deep neural networks remains a challenge in deploying solutions where explanation is as important as precision. We present ConceptX, a human-in-the-loop framework for interpreting and annotating latent representational space in pre-trained Language Models (pLMs). We use an unsupervised method to discover concepts learned in these models and enable a graphical interface for humans to generate explanations for the concepts. To facilitate the process, we provide auto-annotations of the concepts (based on traditional linguistic ontologies). Such annotations enable development of a linguistic resource that directly represents latent concepts learned within deep NLP models. These include not just traditional linguistic concepts, but also task-specific or sensitive concepts (words grouped based on gender or religious connotation) that helps the annotators to mark bias in the model. The framework consists of two parts (i) concept discovery and (ii) annotation platform.
@inproceedings{alam:2023:AAAI,title={ConceptX: A Framework for Latent Concept Analysis},author={Alam, Firoj and Dalvi, Fahim and Durrani, Nadir and Sajjad, Hassan and Khan and Rafae, Abdul and Xu, Jia},booktitle={Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence (AAAI, Poster presentation)},year={2023},month=feb,pages={16395-16397},area={Demos and Tools}}
2022
On the Transformation of Latent Space in Fine-Tuned NLP Models
Nadir Durrani, Hassan Sajjad, Fahim Dalvi, and
1 more author
In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP) Dec 2022
We study the evolution of latent space in fine-tuned NLP models. Different from the commonly used probing-framework, we opt for an unsupervised method to analyze representations. More specifically, we discover latent concepts in the representational space using hierarchical clustering. We then use an alignment function to gauge the similarity between the latent space of a pre-trained model and its fine-tuned version. We use traditional linguistic concepts to facilitate our understanding and also study how the model space transforms towards task-specific information. We perform a thorough analysis, comparing pre-trained and fine-tuned models across three models and three downstream tasks. The notable findings of our work are: i) the latent space of the higher layers evolve towards task-specific concepts, ii) whereas the lower layers retain generic concepts acquired in the pre-trained model, iii) we discovered that some concepts in the higher layers acquire polarity towards the output class, and iv) that these concepts can be used for generating adversarial triggers.
@inproceedings{durrani-etal-2022-latent,title={On the Transformation of Latent Space in Fine-Tuned NLP Models},author={Durrani, Nadir and Sajjad, Hassan and Dalvi, Fahim and Alam, Firoj},booktitle={Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)},month=dec,year={2022},address={Abu Dhabi, UAE},publisher={Association for Computational Linguistics},url={https://aclanthology.org/2022.emnlp-main.97},doi={10.18653/v1/2020.emnlp-main.395},pages={1495--1516},area={Transfer Learning}}
Neuron-level Interpretation of Deep NLP Models: A Survey
Hassan Sajjad, Nadir Durrani, and Fahim Dalvi
Transactions of the Association for Computational Linguistics Dec 2022
The proliferation of Deep Neural Networks in various domains has seen an increased need for interpretability of these models. Preliminary work done along this line, and papers that surveyed such, are focused on high-level representation analysis. However, a recent branch of work has concentrated on interpretability at a more granular level of analyzing neurons within these models. In this paper, we survey the work done on neuron analysis including: i) methods to discover and understand neurons in a network; ii) evaluation methods; iii) major findings including cross architectural comparisons that neuron analysis has unraveled; iv) applications of neuron probing such as: controlling the model, domain adaptation, and so forth; and v) a discussion on open issues and future research directions.
@article{sajjad-etal-2022-neuron,title={Neuron-level Interpretation of Deep {NLP} Models: A Survey},author={Sajjad, Hassan and Durrani, Nadir and Dalvi, Fahim},journal={Transactions of the Association for Computational Linguistics},volume={10},year={2022},address={Cambridge, MA},publisher={MIT Press},url={https://aclanthology.org/2022.tacl-1.74},doi={10.1162/tacl_a_00519},pages={1285--1303},area={Neuron Analysis},}
Discovering Latent Concepts Learned in BERT
Fahim Dalvi, Abdul Rafae Khan, Firoj Alam, and
3 more authors
In International Conference on Learning Representations Apr 2022
A large number of studies that analyze deep neural network models and their ability to encode various linguistic and non-linguistic concepts provide an interpretation of the inner mechanics of these models. The scope of the analyses is limited to pre-defined concepts that reinforce the traditional linguistic knowledge and do not reflect on how novel concepts are learned by the model. We address this limitation by discovering and analyzing latent concepts learned in neural network models in an unsupervised fashion and provide interpretations from the model’s perspective. In this work, we study: i) what latent concepts exist in the pre-trained BERT model, ii) how the discovered latent concepts align or diverge from classical linguistic hierarchy and iii) how the latent concepts evolve across layers. Our findings show: i) a model learns novel concepts (e.g. animal categories and demographic groups), which do not strictly adhere to any pre-defined categorization (e.g. POS, semantic tags), ii) several latent concepts are based on multiple properties which may include semantics, syntax, and morphology, iii) the lower layers in the model dominate in learning shallow lexical concepts while the higher layers learn semantic relations and iv) the discovered latent concepts highlight potential biases learned in the model. We also release a novel BERT ConceptNet dataset consisting of 174 concept labels and 1M annotated instances.
@inproceedings{dalvi2022discovering,title={Discovering Latent Concepts Learned in {BERT}},author={Dalvi, Fahim and Khan, Abdul Rafae and Alam, Firoj and Durrani, Nadir and Xu, Jia and Sajjad, Hassan},booktitle={International Conference on Learning Representations},year={2022},month=apr,url={https://openreview.net/forum?id=POTMtpYI1xH},area={Latent Concepts},}
Analyzing Encoded Concepts in Transformer Language Models
Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and
3 more authors
In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Jul 2022
We propose a novel framework ConceptX, to analyze how latent concepts are encoded in representations learned within pre-trained lan-guage models. It uses clustering to discover the encoded concepts and explains them by aligning with a large set of human-defined concepts. Our analysis on seven transformer language models reveal interesting insights: i) the latent space within the learned representations overlap with different linguistic concepts to a varying degree, ii) the lower layers in the model are dominated by lexical concepts (e.g., affixation) and linguistic ontologies (e.g. Word-Net), whereas the core-linguistic concepts (e.g., morphology, syntactic relations) are better represented in the middle and higher layers, iii) some encoded concepts are multi-faceted and cannot be adequately explained using the existing human-defined concepts.
@inproceedings{sajjad-etal-2022-analyzing,author={Sajjad, Hassan and Durrani, Nadir and Dalvi, Fahim and Alam, Firoj and Khan, Abdul and Xu, Jia},booktitle={Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},month=jul,year={2022},address={Seattle, United States},publisher={Association for Computational Linguistics},url={https://aclanthology.org/2022.naacl-main.225},doi={10.18653/v1/2022.naacl-main.225},pages={3082--3101},area={Latent Concepts}}
Effect of Post-processing on Contextualized Word Representations
Hassan Sajjad, Firoj Alam, Fahim Dalvi, and
1 more author
In Proceedings of the 29th International Conference on Computational Linguistics Oct 2022
Post-processing of static embedding has been shown to improve their performance on both lexical and sequence-level tasks. However, post-processing for contextualized embeddings is an under-studied problem. In this work, we question the usefulness of post-processing for contextualized embeddings obtained from different layers of pre-trained language models. More specifically, we standardize individual neuron activations using z-score, min-max normalization, and by removing top principal components using the all-but-the-top method. Additionally, we apply unit length normalization to word representations. On a diverse set of pre-trained models, we show that post-processing unwraps vital information present in the representations for both lexical tasks (such as word similarity and analogy) and sequence classification tasks. Our findings raise interesting points in relation to the research studies that use contextualized representations, and suggest z-score normalization as an essential step to consider when using them in an application.
@inproceedings{sajjad-etal-2022-effect,title={Effect of Post-processing on Contextualized Word Representations},author={Sajjad, Hassan and Alam, Firoj and Dalvi, Fahim and Durrani, Nadir},booktitle={Proceedings of the 29th International Conference on Computational Linguistics},month=oct,year={2022},address={Gyeongju, Republic of Korea},publisher={International Committee on Computational Linguistics},url={https://aclanthology.org/2022.coling-1.277},pages={3127--3142},area={Representation Analysis}}
Post-hoc analysis of Arabic transformer models
Ahmed Abdelali, Nadir Durrani, Fahim Dalvi, and
1 more author
In Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP Dec 2022
Arabic is a Semitic language which is widely spoken with many dialects. Given the success of pre-trained language models, many transformer models trained on Arabic and its dialects have surfaced. While there have been an extrinsic evaluation of these models with respect to downstream NLP tasks, no work has been carried out to analyze and compare their internal representations. We probe how linguistic information is encoded in the transformer models, trained on different Arabic dialects. We perform a layer and neuron analysis on the models using morphological tagging tasks for different dialects of Arabic and a dialectal identification task. Our analysis enlightens interesting findings such as: i) word morphology is learned at the lower and middle layers, ii) while syntactic dependencies are predominantly captured at the higher layers, iii) despite a large overlap in their vocabulary, the MSA-based models fail to capture the nuances of Arabic dialects, iv) we found that neurons in embedding layers are polysemous in nature, while the neurons in middle layers are exclusive to specific properties.
@inproceedings{abdelali-2022-postHoc,title={Post-hoc analysis of Arabic transformer models},author={Abdelali, Ahmed and Durrani, Nadir and Dalvi, Fahim and Sajjad, Hassan},booktitle={Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP},month=dec,year={2022},address={Abu Dhabi, UAE},publisher={Association for Computational Linguistics},doi={10.18653/v1/2021.blackboxnlp-1.1},pages={91--103},area={Neuron Analysis}}
NatiQ: An End-to-end Text-to-Speech System for Arabic
Ahmed Abdelali, Nadir Durrani, Cenk Demiroglu, and
3 more authors
In Proceedings of the Seventh Arabic Natural Language Processing Workshop Dec 2022
NatiQ is end-to-end text-to-speech system for Arabic. Our speech synthesizer uses an encoder-decoder architecture with attention. We used both tacotron-based models (tacotron- 1 and tacotron-2) and the faster transformer model for generating mel-spectrograms from characters. We concatenated Tacotron1 with the WaveRNN vocoder, Tacotron2 with the WaveGlow vocoder and ESPnet transformer with the parallel wavegan vocoder to synthesize waveforms from the spectrograms. We used in-house speech data for two voices: 1) neu- tral male “Hamza”- narrating general content and news, and 2) expressive female “Amina”- narrating children story books to train our models. Our best systems achieve an aver- age Mean Opinion Score (MOS) of 4.21 and 4.40 for Amina and Hamza respectively.The objective evaluation of the systems using word and character error rate (WER and CER) as well as the response time measured by real- time factor favored the end-to-end architecture ESPnet. NatiQ demo is available online at https://tts.qcri.org.
@inproceedings{abdelali-2022-natiq,title={NatiQ: An End-to-end Text-to-Speech System for Arabic},author={Abdelali, Ahmed and Durrani, Nadir and Demiroglu, Cenk and Dalvi, Fahim and Mubarak, Hamdy and Darwish, Kareem},booktitle={Proceedings of the Seventh Arabic Natural Language Processing Workshop},month=dec,year={2022},address={Abu Dhabi, UAE},publisher={Association for Computational Linguistics},url={https://aclanthology.org/2022.wanlp-1.38},pages={394--398},area={Demos and Tools}}
2021
How transfer learning impacts linguistic knowledge in deep NLP models?
Nadir Durrani, Hassan Sajjad, and Fahim Dalvi
In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 Aug 2021
Transfer learning from pre-trained neural language models towards downstream tasks has been a predominant theme in NLP recently. Several researchers have shown that deep NLP models learn non-trivial amount of linguistic knowledge, captured at different layers of the model. We investigate how fine-tuning towards downstream NLP tasks impacts the learned linguistic knowledge. We carry out a study across popular pre-trained models BERT, RoBERTa and XLNet using layer and neuron-level diagnostic classifiers. We found that for some GLUE tasks, the network relies on the core linguistic information and preserve it deeper in the network, while for others it forgets. Linguistic information is distributed in the pre-trained language models but becomes localized to the lower layers post-fine-tuning, reserving higher layers for the task specific knowledge. The pattern varies across architectures, with BERT retaining linguistic information relatively deeper in the network compared to RoBERTa and XLNet, where it is predominantly delegated to the lower layers.
@inproceedings{durrani-etal-2021-transfer,title={How transfer learning impacts linguistic knowledge in deep {NLP} models?},author={Durrani, Nadir and Sajjad, Hassan and Dalvi, Fahim},booktitle={Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021},month=aug,year={2021},address={Online},publisher={Association for Computational Linguistics},url={https://aclanthology.org/2021.findings-acl.438},doi={10.18653/v1/2021.findings-acl.438},pages={4947--4957},area={Transfer Learning}}
Fine-grained Interpretation and Causation Analysis in Deep NLP Models
Hassan Sajjad, Narine Kokhlikyan, Fahim Dalvi, and
1 more author
In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Tutorials Jun 2021
Deep neural networks have constantly pushed the state-of-the-art performance in natural language processing and are considered as the de-facto modeling approach in solving complex NLP tasks such as machine translation, summarization and question-answering. Despite the proven efficacy of deep neural networks at-large, their opaqueness is a major cause of concern. In this tutorial, we will present research work on interpreting fine-grained components of a neural network model from two perspectives, i) fine-grained interpretation, and ii) causation analysis. The former is a class of methods to analyze neurons with respect to a desired language concept or a task. The latter studies the role of neurons and input features in explaining the decisions made by the model. We will also discuss how interpretation methods and causation analysis can connect towards better interpretability of model prediction. Finally, we will walk you through various toolkits that facilitate fine-grained interpretation and causation analysis of neural models.
@inproceedings{sajjad-etal-2021-fine,title={Fine-grained Interpretation and Causation Analysis in Deep {NLP} Models},author={Sajjad, Hassan and Kokhlikyan, Narine and Dalvi, Fahim and Durrani, Nadir},booktitle={Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Tutorials},month=jun,year={2021},address={Online},publisher={Association for Computational Linguistics},url={https://aclanthology.org/2021.naacl-tutorials.2},doi={10.18653/v1/2021.naacl-tutorials.2},pages={5--10},area={Tutorials},}
Fighting the COVID-19 Infodemic: Modeling the Perspective of Journalists, Fact-Checkers, Social Media Platforms, Policy Makers, and the Society
Firoj Alam, Shaden Shaar, Fahim Dalvi, and
14 more authors
In Findings of the Association for Computational Linguistics: EMNLP 2021 Nov 2021
With the emergence of the COVID-19 pandemic, the political and the medical aspects of disinformation merged as the problem got elevated to a whole new level to become the first global infodemic. Fighting this infodemic has been declared one of the most important focus areas of the World Health Organization, with dangers ranging from promoting fake cures, rumors, and conspiracy theories to spreading xenophobia and panic. Addressing the issue requires solving a number of challenging problems such as identifying messages containing claims, determining their check-worthiness and factuality, and their potential to do harm as well as the nature of that harm, to mention just a few. To address this gap, we release a large dataset of 16K manually annotated tweets for fine-grained disinformation analysis that (i) focuses on COVID-19, (ii) combines the perspectives and the interests of journalists, fact-checkers, social media platforms, policy makers, and society, and (iii) covers Arabic, Bulgarian, Dutch, and English. Finally, we show strong evaluation results using pretrained Transformers, thus confirming the practical utility of the dataset in monolingual vs. multilingual, and single task vs. multitask settings.
@inproceedings{alam-etal-2021-fighting-covid,title={Fighting the {COVID}-19 Infodemic: Modeling the Perspective of Journalists, Fact-Checkers, Social Media Platforms, Policy Makers, and the Society},author={Alam, Firoj and Shaar, Shaden and Dalvi, Fahim and Sajjad, Hassan and Nikolov, Alex and Mubarak, Hamdy and Da San Martino, Giovanni and Abdelali, Ahmed and Durrani, Nadir and Darwish, Kareem and Al-Homaid, Abdulaziz and Zaghouani, Wajdi and Caselli, Tommaso and Danoe, Gijs and Stolk, Friso and Bruntink, Britt and Nakov, Preslav},booktitle={Findings of the Association for Computational Linguistics: EMNLP 2021},month=nov,year={2021},address={Punta Cana, Dominican Republic},publisher={Association for Computational Linguistics},url={https://aclanthology.org/2021.findings-emnlp.56},doi={10.18653/v1/2021.findings-emnlp.56},pages={611--649},area={Medical}}
Fighting the COVID-19 Infodemic in Social Media: A Holistic Perspective and a Call to Arms
Firoj Alam, Fahim Dalvi, Shaden Shaar, and
8 more authors
Proceedings of the International AAAI Conference on Web and Social Media May 2021
With the outbreak of the COVID-19 pandemic, people turned to social media to read and to share timely information including statistics, warnings, advice, and inspirational stories. Unfortunately, alongside all this useful information, there was also a new blending of medical and political misinformation and disinformation, which gave rise to the first global infodemic. While fighting this infodemic is typically thought of in terms of factuality, the problem is much broader as malicious content includes not only fake news, rumors, and conspiracy theories, but also promotion of fake cures, panic, racism, xenophobia, and mistrust in the authorities, among others. This is a complex problem that needs a holistic approach combining the perspectives of journalists, fact-checkers, policymakers, government entities, social media platforms, and society as a whole. With this in mind, we define an annotation schema and detailed annotation instructions that reflect these perspectives. We further deploy a multilingual annotation platform, and we issue a call to arms to the research community and beyond to join the fight by supporting our crowdsourcing annotation efforts. We perform initial annotations using the annotation schema, and our initial experiments demonstrated sizable improvements over the baselines.
@article{Alam_covid_infodemic_2021,title={Fighting the COVID-19 Infodemic in Social Media: A Holistic Perspective and a Call to Arms},author={Alam, Firoj and Dalvi, Fahim and Shaar, Shaden and Durrani, Nadir and Mubarak, Hamdy and Nikolov, Alex and Da San Martino, Giovanni and Abdelali, Ahmed and Sajjad, Hassan and Darwish, Kareem and Nakov, Preslav},volume={15},url={https://ojs.aaai.org/index.php/ICWSM/article/view/18114},number={1},journal={Proceedings of the International AAAI Conference on Web and Social Media},year={2021},month=may,pages={913-922},area={Medical}}
2020
On the Linguistic Representational Power of Neural Machine Translation Models
Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, and
2 more authors
Despite the recent success of deep neural networks in natural language processing and other spheres of artificial intelligence, their interpretability remains a challenge. We analyze the representations learned by neural machine translation (NMT) models at various levels of granularity and evaluate their quality through relevant extrinsic properties. In particular, we seek answers to the following questions: (i) How accurately is word structure captured within the learned representations, which is an important aspect in translating morphologically rich languages? (ii) Do the representations capture long-range dependencies, and effectively handle syntactically divergent languages? (iii) Do the representations capture lexical semantics? We conduct a thorough investigation along several parameters: (i) Which layers in the architecture capture each of these linguistic phenomena; (ii) How does the choice of translation unit (word, character, or subword unit) impact the linguistic properties captured by the underlying representations? (iii) Do the encoder and decoder learn differently and independently? (iv) Do the representations learned by multilingual NMT models capture the same amount of linguistic information as their bilingual counterparts? Our data-driven, quantitative evaluation illuminates important aspects in NMT models and their ability to capture various linguistic phenomena. We show that deep NMT models trained in an end-to-end fashion, without being provided any direct supervision during the training process, learn a non-trivial amount of linguistic information. Notable findings include the following observations: (i) Word morphology and part-of-speech information are captured at the lower layers of the model; (ii) In contrast, lexical semantics or non-local syntactic and semantic dependencies are better represented at the higher layers of the model; (iii) Representations learned using characters are more informed about word-morphology compared to those learned using subword units; and (iv) Representations learned by multilingual models are richer compared to bilingual models.
@article{belinkov-etal-2020-linguistic,title={On the Linguistic Representational Power of Neural Machine Translation Models},author={Belinkov, Yonatan and Durrani, Nadir and Dalvi, Fahim and Sajjad, Hassan and Glass, James},journal={Computational Linguistics},volume={46},number={1},year={2020},address={Cambridge, MA},publisher={MIT Press},url={https://aclanthology.org/2020.cl-1.1},doi={10.1162/coli_a_00367},pages={1--52},area={Representation Analysis}}
Analyzing Individual Neurons in Pre-trained Language Models
Nadir Durrani, Hassan Sajjad, Fahim Dalvi, and
1 more author
In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) Nov 2020
While a lot of analysis has been carried to demonstrate linguistic knowledge captured by the representations learned within deep NLP models, very little attention has been paid towards individual neurons.We carry outa neuron-level analysis using core linguistic tasks of predicting morphology, syntax and semantics, on pre-trained language models, with questions like: i) do individual neurons in pre-trained models capture linguistic information? ii) which parts of the network learn more about certain linguistic phenomena? iii) how distributed or focused is the information? and iv) how do various architectures differ in learning these properties? We found small subsets of neurons to predict linguistic tasks, with lower level tasks (such as morphology) localized in fewer neurons, compared to higher level task of predicting syntax. Our study also reveals interesting cross architectural comparisons. For example, we found neurons in XLNet to be more localized and disjoint when predicting properties compared to BERT and others, where they are more distributed and coupled.
@inproceedings{durrani-etal-2020-analyzing,title={Analyzing Individual Neurons in Pre-trained Language Models},author={Durrani, Nadir and Sajjad, Hassan and Dalvi, Fahim and Belinkov, Yonatan},booktitle={Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},month=nov,year={2020},address={Online},publisher={Association for Computational Linguistics},url={https://aclanthology.org/2020.emnlp-main.395},doi={10.18653/v1/2020.emnlp-main.395},pages={4865--4880},area={Neuron Analysis}}
Analyzing Redundancy in Pretrained Transformer Models
Fahim Dalvi, Hassan Sajjad, Nadir Durrani, and
1 more author
In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) Nov 2020
Transformer-based deep NLP models are trained using hundreds of millions of parameters, limiting their applicability in computationally constrained environments. In this paper, we study the cause of these limitations by defining a notion of Redundancy, which we categorize into two classes: General Redundancy and Task-specific Redundancy. We dissect two popular pretrained models, BERT and XLNet, studying how much redundancy they exhibit at a representation-level and at a more fine-grained neuron-level. Our analysis reveals interesting insights, such as i) 85% of the neurons across the network are redundant and ii) at least 92% of them can be removed when optimizing towards a downstream task. Based on our analysis, we present an efficient feature-based transfer learning procedure, which maintains 97% performance while using at-most 10% of the original neurons.
@inproceedings{dalvi-etal-2020-analyzing,title={Analyzing Redundancy in Pretrained Transformer Models},author={Dalvi, Fahim and Sajjad, Hassan and Durrani, Nadir and Belinkov, Yonatan},booktitle={Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},month=nov,year={2020},address={Online},publisher={Association for Computational Linguistics},url={https://aclanthology.org/2020.emnlp-main.398},doi={10.18653/v1/2020.emnlp-main.398},pages={4908--4926},area={Transfer Learning}}
Similarity Analysis of Contextual Word Representation Models
John Wu, Yonatan Belinkov, Hassan Sajjad, and
3 more authors
In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics Jul 2020
This paper investigates contextual word representation models from the lens of similarity analysis. Given a collection of trained models, we measure the similarity of their internal representations and attention. Critically, these models come from vastly different architectures. We use existing and novel similarity measures that aim to gauge the level of localization of information in the deep models, and facilitate the investigation of which design factors affect model similarity, without requiring any external linguistic annotation. The analysis reveals that models within the same family are more similar to one another, as may be expected. Surprisingly, different architectures have rather similar representations, but different individual neurons. We also observed differences in information localization in lower and higher layers and found that higher layers are more affected by fine-tuning on downstream tasks.
@inproceedings{wu-etal-2020-similarity,title={Similarity Analysis of Contextual Word Representation Models},author={Wu, John and Belinkov, Yonatan and Sajjad, Hassan and Durrani, Nadir and Dalvi, Fahim and Glass, James},booktitle={Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},month=jul,year={2020},address={Online},publisher={Association for Computational Linguistics},url={https://aclanthology.org/2020.acl-main.422},doi={10.18653/v1/2020.acl-main.422},pages={4638--4655},area={Representation Analysis}}
Low-resource machine translation suffers from the scarcity of training data and the unavailability of standard evaluation sets. While a number of research efforts target the former, the unavailability of evaluation benchmarks remain a major hindrance in tracking the progress in low-resource machine translation. In this paper, we introduce AraBench, an evaluation suite for dialectal Arabic to English machine translation. Compared to Modern Standard Arabic, Arabic dialects are challenging due to their spoken nature, non-standard orthography, and a large variation in dialectness. To this end, we pool together already available Dialectal Arabic-English resources and additionally build novel test sets. AraBench offers 4 coarse, 15 fine-grained and 25 city-level dialect categories, belonging to diverse genres, such as media, chat, religion and travel with varying level of dialectness. We report strong baselines using several training settings: fine-tuning, back-translation and data augmentation. The evaluation suite opens a wide range of research frontiers to push efforts in low-resource machine translation, particularly Arabic dialect translation. The evaluation suite and the dialectal system are publicly available for research purposes.
@inproceedings{sajjad-etal-2020-arabench,title={{A}ra{B}ench: Benchmarking Dialectal {A}rabic-{E}nglish Machine Translation},author={Sajjad, Hassan and Abdelali, Ahmed and Durrani, Nadir and Dalvi, Fahim},booktitle={Proceedings of the 28th International Conference on Computational Linguistics},month=dec,year={2020},address={Barcelona, Spain (Online)},publisher={International Committee on Computational Linguistics},url={https://aclanthology.org/2020.coling-main.447},doi={10.18653/v1/2020.coling-main.447},pages={5094--5107},area={MT Evaluation},}
FINDINGS OF THE IWSLT 2020 EVALUATION CAMPAIGN
Ebrahim Ansari, Amittai Axelrod, Nguyen Bach, and
20 more authors
In Proceedings of the 17th International Conference on Spoken Language Translation Jul 2020
The evaluation campaign of the International Conference on Spoken Language Translation (IWSLT 2020) featured this year six challenge tracks: (i) Simultaneous speech translation, (ii) Video speech translation, (iii) Offline speech translation, (iv) Conversational speech translation, (v) Open domain translation, and (vi) Non-native speech translation. A total of teams participated in at least one of the tracks. This paper introduces each track’s goal, data and evaluation metrics, and reports the results of the received submissions.
@inproceedings{ansari-etal-2020-findings,title={{FINDINGS} {OF} {THE} {IWSLT} 2020 {EVALUATION} {CAMPAIGN}},author={Ansari, Ebrahim and Axelrod, Amittai and Bach, Nguyen and Bojar, Ond{\v{r}}ej and Cattoni, Roldano and Dalvi, Fahim and Durrani, Nadir and Federico, Marcello and Federmann, Christian and Gu, Jiatao and Huang, Fei and Knight, Kevin and Ma, Xutai and Nagesh, Ajay and Negri, Matteo and Niehues, Jan and Pino, Juan and Salesky, Elizabeth and Shi, Xing and St{\"u}ker, Sebastian and Turchi, Marco and Waibel, Alexander and Wang, Changhan},booktitle={Proceedings of the 17th International Conference on Spoken Language Translation},month=jul,year={2020},address={Online},publisher={Association for Computational Linguistics},url={https://aclanthology.org/2020.iwslt-1.1},doi={10.18653/v1/2020.iwslt-1.1},pages={1--34},area={Findings},}
Findings of the WMT 2020 Shared Task on Machine Translation Robustness
Lucia Specia, Zhenhao Li, Juan Pino, and
9 more authors
In Proceedings of the Fifth Conference on Machine Translation Nov 2020
We report the findings of the second edition of the shared task on improving robustness in Machine Translation (MT). The task aims to test current machine translation systems in their ability to handle challenges facing MT models to be deployed in the real world, including domain diversity and non-standard texts common in user generated content, especially in social media. We cover two language pairs – English-German and English-Japanese and provide test sets in zero-shot and few-shot variants. Participating systems are evaluated both automatically and manually, with an additional human evaluation for ”catastrophic errors”. We received 59 submissions by 11 participating teams from a variety of types of institutions.
@inproceedings{specia-etal-2020-findings,title={Findings of the {WMT} 2020 Shared Task on Machine Translation Robustness},author={Specia, Lucia and Li, Zhenhao and Pino, Juan and Chaudhary, Vishrav and Guzm{\'a}n, Francisco and Neubig, Graham and Durrani, Nadir and Belinkov, Yonatan and Koehn, Philipp and Sajjad, Hassan and Michel, Paul and Li, Xian},booktitle={Proceedings of the Fifth Conference on Machine Translation},month=nov,year={2020},address={Online},publisher={Association for Computational Linguistics},url={https://aclanthology.org/2020.wmt-1.4},pages={76--91},area={Findings},}
2019
What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, and
3 more authors
In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI, Oral presentation) Jan 2019
Despite the remarkable evolution of deep neural networks in natural language processing (NLP), their interpretability remains a challenge. Previous work largely focused on what these models learn at the representation level. We break this analysis down further and study individual dimensions (neurons) in the vector representation learned by end-to-end neural models in NLP tasks. We propose two methods: Linguistic Correlation Analysis, based on a supervised method to extract the most relevant neurons with respect to an extrinsic task, and Cross-model Correlation Analysis, an unsupervised method to extract salient neurons w.r.t. the model itself. We evaluate the effectiveness of our techniques by ablating the identified neurons and reevaluating the network’s performance for two tasks: neural machine translation (NMT) and neural language modeling (NLM). We further present a comprehensive analysis of neurons with the aim to address the following questions: i) how localized or distributed are different linguistic properties in the models? ii) are certain neurons exclusive to some properties and not others? iii) is the information more or less distributed in NMT vs. NLM? and iv) how important are the neurons identified through the linguistic correlation method to the overall task? Our code is publicly available as part of the NeuroX toolkit (Dalvi et al. 2019a). This paper is a non-archived version of the paper published at AAAI (Dalvi et al. 2019b)
@inproceedings{dalvi:2019:AAAI,title={What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models},author={Dalvi, Fahim and Durrani, Nadir and Sajjad, Hassan and Belinkov, Yonatan and Bau, D. Anthony and Glass, James},booktitle={Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI, Oral presentation)},year={2019},month=jan,area={Neuron Analysis},}
One Size Does Not Fit All: Comparing NMT Representations of Different Granularities
Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and
2 more authors
In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) Jun 2019
Recent work has shown that contextualized word representations derived from neural machine translation are a viable alternative to such from simple word predictions tasks. This is because the internal understanding that needs to be built in order to be able to translate from one language to another is much more comprehensive. Unfortunately, computational and memory limitations as of present prevent NMT models from using large word vocabularies, and thus alternatives such as subword units (BPE and morphological segmentations) and characters have been used. Here we study the impact of using different kinds of units on the quality of the resulting representations when used to model morphology, syntax, and semantics. We found that while representations derived from subwords are slightly better for modeling syntax, character-based representations are superior for modeling morphology and are also more robust to noisy input.
@inproceedings{durrani-etal-2019-one,title={One Size Does Not Fit All: Comparing {NMT} Representations of Different Granularities},author={Durrani, Nadir and Dalvi, Fahim and Sajjad, Hassan and Belinkov, Yonatan and Nakov, Preslav},booktitle={Proceedings of the 2019 Conference of the North {A}merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},month=jun,year={2019},address={Minneapolis, Minnesota},publisher={Association for Computational Linguistics},url={https://aclanthology.org/N19-1154},doi={10.18653/v1/N19-1154},pages={1504--1516},area={Representation Analysis}}
Identifying and Controlling Important Neurons in Neural Machine Translation
Anthony Bau, Yonatan Belinkov, Hassan Sajjad, and
3 more authors
In International Conference on Learning Representations Jun 2019
Neural machine translation (NMT) models learn representations containing substantial linguistic information. However, it is not clear if such information is fully distributed or if some of it can be attributed to individual neurons. We develop unsupervised methods for discovering important neurons in NMT models. Our methods rely on the intuition that different models learn similar properties, and do not require any costly external supervision. We show experimentally that translation quality depends on the discovered neurons, and find that many of them capture common linguistic phenomena. Finally, we show how to control NMT translations in predictable ways, by modifying activations of individual neurons.
@inproceedings{bau2018identifying,title={Identifying and Controlling Important Neurons in Neural Machine Translation},author={Bau, Anthony and Belinkov, Yonatan and Sajjad, Hassan and Durrani, Nadir and Dalvi, Fahim and Glass, James},booktitle={International Conference on Learning Representations},year={2019},url={https://openreview.net/forum?id=H1z-PsR5KX},area={Neuron Analysis}}
NeuroX: A Toolkit for Analyzing Individual Neurons in Neural Networks
Fahim Dalvi, Avery Nortonsmith, Anthony Bau, and
4 more authors
Proceedings of the AAAI Conference on Artificial Intelligence Jul 2019
We present a toolkit to facilitate the interpretation and understanding of neural network models. The toolkit provides several methods to identify salient neurons with respect to the model itself or an external task. A user can visualize selected neurons, ablate them to measure their effect on the model accuracy, and manipulate them to control the behavior of the model at the test time. Such an analysis has a potential to serve as a springboard in various research directions, such as understanding the model, better architectural choices, model distillation and controlling data biases. The toolkit is available for download.
@article{dalvi2019neurox,title={NeuroX: A Toolkit for Analyzing Individual Neurons in Neural Networks},author={Dalvi, Fahim and Nortonsmith, Avery and Bau, Anthony and Belinkov, Yonatan and Sajjad, Hassan and Durrani, Nadir and Glass, James},volume={33},url={https://ojs.aaai.org/index.php/AAAI/article/view/5063},doi={10.1609/aaai.v33i01.33019851},number={01},journal={Proceedings of the AAAI Conference on Artificial Intelligence},year={2019},month=jul,pages={9851-9852},area={Demos and Tools}}
Findings of the First Shared Task on Machine Translation Robustness
Xian Li, Paul Michel, Antonios Anastasopoulos, and
7 more authors
In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1) Aug 2019
We share the findings of the first shared task on improving robustness of Machine Translation (MT). The task provides a testbed representing challenges facing MT models deployed in the real world, and facilitates new approaches to improve models’ robustness to noisy input and domain mismatch. We focus on two language pairs (English-French and English-Japanese), and the submitted systems are evaluated on a blind test set consisting of noisy comments on Reddit and professionally sourced translations. As a new task, we received 23 submissions by 11 participating teams from universities, companies, national labs, etc. All submitted systems achieved large improvements over baselines, with the best improvement having +22.33 BLEU. We evaluated submissions by both human judgment and automatic evaluation (BLEU), which shows high correlations (Pearson’s r = 0.94 and 0.95). Furthermore, we conducted a qualitative analysis of the submitted systems using compare-mt, which revealed their salient differences in handling challenges in this task. Such analysis provides additional insights when there is occasional disagreement between human judgment and BLEU, e.g. systems better at producing colloquial expressions received higher score from human judgment.
@inproceedings{li-etal-2019-findings,title={Findings of the First Shared Task on Machine Translation Robustness},author={Li, Xian and Michel, Paul and Anastasopoulos, Antonios and Belinkov, Yonatan and Durrani, Nadir and Firat, Orhan and Koehn, Philipp and Neubig, Graham and Pino, Juan and Sajjad, Hassan},booktitle={Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)},month=aug,year={2019},address={Florence, Italy},publisher={Association for Computational Linguistics},url={https://aclanthology.org/W19-5303},doi={10.18653/v1/W19-5303},pages={91--102},area={Findings},}
2018
Incremental Decoding and Training Methods for Simultaneous Translation in Neural Machine Translation
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, and
1 more author
In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) Jun 2018
We address the problem of simultaneous translation by modifying the Neural MT decoder to operate with dynamically built encoder and attention. We propose a tunable agent which decides the best segmentation strategy for a user-defined BLEU loss and Average Proportion (AP) constraint. Our agent outperforms previously proposed Wait-if-diff and Wait-if-worse agents (Cho and Esipova, 2016) on BLEU with a lower latency. Secondly we proposed data-driven changes to Neural MT training to better match the incremental decoding framework.
@inproceedings{dalvi-etal-2018-incremental,title={Incremental Decoding and Training Methods for Simultaneous Translation in Neural Machine Translation},author={Dalvi, Fahim and Durrani, Nadir and Sajjad, Hassan and Vogel, Stephan},booktitle={Proceedings of the 2018 Conference of the North {A}merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)},month=jun,year={2018},address={New Orleans, Louisiana},publisher={Association for Computational Linguistics},url={https://aclanthology.org/N18-2079},doi={10.18653/v1/N18-2079},pages={493--499},area={Decoding},}
2017
What do Neural Machine Translation Models Learn about Morphology?
Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, and
2 more authors
In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) Jul 2017
Neural machine translation (MT) models obtain state-of-the-art performance while maintaining a simple, end-to-end architecture. However, little is known about what these models learn about source and target languages during the training process. In this work, we analyze the representations learned by neural MT models at various levels of granularity and empirically evaluate the quality of the representations for learning morphology through extrinsic part-of-speech and morphological tagging tasks. We conduct a thorough investigation along several parameters: word-based vs. character-based representations, depth of the encoding layer, the identity of the target language, and encoder vs. decoder representations. Our data-driven, quantitative evaluation sheds light on important aspects in the neural MT system and its ability to capture word structure.
@inproceedings{belinkov-etal-2017-neural,title={What do Neural Machine Translation Models Learn about Morphology?},author={Belinkov, Yonatan and Durrani, Nadir and Dalvi, Fahim and Sajjad, Hassan and Glass, James},booktitle={Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},month=jul,year={2017},address={Vancouver, Canada},publisher={Association for Computational Linguistics},url={https://aclanthology.org/P17-1080},doi={10.18653/v1/P17-1080},pages={861--872},area={Representation Analysis},}
Domain Adaptation using Neural Network Joint Model
Shafiq Joty, Nadir Durrani, Hassan Sajjad, and
1 more author
We explore neural joint models for the task of domain adaptation in machine translation in two ways: (i) we apply state-of-the-art domain adaptation techniques, such as mixture modelling and data selection using the recently proposed Neural Network Joint Model (NNJM) (Devlin et al., 2014); (ii) we propose two novel approaches to perform adaptation through instance weighting and weight readjustment in the NNJM framework. In our first approach, we propose a pair of models called Neural Domain Adaptation Models (NDAM) that minimizes the cross entropy by regularizing the loss function with respect to in-domain (and optionally to out-domain) model. In the second approach, we present a set of Neural Fusion Models (NFM) that combines the in- and the out-domain models by readjusting their parameters based on the in-domain data. We evaluated our models on the standard task of translating English-to-German and Arabic-to-English TED talks. The NDAM models achieved better perplexities and modest BLEU improvements compared to the baseline NNJM, trained either on in-domain or on a concatenation of in- and out-domain data. On the other hand, the NFM models obtained significant improvements of up to +0.9 and +0.7 BLEU points, respectively. We also demonstrate improvements over existing adaptation methods such as instance weighting, phrasetable fill-up, linear and log-linear interpolations.
@article{joty2017:csl,address={London, UK},author={Joty, Shafiq and Durrani, Nadir and Sajjad, Hassan and Abdelali, Ahmed},issn={0885-2308},issue_date={September 2017},journal={Computer Speech and Language},number={C},publisher={Academic Press Ltd.},title={Domain Adaptation using Neural Network Joint Model},volume={45},year={2017},area={Domain Adaptation}}
Challenging Language-Dependent Segmentation for Arabic: An Application to Machine Translation and Part-of-Speech Tagging
Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and
3 more authors
In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) Jul 2017
Word segmentation plays a pivotal role in improving any Arabic NLP application. Therefore, a lot of research has been spent in improving its accuracy. Off-the-shelf tools, however, are: i) complicated to use and ii) domain/dialect dependent. We explore three language-independent alternatives to morphological segmentation using: i) data-driven sub-word units, ii) characters as a unit of learning, and iii) word embeddings learned using a character CNN (Convolution Neural Network). On the tasks of Machine Translation and POS tagging, we found these methods to achieve close to, and occasionally surpass state-of-the-art performance. In our analysis, we show that a neural machine translation system is sensitive to the ratio of source and target tokens, and a ratio close to 1 or greater, gives optimal performance.
@inproceedings{sajjad-etal-2017-challenging,title={Challenging Language-Dependent Segmentation for {A}rabic: An Application to Machine Translation and Part-of-Speech Tagging},author={Sajjad, Hassan and Dalvi, Fahim and Durrani, Nadir and Abdelali, Ahmed and Belinkov, Yonatan and Vogel, Stephan},booktitle={Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},month=jul,year={2017},address={Vancouver, Canada},publisher={Association for Computational Linguistics},url={https://aclanthology.org/P17-2095},doi={10.18653/v1/P17-2095},pages={601--607},area={Word Segmentation},}
Understanding and Improving Morphological Learning in the Neural Machine Translation Decoder
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, and
2 more authors
In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers) Nov 2017
End-to-end training makes the neural machine translation (NMT) architecture simpler, yet elegant compared to traditional statistical machine translation (SMT). However, little is known about linguistic patterns of morphology, syntax and semantics learned during the training of NMT systems, and more importantly, which parts of the architecture are responsible for learning each of these phenomenon. In this paper we i) analyze how much morphology an NMT decoder learns, and ii) investigate whether injecting target morphology in the decoder helps it to produce better translations. To this end we present three methods: i) simultaneous translation, ii) joint-data learning, and iii) multi-task learning. Our results show that explicit morphological information helps the decoder learn target language morphology and improves the translation quality by 0.2–0.6 BLEU points.
@inproceedings{dalvi-etal-2017-understanding,author={Dalvi, Fahim and Durrani, Nadir and Sajjad, Hassan and Belinkov, Yonatan and Vogel, Stephan},booktitle={Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},month=nov,year={2017},address={Taipei, Taiwan},publisher={Asian Federation of Natural Language Processing},url={https://aclanthology.org/I17-1015},pages={142--151},area={Transfer Learning},}
Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks
Yonatan Belinkov, Lluı́s Màrquez, Hassan Sajjad, and
3 more authors
In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers) Nov 2017
While neural machine translation (NMT) models provide improved translation quality in an elegant framework, it is less clear what they learn about language. Recent work has started evaluating the quality of vector representations learned by NMT models on morphological and syntactic tasks. In this paper, we investigate the representations learned at different layers of NMT encoders. We train NMT systems on parallel data and use the models to extract features for training a classifier on two tasks: part-of-speech and semantic tagging. We then measure the performance of the classifier as a proxy to the quality of the original NMT model for the given task. Our quantitative analysis yields interesting insights regarding representation learning in NMT models. For instance, we find that higher layers are better at learning semantics while lower layers tend to be better for part-of-speech tagging. We also observe little effect of the target language on source-side representations, especially in higher quality models.
@inproceedings{belinkov-etal-2017-evaluating,title={Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks},author={Belinkov, Yonatan and M{\`a}rquez, Llu{\'\i}s and Sajjad, Hassan and Durrani, Nadir and Dalvi, Fahim and Glass, James},booktitle={Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},month=nov,year={2017},address={Taipei, Taiwan},publisher={Asian Federation of Natural Language Processing},url={https://aclanthology.org/I17-1001},pages={1--10},area={Representation Analysis},}
Neural Machine Translation Training in a Multi-Domain Scenario
Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and
2 more authors
In Proceedings of the 14th International Conference on Spoken Language Translation Dec 2017
In this paper, we explore alternative ways to train a neural machine translation system in a multi-domain scenario. We investigate data concatenation (with fine tuning), model stacking (multi-level fine tuning), data selection and multi-model ensemble. Our findings show that the best translation quality can be achieved by building an initial system on a concatenation of available out-of-domain data and then fine-tuning it on in-domain data. Model stacking works best when training begins with the furthest out-of-domain data and the model is incrementally fine-tuned with the next furthest domain and so on. Data selection did not give the best results, but can be considered as a decent compromise between training time and translation quality. A weighted ensemble of different individual models performed better than data selection. It is beneficial in a scenario when there is no time for fine-tuning an already trained model.
@inproceedings{sajjad-etal-2017-neural,title={Neural Machine Translation Training in a Multi-Domain Scenario},author={Sajjad, Hassan and Durrani, Nadir and Dalvi, Fahim and Belinkov, Yonatan and Vogel, Stephan},booktitle={Proceedings of the 14th International Conference on Spoken Language Translation},month=dec,year={2017},address={Tokyo, Japan},publisher={International Workshop on Spoken Language Translation},url={https://aclanthology.org/2017.iwslt-1.10},pages={66--73},area={Domain Adaptation}}
Continuous Space Reordering Models for Phrase-based MT
Nadir Durrani, and Fahim Dalvi
In Proceedings of the 14th International Conference on Spoken Language Translation Dec 2017
Bilingual sequence models improve phrase-based translation and reordering by overcoming phrasal independence assumption and handling long range reordering. However, due to data sparsity, these models often fall back to very small context sizes. This problem has been previously addressed by learning sequences over generalized representations such as POS tags or word clusters. In this paper, we explore an alternative based on neural network models. More concretely we train neuralized versions of lexicalized reordering [1] and the operation sequence models [2] using feed-forward neural network. Our results show improvements of up to 0.6 and 0.5 BLEU points on top of the baseline German!English and English!German systems. We also observed improvements compared to the systems that used POS tags and word clusters to train these models. Because we modify the bilingual corpus to integrate reordering operations, this allows us to also train a sequence-to-sequence neural MT model having explicit reordering triggers. Our motivation was to directly enable reordering information in the encoder-decoder framework, which otherwise relies solely on the attention model to handle long range reordering. We tried both coarser and fine-grained reordering operations. However, these experiments did not yield any improvements over the baseline Neural MT systems.
@inproceedings{durrani-dalvi-2017-continuous,title={Continuous Space Reordering Models for Phrase-based {MT}},author={Durrani, Nadir and Dalvi, Fahim},booktitle={Proceedings of the 14th International Conference on Spoken Language Translation},month=dec,year={2017},address={Tokyo, Japan},publisher={International Workshop on Spoken Language Translation},url={https://aclanthology.org/2017.iwslt-1.18},pages={129--136},area={Translation and Reordering},}
QCRI Live Speech Translation System
Fahim Dalvi, Yifan Zhang, Sameer Khurana, and
6 more authors
In Proceedings of the Software Demonstrations of the 15th Conference of the European Chapter of the Association for Computational Linguistics Apr 2017
This paper presents QCRI’s Arabic-to-English live speech translation system. It features modern web technologies to capture live audio, and broadcasts Arabic transcriptions and English translations simultaneously. Our Kaldi-based ASR system uses the Time Delay Neural Network (TDNN) architecture, while our Machine Translation (MT) system uses both phrase-based and neural frameworks. Although our neural MT system is slower than the phrase-based system, it produces significantly better translations and is memory efficient. The demo is available at \urlhttps://st.qcri.org/demos/livetranslation.
@inproceedings{dalvi-etal-2017-qcri,title={{QCRI} Live Speech Translation System},author={Dalvi, Fahim and Zhang, Yifan and Khurana, Sameer and Durrani, Nadir and Sajjad, Hassan and Abdelali, Ahmed and Mubarak, Hamdy and Ali, Ahmed and Vogel, Stephan},booktitle={Proceedings of the Software Demonstrations of the 15th Conference of the {E}uropean Chapter of the Association for Computational Linguistics},month=apr,year={2017},address={Valencia, Spain},publisher={Association for Computational Linguistics},url={https://aclanthology.org/E17-3016},pages={61--64},area={Demos and Tools},}
The SUMMA Platform Prototype
Renars Liepins, Ulrich Germann, Guntis Barzdins, and
43 more authors
In Proceedings of the Software Demonstrations of the 15th Conference of the European Chapter of the Association for Computational Linguistics Apr 2017
We present the first prototype of the SUMMA Platform: an integrated platform for multilingual media monitoring. The platform contains a rich suite of low-level and high-level natural language processing technologies: automatic speech recognition of broadcast media, machine translation, automated tagging and classification of named entities, semantic parsing to detect relationships between entities, and automatic construction / augmentation of factual knowledge bases. Implemented on the Docker platform, it can easily be deployed, customised, and scaled to large volumes of incoming media streams.
@inproceedings{liepins-etal-2017-summa,title={The {SUMMA} Platform Prototype},author={Liepins, Renars and Germann, Ulrich and Barzdins, Guntis and Birch, Alexandra and Renals, Steve and Weber, Susanne and van der Kreeft, Peggy and Bourlard, Herv{\'e} and Prieto, Jo{\~a}o and Klejch, Ond{\v{r}}ej and Bell, Peter and Lazaridis, Alexandros and Mendes, Alfonso and Riedel, Sebastian and Almeida, Mariana S. C. and Balage, Pedro and Cohen, Shay B. and Dwojak, Tomasz and Garner, Philip N. and Giefer, Andreas and Junczys-Dowmunt, Marcin and Imran, Hina and Nogueira, David and Ali, Ahmed and Miranda, Sebasti{\~a}o and Popescu-Belis, Andrei and Miculicich Werlen, Lesly and Papasarantopoulos, Nikos and Obamuyide, Abiola and Jones, Clive and Dalvi, Fahim and Vlachos, Andreas and Wang, Yang and Tong, Sibo and Sennrich, Rico and Pappas, Nikolaos and Narayan, Shashi and Damonte, Marco and Durrani, Nadir and Khurana, Sameer and Abdelali, Ahmed and Sajjad, Hassan and Vogel, Stephan and Sheppey, David and Hernon, Chris and Mitchell, Jeff},booktitle={Proceedings of the Software Demonstrations of the 15th Conference of the {E}uropean Chapter of the Association for Computational Linguistics},month=apr,year={2017},address={Valencia, Spain},publisher={Association for Computational Linguistics},url={https://aclanthology.org/E17-3029},pages={116--119},area={Demos and Tools}}
2016
Farasa: A Fast and Furious Segmenter for Arabic
Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and
1 more author
In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations Jun 2016
In this paper, we present Farasa, a fast and accurate Arabic segmenter. Our approach is based on SVM-rank using linear kernels. We measure the performance of the segmenter in terms of accuracy and efficiency, in two NLP tasks, namely Machine Translation (MT) and Information Retrieval (IR). Farasa outperforms or is at par with the stateof-the-art Arabic segmenters (Stanford and MADAMIRA), while being more than one order of magnitude faster.
@inproceedings{abdelali-etal-2016-farasa,title={{F}arasa: A Fast and Furious Segmenter for {A}rabic},author={Abdelali, Ahmed and Darwish, Kareem and Durrani, Nadir and Mubarak, Hamdy},booktitle={Proceedings of the 2016 Conference of the North {A}merican Chapter of the Association for Computational Linguistics: Demonstrations},month=jun,year={2016},address={San Diego, California},publisher={Association for Computational Linguistics},url={https://aclanthology.org/N16-3003},doi={10.18653/v1/N16-3003},pages={11--16},area={Demos and Tools}}
A Deep Fusion Model for Domain Adaptation in Phrase-based MT
Nadir Durrani, Hassan Sajjad, Shafiq Joty, and
1 more author
In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers Dec 2016
We present a novel fusion model for domain adaptation in Statistical Machine Translation. Our model is based on the joint source-target neural network Devlin et al., 2014, and is learned by fusing in- and out-domain models. The adaptation is performed by backpropagating errors from the output layer to the word embedding layer of each model, subsequently adjusting parameters of the composite model towards the in-domain data. On the standard tasks of translating English-to-German and Arabic-to-English TED talks, we observed average improvements of +0.9 and +0.7 BLEU points, respectively over a competition grade phrase-based system. We also demonstrate improvements over existing adaptation methods.
@inproceedings{durrani-etal-2016-deep,title={A Deep Fusion Model for Domain Adaptation in Phrase-based {MT}},author={Durrani, Nadir and Sajjad, Hassan and Joty, Shafiq and Abdelali, Ahmed},booktitle={Proceedings of {COLING} 2016, the 26th International Conference on Computational Linguistics: Technical Papers},month=dec,year={2016},address={Osaka, Japan},publisher={The COLING 2016 Organizing Committee},url={https://aclanthology.org/C16-1299},pages={3177--3187},area={Domain Adaptation}}
Eyes Don’t Lie: Predicting Machine Translation Quality Using Eye Movement
Hassan Sajjad, Francisco Guzmán, Nadir Durrani, and
4 more authors
In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Jun 2016
Poorly translated text is often disfluent and difficult to read. In contrast, well-formed translations require less time to process. In this paper, we model the differences in reading patterns of Machine Translation (MT) evaluators using novel features extracted from their gaze data, and we learn to predict the quality scores given by those evaluators. We test our predictions in a pairwise ranking scenario, measuring Kendall’s tau correlation with the judgments. We show that our features provide information beyond fluency, and can be combined with BLEU for better predictions. Furthermore, our results show that reading patterns can be used to build semi-automatic metrics that anticipate the scores given by the evaluators.
@inproceedings{sajjad-etal-2016-eyes,title={Eyes Don{'}t Lie: Predicting Machine Translation Quality Using Eye Movement},author={Sajjad, Hassan and Guzm{\'a}n, Francisco and Durrani, Nadir and Abdelali, Ahmed and Bouamor, Houda and Temnikova, Irina and Vogel, Stephan},booktitle={Proceedings of the 2016 Conference of the North {A}merican Chapter of the Association for Computational Linguistics: Human Language Technologies},month=jun,year={2016},address={San Diego, California},publisher={Association for Computational Linguistics},url={https://aclanthology.org/N16-1125},doi={10.18653/v1/N16-1125},pages={1082--1088},area={MT Evaluation}}
iAppraise: A Manual Machine Translation Evaluation Environment Supporting Eye-tracking
Ahmed Abdelali, Nadir Durrani, and Francisco Guzmán
In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations Jun 2016
We present iAppraise: an open-source framework that enables the use of eye-tracking for MT evaluation. It connects Appraise, an opensource toolkit for MT evaluation, to a low-cost eye-tracking device, to make its usage accessible to a broader audience. It also provides a set of tools for extracting and exploiting gaze data, which facilitate eye-tracking analysis. In this paper, we describe different modules of the framework, and explain how the tool can be used in a MT evaluation scenario. During the demonstration, the users will be able to perform an evaluation task, observe their own reading behavior during a replay of the session, and export and extract features from the data.
@inproceedings{abdelali-etal-2016-iappraise,title={i{A}ppraise: A Manual Machine Translation Evaluation Environment Supporting Eye-tracking},author={Abdelali, Ahmed and Durrani, Nadir and Guzm{\'a}n, Francisco},booktitle={Proceedings of the 2016 Conference of the North {A}merican Chapter of the Association for Computational Linguistics: Demonstrations},month=jun,year={2016},address={San Diego, California},publisher={Association for Computational Linguistics},url={https://aclanthology.org/N16-3004},doi={10.18653/v1/N16-3004},pages={17--21},area={Demos and Tools}}
QCRI’s Machine Translation Systems for IWSLT’16
Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and
1 more author
In Proceedings of the 13th International Conference on Spoken Language Translation Dec 2016
This paper describes QCRI’s machine translation systems for the IWSLT 2016 evaluation campaign. We participated in the Arabic→English and English→Arabic tracks. We built both Phrase-based and Neural machine translation models, in an effort to probe whether the newly emerged NMT framework surpasses the traditional phrase-based systems in Arabic-English language pairs. We trained a very strong phrase-based system including, a big language model, the Operation Sequence Model, Neural Network Joint Model and Class-based models along with different domain adaptation techniques such as MML filtering, mixture modeling and using fine tuning over NNJM model. However, a Neural MT system, trained by stacking data from different genres through fine-tuning, and applying ensemble over 8 models, beat our very strong phrase-based system by a significant 2 BLEU points margin in Arabic→English direction. We did not obtain similar gains in the other direction but were still able to outperform the phrase-based system. We also applied system combination on phrase-based and NMT outputs.
@inproceedings{durrani-etal-2016-qcris,title={{QCRI}{'}s Machine Translation Systems for {IWSLT}{'}16},author={Durrani, Nadir and Dalvi, Fahim and Sajjad, Hassan and Vogel, Stephan},booktitle={Proceedings of the 13th International Conference on Spoken Language Translation},month=dec,year={2016},address={Seattle, Washington D.C},publisher={International Workshop on Spoken Language Translation},url={https://aclanthology.org/2016.iwslt-1.18},area={System Descriptions},}
Enabling Medical Translation for Low-Resource Languages
Ahmad Musleh, Nadir Durrani, Irina P. Temnikova, and
3 more authors
In Computational Linguistics and Intelligent Text Processing - 17th International
Conference, CICLing 2016, Konya, Turkey, April 3-9, 2016, Revised
Selected Papers, Part II Dec 2016
We present research towards bridging the language gap between migrant workers in Qatar and medical staff. In particular, we present the first steps towards the development of a real-world HindiEnglish machine translation system for doctor-patient communication. As this is a low-resource language pair, especially for speech and for the medical domain, our initial focus has been on gathering suitable training data from various sources. We applied a variety of methods ranging from fully automatic extraction from the Web to manual annotation of test data. Moreover, we developed a method for automatically augmenting the training data with synthetically generated variants, which yielded a very sizable improvement of more than 3 BLEU points absolute.
@inproceedings{MuslehDTNVA16,author={Musleh, Ahmad and Durrani, Nadir and Temnikova, Irina P. and Nakov, Preslav and Vogel, Stephan and Alsaad, Osama},editor={Gelbukh, Alexander F.},title={Enabling Medical Translation for Low-Resource Languages},booktitle={Computational Linguistics and Intelligent Text Processing - 17th International
Conference, CICLing 2016, Konya, Turkey, April 3-9, 2016, Revised
Selected Papers, Part {II}},series={Lecture Notes in Computer Science},volume={9624},pages={3--16},publisher={Springer},year={2016},url={https://doi.org/10.1007/978-3-319-75487-1\_1},doi={10.1007/978-3-319-75487-1\_1},timestamp={Fri, 27 Dec 2019 21:23:08 +0100},biburl={https://dblp.org/rec/conf/cicling/MuslehDTNVA16.bib},bibsource={dblp computer science bibliography, https://dblp.org},area={Medical},}
2015
The Operation Sequence Model—Combining N-Gram-Based and Phrase-Based Statistical Machine Translation
Nadir Durrani, Helmut Schmid, Alexander Fraser, and
2 more authors
In this article, we present a novel machine translation model, the Operation Sequence Model(OSM), which combines the benefits of phrase-based and N-gram-based statistical machine translation (SMT) and remedies their drawbacks. The model represents the translation process as a linear sequence of operations. The sequence includes not only translation operations but also reordering operations. As in N-gram-based SMT, the model is: (i) based on minimal translation units, (ii) takes both source and target information into account, (iii) does not make a phrasal independence assumption, and (iv) avoids the spurious phrasal segmentation problem. As in phrase-based SMT, the model (i) has the ability to memorize lexical reordering triggers, (ii) builds the search graph dynamically, and (iii) decodes with large translation units during search. The unique properties of the model are (i) its strong coupling of reordering and translation where translation and reordering decisions are conditioned on n previous translation and reordering decisions, and (ii) the ability to model local and long-range reorderings consistently. Using BLEU as a metric of translation accuracy, we found that our system performs significantly better than state-of-the-art phrase-based systems (Moses and Phrasal) and N-gram-based systems (Ncode) on standard translation tasks. We compare the reordering component of the OSM to the Moses lexical reordering model by integrating it into Moses. Our results show that OSM outperforms lexicalized reordering on all translation tasks. The translation quality is shown to be improved further by learning generalized representations with a POS-based OSM.
@article{durrani-etal-2015-operation,title={The Operation Sequence {M}odel{---}{C}ombining N-Gram-Based and Phrase-Based Statistical Machine Translation},author={Durrani, Nadir and Schmid, Helmut and Fraser, Alexander and Koehn, Philipp and Sch{\"u}tze, Hinrich},journal={Computational Linguistics},volume={41},number={2},month=jun,year={2015},address={Cambridge, MA},publisher={MIT Press},url={https://aclanthology.org/J15-2001},doi={10.1162/COLI_a_00218},pages={157--186},area={Translation and Reordering},}
How to Avoid Unwanted Pregnancies: Domain Adaptation using Neural Network Models
Shafiq Joty, Hassan Sajjad, Nadir Durrani, and
3 more authors
In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing Sep 2015
We present novel models for domain adaptation based on the neural network joint model (NNJM). Our models maximize the cross entropy by regularizing the loss function with respect to in-domain model. Domain adaptation is carried out by assigning higher weight to out-domain sequences that are similar to the in-domain data. In our alternative model we take a more restrictive approach by additionally penalizing sequences similar to the outdomain data. Our models achieve better perplexities than the baseline NNJM models and give improvements of up to 0.5 and 0.6 BLEU points in Arabic-to-English and English-to-German language pairs, on a standard task of translating TED talks
@inproceedings{joty-etal-2015-avoid,title={How to Avoid Unwanted Pregnancies: Domain Adaptation using Neural Network Models},author={Joty, Shafiq and Sajjad, Hassan and Durrani, Nadir and Al-Mannai, Kamla and Abdelali, Ahmed and Vogel, Stephan},booktitle={Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing},month=sep,year={2015},address={Lisbon, Portugal},publisher={Association for Computational Linguistics},url={https://aclanthology.org/D15-1147},doi={10.18653/v1/D15-1147},pages={1259--1270},area={Domain Adaptation},}
Using joint models or domain adaptation in statistical machine translation
Nadir Durrani, Hassan Sajjad, Shafiq Joty, and
2 more authors
In Proceedings of Machine Translation Summit XV: Papers Oct 2015
Joint models have recently shown to improve the state-of-the-art in machine translation (MT). We apply EM-based mixture modeling and data selection techniques using two joint models, namely the Operation Sequence Model or OSM — an ngram-based translation and reordering model, and the Neural Network Joint Model or NNJM — a continuous space translation model, to carry out domain adaptation for MT. The diversity of the two models, OSM with inherit reordering information and NNJM with continuous space modeling makes them interesting to be explored for this task. Our contribution in this paper is fusing the existing known techniques (linear interpolation, cross-entropy) with the state-of-the-art MT models (OSM, NNJM). On a standard task of translating German-to-English and Arabic-to-English IWSLT TED talks, we observed statistically significant improvements of up to +0.9 BLEU points.
@inproceedings{durrani-etal-2015-using,title={Using joint models or domain adaptation in statistical machine translation},author={Durrani, Nadir and Sajjad, Hassan and Joty, Shafiq and Abdelali, Ahmed and Vogel, Stephan},booktitle={Proceedings of Machine Translation Summit XV: Papers},month=oct,year={2015},address={Miami, USA},url={https://aclanthology.org/2015.mtsummit-papers.10},area={Domain Adaptation},}
QCMUQ@QALB-2015 Shared Task: Combining Character level MT and Error-tolerant Finite-State Recognition for Arabic Spelling Correction
Houda Bouamor, Hassan Sajjad, Nadir Durrani, and
1 more author
In Proceedings of the Second Workshop on Arabic Natural Language Processing Jul 2015
We describe the CMU-Q and QCRI’s joint efforts in building a spelling correction system for Arabic in the QALB 2015 Shared Task. Our system is based on a hybrid pipeline that combines rule-based linguistic techniques with statistical methods using language modeling and machine translation, as well as an error-tolerant finite-state automata method. We trained and tested our spelling corrector using the dataset provided by the shared task organizers. Our system outperforms the baseline system and yeilds better correction quality with an F-score of 68.12 on L1-test-2015 testset and 38.90 on the L2-test2015. This ranks us 2nd in the L2 subtask and 5th in the L1 subtask.
@inproceedings{bouamor-etal-2015-qcmuq,title={{QCMUQ}@{QALB}-2015 Shared Task: Combining Character level {MT} and Error-tolerant Finite-State Recognition for {A}rabic Spelling Correction},author={Bouamor, Houda and Sajjad, Hassan and Durrani, Nadir and Oflazer, Kemal},booktitle={Proceedings of the Second Workshop on {A}rabic Natural Language Processing},month=jul,year={2015},address={Beijing, China},publisher={Association for Computational Linguistics},url={https://aclanthology.org/W15-3217},doi={10.18653/v1/W15-3217},pages={144--149},area={System Descriptions},}
QCN Egyptian Arabic to English Machine
Translation System for NIST OpenMT15
Hassan Sajjad, Nadir Durrani, Francisco Guzman, and
6 more authors
The paper describes the Egyptian Arabicto-English statistical machine translation (SMT) system that the QCRI-Columbia NYUAD (QCN) group submitted to the NIST OpenMT’2015 competition. The competition focused on informal dialectal Arabic, as used in SMS, chat, and speech. Thus, our efforts focused on processing and standardizing Arabic, e.g., using tools such as 3arrib and MADAMIRA. We further trained a phrase-based SMT system using state-of-the-art features and components such as operation sequence model, class-based language model, sparse features, neural network joint model, genrebased hierarchically-interpolated language model, unsupervised transliteration mining, phrase-table merging, and hypothesis combination. Our system ranked second on all three genres
@inproceedings{sajjad:nist15,author={Sajjad, Hassan and Durrani, Nadir and Guzman, Francisco and Nakov, Preslav and Abdelali, Ahmed and Vogel, Stephan and Salloum, Wael and Kholy, Ahmed El and Habash, Nizar},booktitle={Workshop of NIST OpenMT15},location={Washington DC, US},month=jun,title={QCN Egyptian Arabic to English Machine
Translation System for NIST OpenMT15},year={2015},area={System Descriptions},}
2014
Integrating an Unsupervised Transliteration Model into Statistical Machine Translation
Nadir Durrani, Hassan Sajjad, Hieu Hoang, and
1 more author
In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, volume 2: Short Papers Apr 2014
We investigate three methods for integrating an unsupervised transliteration model into an end-to-end SMT system. We induce a transliteration model from parallel data and use it to translate OOV words. Our approach is fully unsupervised and language independent. In the methods to integrate transliterations, we observed improvements from 0.23-0.75 (∆ 0.41) BLEU points across 7 language pairs. We also show that our mined transliteration corpora provide better rule coverage and translation quality compared to the gold standard transliteration corpora.
@inproceedings{durrani-etal-2014-integrating,title={Integrating an Unsupervised Transliteration Model into Statistical Machine Translation},author={Durrani, Nadir and Sajjad, Hassan and Hoang, Hieu and Koehn, Philipp},booktitle={Proceedings of the 14th Conference of the {E}uropean Chapter of the Association for Computational Linguistics, volume 2: Short Papers},month=apr,year={2014},address={Gothenburg, Sweden},publisher={Association for Computational Linguistics},url={https://aclanthology.org/E14-4029},doi={10.3115/v1/E14-4029},pages={148--153},area={Transliteration},}
Investigating the Usefulness of Generalized Word Representations in SMT
Nadir Durrani, Philipp Koehn, Helmut Schmid, and
1 more author
In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers Aug 2014
We investigate the use of generalized representations (POS, morphological analysis and word clusters) in phrase-based models and the N-gram-based Operation Sequence Model (OSM). Our integration enables these models to learn richer lexical and reordering patterns, consider wider contextual information and generalize better in sparse data conditions. When interpolating generalized OSM models on the standard IWSLT and WMT tasks we observed improvements of up to +1.35 on the English-to-German task and +0.63 for the German-to-English task. Using automatically generated word classes in standard phrase-based models and the OSM models yields an average improvement of +0.80 across 8 language pairs on the IWSLT shared task
@inproceedings{durrani-etal-2014-investigating,title={Investigating the Usefulness of Generalized Word Representations in {SMT}},author={Durrani, Nadir and Koehn, Philipp and Schmid, Helmut and Fraser, Alexander},booktitle={Proceedings of {COLING} 2014, the 25th International Conference on Computational Linguistics: Technical Papers},month=aug,year={2014},address={Dublin, Ireland},publisher={Dublin City University and Association for Computational Linguistics},url={https://aclanthology.org/C14-1041},pages={421--432},area={Translation and Reordering}}
Improving machine translation via triangulation and transliteration
Nadir Durrani, and Philipp Koehn
In Proceedings of the 17th Annual conference of the European Association for Machine Translation Jun 2014
In this paper we improve Urdu-Hindi-English machine translation through triangulation and transliteration. First we built an Urdu-Hindi SMT system by inducing triangulated and transliterated phrase-tables from Urdu–English and Hindi–English phrase translation models. We then use it to translate the Urdu part of the Urdu-English parallel data into Hindi, thus creating an artificial Hindi-English parallel data. Our phrase-translation strategies give an improvement of up to +3.35 BLEU points over a baseline Urdu-Hindi system. The synthesized data improve Hindi-English system by +0.35 and English-Hindi system by +1.0 BLEU points.
@inproceedings{durrani-koehn-2014-improving,title={Improving machine translation via triangulation and transliteration},author={Durrani, Nadir and Koehn, Philipp},booktitle={Proceedings of the 17th Annual conference of the European Association for Machine Translation},month=jun,year={2014},address={Dubrovnik, Croatia},publisher={European Association for Machine Translation},url={https://aclanthology.org/2014.eamt-1.17},pages={71--78},area={Transliteration}}
Improving Egyptian-to-English SMT by Mapping Egyptian into MSA
Nadir Durrani, Yaser Al-Onaizan, and Abraham Ittycheriah
In Computational Linguistics and Intelligent Text Processing - 15th International
Conference, CICLing 2014, Kathmandu, Nepal, April 6-12, 2014, Proceedings,
Part II Jun 2014
One of the aims of DARPA BOLT project is to translate the Egyptian blog data into English. While the parallel data for MSA-English is abundantly available, sparsely exists for Egyptian-English and Egyptian-MSA. A notable drop in the translation quality is observed when translating Egyptian to English in comparison with translating from MSA to English. One of the reasons for this drop is the high OOV rate, where as another is the dialectal differences between training and test data. This work is focused on improving Egyptian-to-English translation by bridging the gap between Egyptian and MSA. First we try to reduce the OOV rate by proposing MSA candidates for the unknown Egyptian words through different methods such as spelling correction, suggesting synonyms based on context etc. Secondly we apply convolution model using English as a pivot to map Egyptian words into MSA. We then evaluate our edits by running decoder built on MSA-to-English data. Our spelling-based correction shows an improvement of 1.7 BLEU points over the baseline system, that translates unedited Egyptian into English.
@inproceedings{DurraniAI14,author={Durrani, Nadir and Al{-}Onaizan, Yaser and Ittycheriah, Abraham},editor={Gelbukh, Alexander F.},title={Improving Egyptian-to-English {SMT} by Mapping Egyptian into {MSA}},booktitle={Computational Linguistics and Intelligent Text Processing - 15th International
Conference, CICLing 2014, Kathmandu, Nepal, April 6-12, 2014, Proceedings,
Part {II}},series={Lecture Notes in Computer Science},volume={8404},pages={271--282},publisher={Springer},year={2014},url={https://doi.org/10.1007/978-3-642-54903-8\_23},doi={10.1007/978-3-642-54903-8\_23},timestamp={Tue, 14 May 2019 10:00:46 +0200},biburl={https://dblp.org/rec/conf/cicling/DurraniAI14.bib},bibsource={dblp computer science bibliography, https://dblp.org},area={Translation and Reordering}}
Edinburgh’s Phrase-based Machine Translation Systems for WMT-14
Nadir Durrani, Barry Haddow, Philipp Koehn, and
1 more author
In Proceedings of the Ninth Workshop on Statistical Machine Translation Jun 2014
This paper describes the University of Edinburgh’s (UEDIN) phrase-based submissions to the translation and medical translation shared tasks of the 2014 Workshop on Statistical Machine Translation (WMT). We participated in all language pairs. We have improved upon our 2013 system by i) using generalized representations, specifically automatic word clusters for translations out of English, ii) using unsupervised character-based models to translate unknown words in RussianEnglish and Hindi-English pairs, iii) synthesizing Hindi data from closely-related Urdu data, and iv) building huge language on the common crawl corpus.
@inproceedings{durrani-etal-2014-edinburghs,title={{E}dinburgh{'}s Phrase-based Machine Translation Systems for {WMT}-14},author={Durrani, Nadir and Haddow, Barry and Koehn, Philipp and Heafield, Kenneth},booktitle={Proceedings of the Ninth Workshop on Statistical Machine Translation},month=jun,year={2014},address={Baltimore, Maryland, USA},publisher={Association for Computational Linguistics},url={https://aclanthology.org/W14-3309},doi={10.3115/v1/W14-3309},pages={97--104},area={System Descriptions}}
Edinburgh SLT and MT system description for the IWSLT 2014 evaluation
Alexandra Birch, Matthias Huck, Nadir Durrani, and
2 more authors
In Proceedings of the 11th International Workshop on Spoken Language Translation: Evaluation Campaign Dec 2014
This paper describes the University of Edinburgh’s spoken language translation (SLT) and machine translation (MT) systems for the IWSLT 2014 evaluation campaign. In the SLT track, we participated in the German↔English and English→French tasks. In the MT track, we participated in the German↔English, English→French, Arabic↔English, Farsi→English, Hebrew→English, Spanish↔English, and Portuguese-Brazil↔English tasks. For our SLT submissions, we experimented with comparing operation sequence models with bilingual neural network language models. For our MT submissions, we explored using unsupervised transliteration for languages which have a different script than English, in particular for Arabic, Farsi, and Hebrew. We also investigated syntax-based translation and system combination.
@inproceedings{birch-etal-2014-edinburgh,title={{E}dinburgh {SLT} and {MT} system description for the {IWSLT} 2014 evaluation},author={Birch, Alexandra and Huck, Matthias and Durrani, Nadir and Bogoychev, Nikolay and Koehn, Philipp},booktitle={Proceedings of the 11th International Workshop on Spoken Language Translation: Evaluation Campaign},month=dec,year={2014},address={Lake Tahoe, California},url={https://aclanthology.org/2014.iwslt-evaluation.6},pages={49--56},area={System Descriptions}}
EU-BRIDGE MT: Combined Machine Translation
Markus Freitag, Stephan Peitz, Joern Wuebker, and
10 more authors
In Proceedings of the Ninth Workshop on Statistical Machine Translation Jun 2014
This paper describes one of the collaborative efforts within EU-BRIDGE to further advance the state of the art in machine translation between two European language pairs, German→English and English→German. Three research institutes involved in the EU-BRIDGE project combined their individual machine translation systems and participated with a joint setup in the shared translation task of the evaluation campaign at the ACL 2014 Eighth Workshop on Statistical Machine Translation (WMT 2014). We combined up to nine different machine translation engines via system combination. RWTH Aachen University, the University of Edinburgh, and Karlsruhe Institute of Technology developed several individual systems which serve as system combination input. We devoted special attention to building syntax-based systems and combining them with the phrasebased ones. The joint setups yield empirical gains of up to 1.6 points in BLEU and 1.0 points in TER on the WMT newstest2013 test set compared to the best single systems.
@inproceedings{freitag-etal-2014-eu,title={{EU-BRIDGE} {MT}: Combined Machine Translation},author={Freitag, Markus and Peitz, Stephan and Wuebker, Joern and Ney, Hermann and Huck, Matthias and Sennrich, Rico and Durrani, Nadir and Nadejde, Maria and Williams, Philip and Koehn, Philipp and Herrmann, Teresa and Cho, Eunah and Waibel, Alex},booktitle={Proceedings of the Ninth Workshop on Statistical Machine Translation},month=jun,year={2014},address={Baltimore, Maryland, USA},publisher={Association for Computational Linguistics},url={https://aclanthology.org/W14-3310},doi={10.3115/v1/W14-3310},pages={105--113},area={System Descriptions}}
Combined spoken language translation
Markus Freitag, Joern Wuebker, Stephan Peitz, and
13 more authors
In Proceedings of the 11th International Workshop on Spoken Language Translation: Evaluation Campaign Dec 2014
EU-BRIDGE is a European research project which is aimed at developing innovative speech translation technology. One of the collaborative efforts within EU-BRIDGE is to produce joint submissions of up to four different partners to the evaluation campaign at the 2014 International Workshop on Spoken Language Translation (IWSLT). We submitted combined translations to the German→English spoken language translation (SLT) track as well as to the German→English, English→German and English→French machine translation (MT) tracks. In this paper, we present the techniques which were applied by the different individual translation systems of RWTH Aachen University, the University of Edinburgh, Karlsruhe Institute of Technology, and Fondazione Bruno Kessler. We then show the combination approach developed at RWTH Aachen University which combined the individual systems. The consensus translations yield empirical gains of up to 2.3 points in BLEU and 1.2 points in TER compared to the best individual system.
@inproceedings{freitag-etal-2014-combined,title={Combined spoken language translation},author={Freitag, Markus and Wuebker, Joern and Peitz, Stephan and Ney, Hermann and Huck, Matthias and Birch, Alexandra and Durrani, Nadir and Koehn, Philipp and Mediani, Mohammed and Slawik, Isabel and Niehues, Jan and Cho, Eunach and Waibel, Alex and Bertoldi, Nicola and Cettolo, Mauro and Federico, Marcello},booktitle={Proceedings of the 11th International Workshop on Spoken Language Translation: Evaluation Campaign},month=dec,year={2014},address={Lake Tahoe, California},url={https://aclanthology.org/2014.iwslt-evaluation.7},pages={57--64},area={System Descriptions},}
2013
Can Markov Models Over Minimal Translation Units Help Phrase-Based SMT?
Nadir Durrani, Alexander Fraser, Helmut Schmid, and
2 more authors
In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) Aug 2013
The phrase-based and N-gram-based SMT frameworks complement each other. While the former is better able to memorize, the latter provides a more principled model that captures dependencies across phrasal boundaries. Some work has been done to combine insights from these two frameworks. A recent successful attempt showed the advantage of using phrasebased search on top of an N-gram-based model. We probe this question in the reverse direction by investigating whether integrating N-gram-based translation and reordering models into a phrase-based decoder helps overcome the problematic phrasal independence assumption. A large scale evaluation over 8 language pairs shows that performance does significantly improve.
@inproceedings{durrani-etal-2013-markov,title={Can {M}arkov Models Over Minimal Translation Units Help Phrase-Based {SMT}?},author={Durrani, Nadir and Fraser, Alexander and Schmid, Helmut and Hoang, Hieu and Koehn, Philipp},booktitle={Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},month=aug,year={2013},address={Sofia, Bulgaria},publisher={Association for Computational Linguistics},url={https://aclanthology.org/P13-2071},pages={399--405},area={Decoding}}
Model With Minimal Translation Units, But Decode With Phrases
Nadir Durrani, Alexander Fraser, and Helmut Schmid
In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Jun 2013
N-gram-based models co-exist with their phrase-based counterparts as an alternative SMT framework. Both techniques have pros and cons. While the N-gram-based framework provides a better model that captures both source and target contexts and avoids spurious phrasal segmentation, the ability to memorize and produce larger translation units gives an edge to the phrase-based systems during decoding, in terms of better search performance and superior selection of translation units. In this paper we combine N-grambased modeling with phrase-based decoding, and obtain the benefits of both approaches. Our experiments show that using this combination not only improves the search accuracy of the N-gram model but that it also improves the BLEU scores. Our system outperforms state-of-the-art phrase-based systems (Moses and Phrasal) and N-gram-based systems by a significant margin on German, French and Spanish to English translation tasks.
@inproceedings{durrani-etal-2013-model,author={Durrani, Nadir and Fraser, Alexander and Schmid, Helmut},booktitle={Proceedings of the 2013 Conference of the North {A}merican Chapter of the Association for Computational Linguistics: Human Language Technologies},month=jun,year={2013},address={Atlanta, Georgia},publisher={Association for Computational Linguistics},url={https://aclanthology.org/N13-1001},pages={1--11},area={Decoding},}
Edinburgh’s Machine Translation Systems for European Language Pairs
Nadir Durrani, Barry Haddow, Kenneth Heafield, and
1 more author
In Proceedings of the Eighth Workshop on Statistical Machine Translation Aug 2013
We validated various novel and recently proposed methods for statistical machine translation on 10 language pairs, using large data resources. We saw gains from optimizing parameters, training with sparse features, the operation sequence model, and domain adaptation techniques. We also report on utilizing a huge language model trained on 126 billion tokens.
@inproceedings{durrani-etal-2013-edinburghs,title={{E}dinburgh{'}s Machine Translation Systems for {E}uropean Language Pairs},author={Durrani, Nadir and Haddow, Barry and Heafield, Kenneth and Koehn, Philipp},booktitle={Proceedings of the Eighth Workshop on Statistical Machine Translation},month=aug,year={2013},address={Sofia, Bulgaria},publisher={Association for Computational Linguistics},url={https://aclanthology.org/W13-2212},pages={114--121},area={System Descriptions},}
English SLT and MT system description for the IWSLT 2013 evaluation
Alexandra Birch, Nadir Durrani, and Philipp Koehn
In Proceedings of the 10th International Workshop on Spoken Language Translation: Evaluation Campaign Dec 2013
This paper gives a description of the University of Edinburgh’s (UEDIN) systems for IWSLT 2013. We participated in all the MT tracks and the German-to-English and Englishto-French SLT tracks. Our SLT submissions experimented with including ASR uncertainty into the decoding process via confusion networks, and looked at different ways of punctuating ASR output. Our MT submissions are mainly based on a system used in the recent evaluation campaign at the Workshop on Statistical Machine Translation [1]. We additionally explored the use of generalized representations (Brown clusters, POS and morphological tags) translating out of English into European languages.
@inproceedings{birch-etal-2013-english,title={{E}nglish {SLT} and {MT} system description for the {IWSLT} 2013 evaluation},author={Birch, Alexandra and Durrani, Nadir and Koehn, Philipp},booktitle={Proceedings of the 10th International Workshop on Spoken Language Translation: Evaluation Campaign},month=dec,year={2013},address={Heidelberg, Germany},url={https://aclanthology.org/2013.iwslt-evaluation.3},area={System Descriptions}}
EU-BRIDGE MT: text translation of talks in the EU-BRIDGE project
Markus Freitag, Stephan Peitz, Joern Wuebker, and
12 more authors
In Proceedings of the 10th International Workshop on Spoken Language Translation: Evaluation Campaign Dec 2013
EU-BRIDGE1 is a European research project which is aimed at developing innovative speech translation technology. This paper describes one of the collaborative efforts within EUBRIDGE to further advance the state of the art in machine translation between two European language pairs, English→French and German→English. Four research institutions involved in the EU-BRIDGE project combined their individual machine translation systems and participated with a joint setup in the machine translation track of the evaluation campaign at the 2013 International Workshop on Spoken Language Translation (IWSLT). We present the methods and techniques to achieve high translation quality for text translation of talks which are applied at RWTH Aachen University, the University of Edinburgh, Karlsruhe Institute of Technology, and Fondazione Bruno Kessler. We then show how we have been able to considerably boost translation performance (as measured in terms of the metrics BLEU and TER) by means of system combination. The joint setups yield empirical gains of up to 1.4 points in BLEU and 2.8 points in TER on the IWSLT test sets compared to the best single systems.
@inproceedings{freitag-etal-2013-eu,title={{EU}-{BRIDGE} {MT}: text translation of talks in the {EU}-{BRIDGE} project},author={Freitag, Markus and Peitz, Stephan and Wuebker, Joern and Ney, Hermann and Durrani, Nadir and Huck, Matthias and Koehn, Philipp and Ha, Thanh-Le and Niehues, Jan and Mediani, Mohammed and Herrmann, Teresa and Waibel, Alex and Bertoldi, Nicola and Cettolo, Mauro and Federico, Marcello},booktitle={Proceedings of the 10th International Workshop on Spoken Language Translation: Evaluation Campaign},month=dec,year={2013},address={Heidelberg, Germany},url={https://aclanthology.org/2013.iwslt-evaluation.16},area={System Descriptions}}
Munich-Edinburgh-Stuttgart Submissions of OSM Systems at WMT13
Nadir Durrani, Alexander Fraser, Helmut Schmid, and
2 more authors
In Proceedings of the Eighth Workshop on Statistical Machine Translation Aug 2013
This paper describes Munich-EdinburghStuttgart’s submissions to the Eighth Workshop on Statistical Machine Translation. We report results of the translation tasks from German, Spanish, Czech and Russian into English and from English to German, Spanish, Czech, French and Russian. The systems described in this paper use OSM (Operation Sequence Model). We explain different pre-/post-processing steps that we carried out for different language pairs. For German-English we used constituent parsing for reordering and compound splitting as preprocessing steps. For Russian-English we transliterated the unknown words. The transliteration system is learned with the help of an unsupervised transliteration mining algorithm
@inproceedings{durrani-etal-2013-munich,title={{M}unich-{E}dinburgh-{S}tuttgart Submissions of {OSM} Systems at {WMT}13},author={Durrani, Nadir and Fraser, Alexander and Schmid, Helmut and Sajjad, Hassan and Farkas, Rich{\'a}rd},booktitle={Proceedings of the Eighth Workshop on Statistical Machine Translation},month=aug,year={2013},address={Sofia, Bulgaria},publisher={Association for Computational Linguistics},url={https://aclanthology.org/W13-2213},pages={122--127},area={System Descriptions}}
Munich-Edinburgh-Stuttgart Submissions at WMT13: Morphological and Syntactic Processing for SMT
Marion Weller, Max Kisselew, Svetlana Smekalova, and
5 more authors
In Proceedings of the Eighth Workshop on Statistical Machine Translation Aug 2013
@inproceedings{weller-etal-2013-munich,title={{M}unich-{E}dinburgh-{S}tuttgart Submissions at {WMT}13: Morphological and Syntactic Processing for {SMT}},author={Weller, Marion and Kisselew, Max and Smekalova, Svetlana and Fraser, Alexander and Schmid, Helmut and Durrani, Nadir and Sajjad, Hassan and Farkas, Rich{\'a}rd},booktitle={Proceedings of the Eighth Workshop on Statistical Machine Translation},month=aug,year={2013},address={Sofia, Bulgaria},publisher={Association for Computational Linguistics},url={https://aclanthology.org/W13-2230},pages={232--239},area={System Descriptions}}
QCRI-MES Submission at WMT13: Using Transliteration Mining to Improve Statistical Machine Translation
Hassan Sajjad, Svetlana Smekalova, Nadir Durrani, and
2 more authors
In Proceedings of the Eighth Workshop on Statistical Machine Translation Aug 2013
This paper describes QCRI-MES’s submission on the English-Russian dataset to the Eighth Workshop on Statistical Machine Translation. We generate improved word alignment of the training data by incorporating an unsupervised transliteration mining module to GIZA++ and build a phrase-based machine translation system. For tuning, we use a variation of PRO which provides better weights by optimizing BLEU+1 at corpus-level. We transliterate out-of-vocabulary words in a postprocessing step by using a transliteration system built on the transliteration pairs extracted using an unsupervised transliteration mining system. For the Russian to English translation direction, we apply linguistically motivated pre-processing on the Russian side of the data.
@inproceedings{sajjad-etal-2013-qcri,title={{QCRI}-{MES} Submission at {WMT}13: Using Transliteration Mining to Improve Statistical Machine Translation},author={Sajjad, Hassan and Smekalova, Svetlana and Durrani, Nadir and Fraser, Alexander and Schmid, Helmut},booktitle={Proceedings of the Eighth Workshop on Statistical Machine Translation},month=aug,year={2013},address={Sofia, Bulgaria},publisher={Association for Computational Linguistics},url={https://aclanthology.org/W13-2228},pages={219--224},area={System Descriptions},}
English to Urdu Hierarchical Phrase-based Statistical Machine Translation
Nadeem Khan, Muhammad Waqas Anwar, Usama Ijaz Bajwa, and
1 more author
In Proceedings of the 4th Workshop on South and Southeast Asian Natural Language Processing Oct 2013
@inproceedings{khan-etal-2013-english,title={{E}nglish to {U}rdu Hierarchical Phrase-based Statistical Machine Translation},author={Khan, Nadeem and Anwar, Muhammad Waqas and Bajwa, Usama Ijaz and Durrani, Nadir},booktitle={Proceedings of the 4th Workshop on South and Southeast {A}sian Natural Language Processing},month=oct,year={2013},address={Nagoya Congress Center, Nagoya, Japan},publisher={Asian Federation of Natural Language Processing},url={https://aclanthology.org/W13-4709},pages={72--76},}
Machine Translation Approaches and Survey for Indian Languages
Nadeem Jadoon Khan, Waqas Anwar, and Nadir Durrani
In this study, we present an analysis regarding the performance of the state-of-art Phrasebased Statistical Machine Translation (SMT) on multiple Indian languages. We report baseline systems on several language pairs. The motivation of this study is to promote the development of SMT and linguistic resources for these language pairs, as the current state-of-the-art is quite bleak due to sparse data resources. The success of an SMT system is contingent on the availability of a large parallel corpus. Such data is necessary to reliably estimate translation probabilities. We report the performance of baseline systems translating from Indian languages (Bengali, Guajarati, Hindi, Malayalam, Punjabi, Tamil, Telugu and Urdu) into English with average 10% accurate results for all the language pairs.
doi = {10.48550/ARXIV.1701.04290},author = {Khan, Nadeem Jadoon and Anwar, Waqas and Durrani, Nadir},keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},title = {Machine Translation Approaches and Survey for Indian Languages},publisher = {arXiv},year = {2013},copyright = {arXiv.org perpetual, non-exclusive license},area = {Surveys},}
2012
A joint translation model with integrated reordering
This dissertation aims at combining the benefits and to remedy the flaws of the two popular frameworks in statistical machine translation, namely Phrasebased MT and N-gram-based MT. Phrase-based MT advanced the state-of-the art towards translating phrases3 than words. By memorizing phrases, phrasal MT, is able to learn local reorderings, and handling of other local dependencies such as insertions, deletions etc. Inter-phrasal reorderings are handled through the lexicalized reordering model, which remains the state-of-the-art model for reordering in phrase-based SMT till date.
@phdthesis{DBLP:phd/dnb/Durrani12,author={Durrani, Nadir},title={A joint translation model with integrated reordering},school={University of Stuttgart},year={2012},url={http://elib.uni-stuttgart.de/opus/volltexte/2012/7977/},urn={urn:nbn:de:bsz:93-opus-79774},timestamp={Sat, 17 Jul 2021 09:07:30 +0200},biburl={https://dblp.org/rec/phd/dnb/Durrani12.bib},bibsource={dblp computer science bibliography, https://dblp.org},area={Theses}}
2011
A Joint Sequence Translation Model with Integrated Reordering
Nadir Durrani, Helmut Schmid, and Alexander Fraser
In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies Jun 2011
We present a novel machine translation model which models translation by a linear sequence of operations. In contrast to the “N-gram” model, this sequence includes not only translation but also reordering operations. Key ideas of our model are (i) a new reordering approach which better restricts the position to which a word or phrase can be moved, and is able to handle short and long distance reorderings in a unified way, and (ii) a joint sequence model for the translation and reordering probabilities which is more flexible than standard phrase-based MT. We observe statistically significant improvements in BLEU over Moses for German-to-English and Spanish-to-English tasks, and comparable results for a French-to-English task.
@inproceedings{durrani-etal-2011-joint,title={A Joint Sequence Translation Model with Integrated Reordering},author={Durrani, Nadir and Schmid, Helmut and Fraser, Alexander},booktitle={Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies},month=jun,year={2011},address={Portland, Oregon, USA},publisher={Association for Computational Linguistics},url={https://aclanthology.org/P11-1105},pages={1045--1054},area={Translation and Reordering},}
Comparing Two Techniques for Learning Transliteration Models Using a Parallel Corpus
Hassan Sajjad, Nadir Durrani, Helmut Schmid, and
1 more author
In Proceedings of 5th International Joint Conference on Natural Language Processing (IJCNLP) Nov 2011
We compare the use of an unsupervised transliteration mining method and a rulebased method to automatically extract lists of transliteration word pairs from a parallel corpus of Hindi/Urdu. We build joint source channel models on the automatically aligned orthographic transliteration units of the automatically extracted lists of transliteration pairs resulting in two transliteration systems. We compare our systems with three transliteration systems available on the web, and show that our systems have better performance. We perform an extensive analysis of the results of using both methods and show evidence that the unsupervised transliteration mining method is superior for applications requiring high recall transliteration lists, while the rule-based method is useful for obtaining high precision lists.
@inproceedings{sajjad-EtAl:2011:IJCNLP-2011,author={Sajjad, Hassan and Durrani, Nadir and Schmid, Helmut and Fraser, Alexander},booktitle={Proceedings of 5th International Joint Conference on Natural Language Processing (IJCNLP)},location={Chiang Mai, Thailand},month=nov,title={Comparing Two Techniques for Learning Transliteration Models Using a Parallel Corpus},year={2011},area={Transliteration},}
2010
Hindi-to-Urdu Machine Translation through Transliteration
Nadir Durrani, Hassan Sajjad, Alexander Fraser, and
1 more author
In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics Jul 2010
We present a novel approach to integrate transliteration into Hindi-to-Urdu statistical machine translation. We propose two probabilistic models, based on conditional and joint probability formulations, that are novel solutions to the problem. Our models consider both transliteration and translation when translating a particular Hindi word given the context whereas in previous work transliteration is only used for translating OOV (out-of-vocabulary) words. We use transliteration as a tool for disambiguation of Hindi homonyms which can be both translated or transliterated or transliterated differently based on different contexts. We obtain final BLEU scores of 19.35 (conditional probability model) and 19.00 (joint probability model) as compared to 14.30 for a baseline phrase-based system and 16.25 for a system which transliterates OOV words in the baseline system. This indicates that transliteration is useful for more than only translating OOV words for language pairs like Hindi-Urdu.
@inproceedings{durrani-etal-2010-hindi,title={{H}indi-to-{U}rdu Machine Translation through Transliteration},author={Durrani, Nadir and Sajjad, Hassan and Fraser, Alexander and Schmid, Helmut},booktitle={Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics},month=jul,year={2010},address={Uppsala, Sweden},publisher={Association for Computational Linguistics},url={https://aclanthology.org/P10-1048},pages={465--474},area={Transliteration},}
Urdu Word Segmentation
Nadir Durrani, and Sarmad Hussain
In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics Jun 2010
Word Segmentation is the foremost obligatory task in almost all the NLP applications where the initial phase requires tokenization of input into words. Urdu is amongst the Asian languages that face word segmentation challenge. However, unlike other Asian languages, word segmentation in Urdu not only has space omission errors but also space insertion errors. This paper discusses how orthographic and linguistic features in Urdu trigger these two problems. It also discusses the work that has been done to tokenize input text. We employ a hybrid solution that performs an n-gram ranking on top of rule based maximum matching heuristic. Our best technique gives an error detection of 85.8% and overall accuracy of 95.8%. Further issues and possible future directions are also discussed.
@inproceedings{durrani-hussain-2010-urdu,title={{U}rdu Word Segmentation},author={Durrani, Nadir and Hussain, Sarmad},booktitle={Human Language Technologies: The 2010 Annual Conference of the North {A}merican Chapter of the Association for Computational Linguistics},month=jun,year={2010},address={Los Angeles, California},publisher={Association for Computational Linguistics},url={https://aclanthology.org/N10-1077},pages={528--536},area={Word Segmentation},}
2008
PAN localization: A study on collation of languages from developing Asia
Collation of all written languages are defined in their dictionaries, developed over centuries, and are thus very representative of cultural tradition. However, though it is well understood in these cultures, it is not always thoroughly documented or well understood in the context of existing character encodings, especially the Unicode. This volume aims to address the complex algorithms needed for sorting out the words in sequence for a small but diverse set of scripts and languages chosen from developing Asian region. The set is chosen for the variety it exhibits and to show the challenges it poses to solve the collation puzzle. This work must be taken as an initial step towards addressing the collation of languages in the region as there is still more which can be said about collation of these languages, and there are many more languages which need to be documented. The data on different languages has been obtained from the dictionaries published in these languages, and through interacting with the PAN Localization project teams in relevant countries
@book{hussain2008pan,title={PAN localization: A study on collation of languages from developing Asia},author={Hussain, Sarmad and Durrani, Nadir},year={2008},publisher={Center for Research in Urdu Language Processing, National University of Computer and Emerging Sciences},area={Surveys},}
2007
Typology of word and automatic
word segmentation in Urdu text corpus
Word Segmentation is the foremost obligatory task in almost all the NLP applications where the initial phase requires tokenization of input into words. Urdu is amongst the Asian languages that face word segmentation challenge. However, unlike other Asian languages, word segmentation in Urdu not only has space omission errors but also space insertion errors. This paper discusses how orthographic and linguistic features in Urdu trigger these two problems. It also discusses the work that has been done to tokenize input text. We employ a hybrid solution that performs an n-gram ranking on top of rule based maximum matching heuristic. Our best technique gives an error detection of 85.8% and overall accuracy of 95.8%. Further issues and possible future directions are also discussed.
@mastersthesis{durrani:2007,author={Durrani, Nadir},title={{Typology of word and automatic
word segmentation in Urdu text corpus}},school={National University of Computer and Emerging Sciences},month=aug,year={2007},address={Lahore, Pakistan},area={Theses},}
2006
Urdu Domain Names
Sarmad Hussain, and Nadir Durrani
In 2006 IEEE International Multitopic Conference Aug 2006
With of international standards, including Unicode, CLDR, HTML, etc., it is now becoming increasingly possible to develop and deploy online content in local languages across the globe. However, a user is still required to write the domain name in Latin script to access this information on the Internet, which still a barrier for non-Latin script based language speakers. This paper overviews the emerging Internationalized Domain Name (IDN) standards being proposed by Internet Corporation for Assigned Names and Numbers (ICANN). The paper also discusses challenges for implementing IDN for Urdu and a possible solution which has been implemented and is currently deployed
@inproceedings{hussain2006urdu,title={Urdu Domain Names},author={Hussain, Sarmad and Durrani, Nadir},booktitle={2006 IEEE International Multitopic Conference},pages={299--304},year={2006},organization={IEEE},doi={10.1109/INMIC.2006.358181},area={Misc}}
Languages of the world exhibit tremendous diversity when it comes to defining their grammatical traits. Some of them act to be accusative while others behave ergative. The one’s those are classified as ergative often have in fact dual personalities, which means occasionally they show nominative-accusative patterns and sometimes they display ergative-absolutive system. These are termed as split-ergative languages, whereby syntactic and/or morphological ergative patterns are conditioned by the grammatical context, typically person or the tense/aspect of the verb. This paper provides an analysis of split ergativity in Urdu using standard mechanisms of structural case and agreement licensing.
@article{durrani2006system,title={System for Grammatical relations in Urdu},author={Durrani, Nadir},booktitle={12th Himalayan Language Symposium and 27th Annual Conference of Linguistic Society of Nepal},year={2006},area={Misc},address={Kathmandu, Nepal}}
2005
Pan localization: Survey of language computing in Asia
Sarmad Hussain, Nadir Durrani, and Sana Gul
Center for Research in Urdu Language Processing, Lahore, Pakistan. Aug 2005
@article{hussain2005pan,title={Pan localization: Survey of language computing in Asia},author={Hussain, Sarmad and Durrani, Nadir and Gul, Sana},journal={Center for Research in Urdu Language Processing, Lahore, Pakistan.},year={2005},area={Surveys},}
Syllabification of lao script for line breaking
Phonpasit Phissamay, and Nadir Durrani
Center for Research in Urdu Language Processing, Lahore, Pakistan. Aug 2005
@article{phissamay2004syllabification,title={Syllabification of lao script for line breaking},author={Phissamay, Phonpasit and Durrani, Nadir},journal={Center for Research in Urdu Language Processing, Lahore, Pakistan.},year={2005},area={Word Segmentation},}
Syllabification of lao script for line breakingCenter for Research in Urdu Language Processing, Lahore, Pakistan. Aug 2005