publications

2024

1. ICML

   

   STREAM: Embodied Reasoning through Code Generation

   Daniil Cherniavskii, Phillip Lippe, Andrii Zadaianchuk, and 1 more author

   In Multi-modal Foundation Model meets Embodied AI Workshop @ ICML, 2024

   Abs HTML

   

   Recent advancements in the reasoning and code generation abilities of Large Language Models (LLMs) have provided new perspectives on Embodied AI tasks, enhancing planning for both high-level control problems and low-level manipulation. However, efficiently informing the embodied agent about the environment in a concise and task-specific manner remains a challenge. Inspired by modular visual reasoning, we propose a novel approach that utilizes code generation to ground the planner in the environmental context and enable reasoning about past agent experiences. Our modular framework allows the code-generating LLM to extract and aggregate information from relevant observations via API calls to image understanding models, including flexible VLMs. To evaluate our approach, we choose Embodied Question Answering (EQA) as a target task and develop a procedure for synthetic data collection by utilizing the ground truth states of a simulator. Our framework demonstrates notable improvements over baseline methods.

2. NeurIPS

   

   Intrinsic dimension estimation for robust detection of ai-generated texts

   Eduard Tulchinskii, Kristian Kuznetsov, Laida Kushnareva, and 5 more authors

   Advances in Neural Information Processing Systems, 2024

   Abs arXiv Code

   

   Rapidly increasing quality of AI-generated content makes it difficult to distinguish between human and AI-generated texts, which may lead to undesirable consequences for society. Therefore, it becomes increasingly important to study the properties of human texts that are invariant over different text domains and varying proficiency of human writers, can be easily calculated for any language, and can robustly separate natural and AI-generated texts regardless of the generation model and sampling method. In this work, we propose such an invariant for human-written texts, namely the intrinsic dimensionality of the manifold underlying the set of embeddings for a given text sample. We show that the average intrinsic dimensionality of fluent texts in a natural language is hovering around the value 9 for several alphabet-based languages and around 7 for Chinese, while the average intrinsic dimensionality of AI-generated texts for each language is ≈1.5 lower, with a clear statistical separation between human-generated and AI-generated distributions. This property allows us to build a score-based artificial text detector. The proposed detector’s accuracy is stable over text domains, generator models, and human writer proficiency levels, outperforming SOTA detectors in model-agnostic and cross-domain scenarios by a significant margin.

2023

1. ICLR

   

   Learning topology-preserving data representations

   Ilya Trofimov, Daniil Cherniavskii, Eduard Tulchinskii, and 3 more authors

   In ICLR 2023 International Conference on Learning Representations, 2023

   Abs arXiv Code Website

   

   We propose a method for learning topology-preserving data representations (dimen- sionality reduction). The method aims to provide topological similarity between the data manifold and its latent representation via enforcing the similarity in topo- logical features (clusters, loops, 2D voids, etc.) and their localization. The core of the method is the minimization of the Representation Topology Divergence (RTD) between original high-dimensional data and low-dimensional representation in latent space. RTD minimization provides closeness in topological features with strong theoretical guarantees. We develop a scheme for RTD differentiation and apply it as a loss term for the autoencoder. The proposed method “RTD-AE” better preserves the global structure and topology of the data manifold than state-of-the- art competitors as measured by linear correlation, triplet distance ranking accuracy, and Wasserstein distance between persistence barcodes.

2022

1. EMNLP

   

   Acceptability Judgements via Examining the Topology of Attention Maps

   Daniil Cherniavskii, Eduard Tulchinskii, Vladislav Mikhailov, and 7 more authors

   In Findings of the Association for Computational Linguistics: EMNLP 2022, 2022

   Abs arXiv Code

   

   The role of the attention mechanism in encoding linguistic knowledge has received special interest in NLP. However, the ability of the attention heads to judge the grammatical acceptability of a sentence has been underexplored. This paper approaches the paradigm of acceptability judgments with topological data analysis (TDA), showing that the geometric properties of the attention graph can be efficiently exploited for two standard practices in linguistics: binary judgments and linguistic minimal pairs. Topological features enhance the BERT-based acceptability classifier scores by 8%-24% on CoLA in three languages (English, Italian, and Swedish). By revealing the topological discrepancy between attention maps of minimal pairs, we achieve the human-level performance on the BLiMP benchmark, outperforming nine statistical and Transformer LM baselines. At the same time, TDA provides the foundation for analyzing the linguistic functions of attention heads and interpreting the correspondence between the graph features and grammatical phenomena.

2021

1. EMNLP

   

   Artificial Text Detection via Examining the Topology of Attention Maps

   Laida Kushnareva^*, Daniil Cherniavskii^*, Vladislav Mikhailov^*, and 6 more authors

   In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021

   *equal contribution

   Abs arXiv Code

   

   The impressive capabilities of recent generative models to create texts that are challenging to distinguish from the human-written ones can be misused for generating fake news, product reviews, and even abusive content. Despite the prominent performance of existing methods for artificial text detection, they still lack interpretability and robustness towards unseen models. To this end, we propose three novel types of interpretable topological features for this task based on Topological Data Analysis (TDA) which is currently understudied in the field of NLP. We empirically show that the features derived from the BERT model outperform count- and neural-based baselines up to 10% on three common datasets, and tend to be the most robust towards unseen GPT-style generation models as opposed to existing methods. The probing analysis of the features reveals their sensitivity to the surface and syntactic properties. The results demonstrate that TDA is a promising line with respect to NLP tasks, specifically the ones that incorporate surface and structural information.