Mohamed Abdelaal, PhD.

In this paper, we present a new method for learned data cleaning. In contrast to existing methods, our method learns to clean data in the latent space. The main idea is that we (1) shape the latent space such that we know the area where clean data resides and (2) learn latent operators trained on error repair (Lopster) which shift erroneous data (e.g., table rows with noise, outliers, or missing values) in their latent representation back to a “clean” region, thus abstracting the complexities of the input domain. When formulating data cleaning as a simple shift operation in latent space, we can repair all types of errors using the same method which makes it more robust than other methods. Importantly, with our method, we can handle errors that are unseen during the training of our error repair model. We do not rely on an external error detection method as seen in the state-of-the-art, instead, we handle both detection and repair within the Lopster framework. In our evaluation, we show that our approach outperforms existing cleaning methods even when trained on only a subset of the errors that occur in the dirty data.

Numerous automated error detection tools for tabular data have been introduced. Nevertheless, these tools suffer from several shortcomings, encompassing the necessity for domain-specific expertise and substantial time requirements. To address these shortcomings, we introduce a novel error detection tool, denoted as SAGED, which leverage meta-learning principles. Specifically, SAGED exploits an ensemble of pre-trained models derived from historical datasets to facilitate error detection in new data with limited labeled instances. The method consists of two key phases: knowledge extraction and error detection. In the knowledge extraction phase, ML models are trained to distinguish erroneous instances within historical datasets, thereby accumulating valuable insights. In the error detection phase, pretrained base models, chosen through rigorous matching, generate a comprehensive feature vector based on predictions, facilitating the role of a meta-classifier in pinpointing errors efficiently. As a proof of concept, we conduct a comparative study of SAGED against ten state-of-the-art error detection tools, employing a set of 14 real-world datasets. The findings reveal the superior performance of SAGED in error detection tasks, with limited user intervention. SAGED’s promising results demonstrate its effectiveness and efficiency in real-world scenarios.

Nowadays, machine learning (ML) plays a vital role in many aspects of our daily life. In essence, building well-performing ML applications requires the provision of high-quality data throughout the entire life-cycle of such applications. Nevertheless, most of the real-world tabular data suffer from different types of discrepancies, such as missing values, outliers, duplicates, pattern violation, and inconsistencies. Such discrepancies typically emerge while collecting, transferring, storing, and/or integrating the data. To deal with these discrepancies, numerous data cleaning methods have been introduced. However, the majority of such methods broadly overlook the requirements imposed by downstream ML models. As a result, the potential of utilizing these data cleaning methods in ML pipelines is predominantly unrevealed. In this work, we introduce a comprehensive benchmark, called REIN, to thoroughly investigate the impact of data cleaning methods on various ML models. Through the benchmark, we provide answers to important research questions, e.g., where and whether data cleaning is a necessary step in ML pipelines. To this end, the benchmark examines 38 simple and advanced error detection and repair methods. To evaluate these methods, we utilized a wide collection of ML models trained on 14 publicly-available datasets covering different domains and encompassing realistic as well as synthetic error profiles.

Recently, point clouds have been efficiently utilized for medical imaging, modeling urban environments, and indoor modeling. In this realm, several mobile platforms, such as Google Tango and Apple ARKit, have been released leveraging 3D mapping, augmented reality, etc. In modeling applications, these modern mobile devices opened the door for crowd-sourcing point clouds to distribute the overhead of data collection. However, uploading these large points clouds from resources-constrained mobile devices to the back-end servers consumes excessive energy. Accordingly, participation rates in such crowd-sensing systems can be negatively influenced. To tackle this challenge, this paper introduces our ComNSense approach that dramatically reduces the energy consumption of processing and uploading point clouds. To this end, ComNSense reports only a set of extracted geometrical data to the servers. To optimize the geometry extraction, ComNSense leverages formal grammars which encode design-time knowledge, i.e. structural information. To demonstrate the effectiveness of ComNSense, we performed several experiments of collecting point clouds from two different buildings to extract the walls location, as a case study. We also assess the performance of ComNSense relative to a grammar-free method. The results showed a significant reduction of the energy consumption while achieving a comparable detection accuracy.