Thesis

„AI-assisted wood knots detection from historic timber structure imaging“ - Junquan Pan

Autor: Junquan Pan, betreut von Christian Ledig

This thesis presents an AI-based system for the automated detection of wood knots in historic timber structures, designed to support heritage conservation efforts. Traditional manual methods for assessing the condition of timber structures are often inefficient, error-prone and unsuitable for challenging environments. By integrating advanced machine learning and deep learning technologies, such as Detectron2 and YOLOv8, this work introduces a robust and systematic workflow to improve the accuracy and efficiency of timber defect analysis.

The research involves two main stages: segmentation to identify wood surfaces, and detection to locate and analyse wood knots. High-resolution datasets were meticulously collected from heritage sites, including the Dominican Church in Bamberg, as well as timber workshops, to ensure diverse and high-quality training data. These datasets were used to train and validate the proposed models, overcoming challenges such as varying lighting conditions, irregular wood textures and knot complexity.

The system not only facilitates accurate assessment of wood condition, but also contributes to the conservation of historic resources by minimising unnecessary material replacement. The ultimate goal is to provide conservators with a mobile application that integrates these AI-driven tools, enabling efficient and detailed in-situ analysis of wooden structures. This work highlights the transformative potential of AI in heritage conservation, bridging the gap between traditional techniques and modern computational methods.

Link to code

Link to thesis(14.3 MB)

Link to paper extending the thesis

„Towards Multimodal Deep Learning for Medical Image Analysis: Developing a Cross-Modality Data Augmentation Technique by Interpolating Modality-Specific Characteristics in Medical Images" - Julius Stutz

Autor: Julius Stutz, betreut von Sebastian Dörrich

This thesis explores the challenge of data scarcity in medical imaging for deep learning applications and presents the Cross-Modality Data Augmentation (CMDA) as a new approach. Medical imaging data is limited due to privacy concerns, ethical restrictions, and technical constraints, which makes model development substantially harder in this domain. CMDA addresses these challenges by translating images between modalities (medical imaging techniques) such as PET, MRI, and CT to enhance dataset diversity and improve model robustness.

CMDA consists of four augmentations tailored to modality-specific characteristics: color, artifacts, spatial resolution, and noise. It allows to adjust its settings appropriately for each use-case and integrates seamlessly into existing data augmentation pipelines. The method aims to synthesize new training samples that visually align with the target modality, potentially improving generalization in neural networks.

The approach was evaluated through quantitative experiments, comparing classification performance of models trained with CMDA and other augmentations. Results showed marginal improvements in some cases but noticeable performance drops in others. Qualitative experiments, however, demonstrated CMDA’s success in aligning images to target modalities, with two experiments showing an average alignment improvement of 23.5%.

Despite limitations in model generalization and robustness, CMDA demonstrates its potential in addressing cross-modality challenges, offering a foundation for future research in medical data augmentation and image translation.

Link to code

Link to thesis(18.7 MB)

‚‚Exploring and Evaluating Deep Hashing Methods within Vision Foundation Model Feature Spaces for Similarity Search and Privacy Preservation’’ - Peiyao Mao

Autor: Peiyao Mao, betreut von Francesco Di Salvo

This thesis investigates the efficacy of deep hashing methods applied to image embeddings derived from state-of-the-art Vision Transformer (ViT) models, focusing on both the semantic preservation be tween original image embeddings and hashed image embeddings and the strengthening of data privacy. The contribution of the experiment involves the transformation of image embeddings to hashed image embeddings using various deep supervised hashing methods, making sure the semantic similarities are preserved, and data privacy is enhanced. We innovatively apply Triplet Center Loss (TCL) in the domain of deep hashing, aiming to achieve both high performance and computational efficiency. By comparing various deep supervised hashing methods, including pairwise and triplet methods, the experiment try to understand and evaluate how different approaches perform under the same conditions. It provides insights into which methods are most effective in retaining important data characteristics after hashing. A key aspect of this research is the emphasis on privacy preservation. By converting raw image data into hashed forms, this work explores how advanced hashing techniques can obscure original data features, thereby enhancing privacy without substantially compromising the utility for tasks such as medical image retrieval. This dual focus addresses the critical challenge of using sensitive image data in environments where privacy concerns are important.

Link to code

Link to thesis(2.7 MB)

„Generative Data Augmentation in the Embedding Space of Vision Foundation Models to Address Long-Tailed Learning and Privacy Constraints” - David Elias Tafler

Autor: David Elias Tafler, betreut von Francesco Di Salvo

This thesis explores the potential of generative data augmentation in the embedding space of vision foundation models, aiming to address the challenges of long-tailed learning and privacy constraints. Our work leverages Conditional Variational Autoencoders (CVAEs) to enrich the representation space for underrepresented classes in highly imbalanced datasets and to enhance data privacy without compromising utility. We develop and assess methods that generate synthetic data embeddings conditioned on class labels, which both mimic the distribution of original data for privacy purposes and augment data for tail classes to balance datasets. Our methodology shows that embedding-based augmentation can effectively improve classification accuracy in long-tailed scenarios by increasing the diversity and volume of minor class samples. Additionally, we demonstrate that our approach can generate data that maintains privacy through effective anonymization of embeddings. The outcomes suggest that generative augmentation in embedding spaces of foundation models offers a promising avenue for enhancing model robustness and data security in practical applications. The findings have significant implications for deploying machine learning models in sensitive domains, where data imbalance and privacy are critical concerns.

Link to code

Link to thesis(8.8 MB)

Link to paper extending the thesis

„Unveiling CNN Layer Contributions: Application of Feature Visualization in Medical Image Classification Tasks” - Jonida Mukaj

Autor: Jonida Mukaj, betreut von Ines Rieger

This thesis explores the application of feature visualisations on medical datasets, specifically for skin cancer imaging, using pre-trained convolutional neural networks like AlexNet, VGG16, and ResNet50 to enhance model interpretability, and fine-tuning those models to the ISIC benchmark dataset. Comparative analysis of ISIC 2019 and 2020 datasets shows varying model strengths, with VGG16 leading in accuracy and ResNet50 generalizability. Feature visualizations reveal diagnostic patterns in skin cancer, aiding in understanding network decision-making, yet pose challenges in medical interpretation. The study underscores the importance of deep learning in medical imaging and suggests combining feature visualizations with other interpretability techniques for future advancements.

Link to thesis(17.2 MB)

„Human Activity Recognition via Deep Learning based on active exoskeleton data” - Christoph Zink

Autor: Christoph Zink, betreut von Prof. Dr. Christian Ledig

To optimize the behavior of exoskeletons with built-in motors, so-called active exoskeletons, real-time classification of the user's activity has become increasingly popular in recent years. However, no studies have evaluated the performance of different neural network architectures for this task in a real-world scenario where both the observed locations and the observed subjects are absent from the training data. To fill this gap, this study trained four architectures of neural networks and compared their performance on a self-recorded test set containing 10 new subjects at 5 previously unseen locations. In addition, a comparison was made with a model representing the standard approach prior to the advent of deep learning models in the field, to answer whether deep learning models consistently classify better in a robust manner.

The results indicate that deep learning models are overall well suited to this task, i.e. all neural networks outperformed the baseline method. Regarding the robustness of the models, it appears that the neural networks can generalize well beyond the single location and few subjects observed within the training data. This ability to robustly generalize appears to be strongly dependent on the overall amount of training data available, i.e. the models generalized rather poorly when applied to activities rarely observed within the training data.

The study resulted from a cooperation with the company German Bionic, an Augsburg-based producer of active exoskeletons intended for the relief of the lower back. For the study data coming from the IoT-connected device Cray-X was used, which is depicted in the attached picture.

Link to code

Link to theses(2.1 MB)

Benchmarking selected State-of-the-Art Baseline Neural Networks for 2D Biomedical Image Classification, Inspired by the MedMNIST v2 Framework"- Julius Brockmann

Author: Julius Brockmann, supervised by Sebastian Dörrich

This thesis examines the benchmarking of state-of-the-art baseline neural networks in the field of 2D biomedical image classification. Focusing on the effectiveness of deep learning models on high-quality medical databases, the study employs pre-trained baseline networks to establish benchmarks. The research investigates four convolutional neural
networks and one transformer-based architecture, exploring how changes in image resolution affect performance. The findings highlight the advanced capabilities of newer convolutional networks and demonstrate the effectiveness of transformer architectures for handling large datasets. Common misclassifications and their causes are also briefly analyzed, offering insights into potential areas for improvement in future studies.

Link to code

Link to thesis(8.4 MB)

"Development of a dataset and AI-based proof-of-concept algorithm for the classification of digitized whole slide images of gastric tissue"- Tom Hempel

Author: Tom Hempel, supervised by Prof. Dr. Christian Ledig

The thesis focuses on the development of a dataset and AI algorithms for classifying digitized whole slide images (WSIs) of gastric tissue. It details the creation and meticulous annotation of the dataset, which is crucial for effectively training the AI. The process involved gathering, anonymizing, and annotating a vast array of WSIs, aimed at building robust AI models that can accurately classify different regions of the stomach and identify inflammatory conditions.

Two AI models were developed, one for assessing gastric regions and another for inflammation detection, achieving high accuracy in areas like the antrum and corpus but facing challenges with intermediate regions due to dataset limitations and the specificity of training samples.

The challenges encountered during the dataset creation, such as data collection and the necessity for detailed annotation to ensure data integrity and privacy, highlight the complexity of this research.

The dataset and initial models serve as a foundation for further research by Philipp Andreas Höfling in his master thesis, aiming to refine these AI algorithms and enhance their utility in medical diagnostics.

Link to code

Link to thesis(5.1 MB)

"Development of an AI-based algorithm for the classification of gastric tissue for computational pathology"- Philipp Andreas Höfling

Author: Philipp Andreas Höfling, supervised by Prof. Dr. Christian Ledig

Computational pathology has significantly advanced in recent years, yet a notable gap exists in the specific area of gastric tissue research. To address this issue, this study focuses on developing AI algorithms specifically for classifying gastric tissue types and inflammation caused by gastritis.

Using two ResNet18 models, trained on annotated tiles from over 200 slides, high accuracy in both inflammation and tissue type classification has been achived. However, challenges remain, especially in generalizing inflammation detection across all types of gastritis.

While promising, further research is needed, including expanding datasets and refining annotations, to fully harness AI's potential in gastric tissue analysis.

Link to code

Link to thesis(7.7 MB)