Ivan Nikolov

A researcher bridging the fields of computer vision, graphics and human-computer interaction

Applied SfM - Capturing and Analysing 3D Surface Data

This PhD thesis looks at the problems associated with Structure from Motion 3D reconstruction for object and surface analysis and inspection. The use of 3D data has become an important part of many research fields, because of the additional information and versatility that it provides, compared to more traditional 2D data. Two main parts of the SfM processing pipeline are featured in the thesis, with additional sub-parts in each.Structure from Motion data capturing plays an important part in the success and quality of the produced 3D reconstructions. Special care needs to be taken in the way images are shot, the capturing setup, used cameras, as well as the processing done to the said images. The thesis contains a number of publications, which present benchmarks and analysis methods for determining the quality and robustness of the produced reconstructions. These benchmarks come with datasets and ground truths, which can be used by others to further the research. In addition, a capturing environment for producing synthetic images is presented, to cut the preparation time and help users that start in the field. The image capturing can be done both terrestrially and in the air, using unmanned aerial vehicles (UAVs), so the thesis presents also a straightforward approach for localization of drones.Because it relies on 2D images, as input data for creating 3D reconstructions, SfM does not contain a straightforward way to determine the real life scale of the captured objects or surfaces. This is why this thesis presents two ways to determine both the reconstructions’ absolute scale and its uncertainty, by using additional sensors and fusing their data with the SfM data. Both position sensors like GPS and distance sensors like LiDAR are used for this and processing pipelines are presented for each.
The produced 3D surface data needs to be also analysed. First, for removing any noise and geometrical errors introduced by SfM and later for determining the quality of the reconstructions and if they can be used for their intended tasks. An automatic method for separating noise and real surface roughness is presented in the thesis, together with a comparison between SfM and microscopy reconstructions, for determining the accuracy of the captured data.

About

My name is Ivan Nikolov. I was born in the little town of Vratsa, Bulgaria. I got my bachelor's degree from the Technical University of Sofia in the field of Computer Engineering. I moved to Denmark, together with my wife to study at Aalborg University. There I completed my master's degree in Vision, Graphics, and Interactive Systems.
I got my Computer Engineering Ph.D. in 2020 and worked as a postdoc through 2021 with self-supervised learning for anomaly detection and harbor safety.
In 2022 I have become an assistant professor in Computer Graphics with a primary interest in XR and developing synthetic environments for deep learning.I have interests in deep learning, computer graphics, interactive systems, VR and AR, as well as game development. Statistics and data visualization are my second nature.

Contact

I am open to new opportunities for work and collaboration, as well as consultancy.

Copenhagen, DenmarkPhone: 42688859
Email: inikolov17(at)gmail.com

Teaching

I am currently teaching classes in:
- Programming 101
- Unity Game Development
- Shader Programming
- Virtual and Augmented Reality Programming (XR)
- 2D Pixel Art

My Pixel Art workshop courses are avaible online directly on Youtube

Portfolio Examples

Examples of some of my work that does not fit in the other categories.
For an in-depth PDF version of the portfolio click on the button below.

Manipulating CAD models in VR using hand tracking and gestures

Visualization of CAD models in VR and AR

VR simulation of touching "physical" objects

Impainting and segmentation of children's books characters

3D printing and CnC examples

Arduino, Raspberry Pi and Jetson prototypes using IMUs, LiDARs, Cameras, GPS, drone, etc.

Virtual Reality headset prototypes and robot arm experiments

Initial Development of "Infection Defender": A Children's Educational Game for Pandemic Prevention Measurements (2021)

Using serious games to communicate and teach complex topics to children and adolescence has gained a lot of popularity, especially in the medical fields. The spread of COVID-19 and the need to change everyday habits has opened up the need to teach children the required precautions for limiting the spread of potential pandemics. In this paper we present the initial development of the game Infection Defender, promoting children’s awareness of closing schools, social distancing, testing and hospitalization for fighting the spread of infectious diseases in Denmark. These activities are given in the hands of children, between 10 and 13 years old, and the goal of the game is achieving a balanced response to a possible infectious decease outbreak. We present the game, its design considerations and how the learning objectives are integrated into it. An analysis of the game by pedagogical workers is made and a pilot test is carried out assessing children’s reactions to it. Initial positive feedback shows that the game sparks interest and discussion in children and can be used as part of the study curriculum to help children understand the need for certain measurements

Generating Synthetic Point Clouds of Sewer Networks: An Initial Investigation (2020)

Automatic robot inspections of sewer systems are progressively becoming more used for extending the lifetime of sewers and lowering the costs of maintenance. These automatic systems rely on machine learning and the acquisition of varied training data is therefore necessary. Capturing such data can be a costly and time consuming process. This paper proposes a system for generation and acquisition of synthetic training data from sewer systems. The system utilizes Structured Domain Randomization (SDR) for the generation of the sewer systems and an approximated model of a Pico Flexx Time-of-Flight camera for capturing depth and point cloud data from the generated sewer network. We evaluate the proposed system by comparing its output to ground truth data acquired from a Pico Flexx sensor in sewer pipes. We demonstrate that on average our system provides an absolute error of 5.78 ± 8.92 and 7.58 ± 8.68 mm, between data captured from real life and our proposed system, for two different scenarios. These results prove satisfactory for capturing training data.

Preliminary Study on the use of Off-the-Shelf VR Controllers for Vibrotactile Differentiation of Levels of Roughness on Meshes (2020)

With the introduction of new specialized hardware, Virtual Reality (VR) has gained more and more popularity in recent years. VR is particularly immersive if suitable auditory and haptic feedback is provided to users. Many proposed forms of haptic feedback require custom hardware components that are often bulky, costly, and/or require lengthy setup times. We explored the possibility of using the built-in vibrotactile feedback of HTC Vive controllers to simulate the sensation of interacting with surfaces with varying degrees of roughness. We conducted initial testing on the proposed system, which shows promising results as users could accurately and within short time discern the amount of roughness of 3D models based on the vibrotactile feedback alone.

Testing VR headset cameras for capturing written content (2020)

Virtual reality (VR) has become an important tool for providing immersive and collaborative teleprensence experiences. The technology has the possibility of bringing people in different geographical places together in an immersive way and makes sharing knowledge, ideas and experience easier. An important evolution in the immersive experience is the introduction of items from the real world into the virtual environment and the creation of a mixed reality experience. We present a pilot study in the use of the built-in head mounted display (HMD) cameras, for capturing parts of the environment and presenting them in VR. Particular emphasis is put on capturing physical written content on flat surfaces and visualizing it in VR for sharing with other users. We test the HTC Vive and Valve Index cameras and compare them to two external solutions by applying optical character recognition (OCR) on captured...

Preliminary Study on the Influence of Visual Cues, Transitional Environments and Tactile Augmentation on the Perception of Scale in VR (2020)

Virtual reality (VR) is being used more and more as a way to easily visualize and share ideas, as well as a step in product designing. The initial study presented in this paper is part of a project for getting the general public involved in the design of new busses for Northern Jutland in Denmark, by using VR visualization. An important part of VR visualizations is the correct understanding of scale. Studies show that the perception of scale in VR undergoes compression compared to the real world. In this paper, we test how additional visual cues, transitional environments, and tactile augmentation in a VR environment can help with the perception of scale. We show that familiar visual cues can help with the perception of scale, but do not remove the compression of perception. We can further mitigate the problem by introducing transitional environments and tactile augmentation, but transitional environments provide a better perception of scale than tactile augmentation.

Interactive Environment for Testing SfM Image Capture Configurations (2019)

In recent years 3D reconstruction has become an important part of the manufacturing industry, product design,digital cultural heritage preservation, etc. Structure from Motion (SfM) is widely adopted, since it does not require specialized hardware and easily scales with the size of the scanned object. However, one of the drawbacks of SfM is the initial time and resource investment required for setting up a proper scanning environment and equipment, such as proper lighting and camera, number of images, the need of green screen, etc, as well as to determine if an object can be scanned successfully. This is why we propose a simple solution for approximating the whole capturing process. This way users can test fast and effortlessly different capturing setups.We introduce a visual indicator on how much of the scanned object is captured with each image in our environment, giving users a better idea of how many images would be needed. We compare the 3D reconstructioncreated from images from our solution, with ones ated from rendered images using Autodesk Maya and V-Ray. We demonstrate that we provide comparable reconstruction accuracy at a fraction of the time.

Inside - Outside Model Viewing
A Low-cost Hybrid Approach to Visualization and Demonstration of 3D Models (2016)

Visualization of large scale 3D models has become an important part of the development cycle in many fields like building design, machine design, construction and many more. Whether for design communication or demonstration, it is necessary for the viewers to fully understand the different components of the model, their proportions compared to each other and the overall design. A variety of augmented reality (AR) applications have been created for overall visualization of large scale models. For tours inside 3D renderings of models many immersive virtual reality (VR) applications exist. Both types of applications have their limitation, omitting either important details in the AR case or the full picture in the case of VR. This paper presents a low-cost way to demonstrate models using a hybrid virtual environment system (HVE), combining virtual reality and augmented reality visualization. The solution is built using a fully occlusive head mounted display (HMD), together with off-the-shelf web cameras and a game controller for interaction.

Modelling of Clouds from a Hemispherical Image (2014)

This paper presents an image-based method for modelling clouds. Unlike previous image-based approaches, a hemispherical photograph is used as input, enabling to consider an entire sky instead of merely a portion. Our method computes the intensity and opacity of the clouds from the photograph. For this purpose, beforehand, the sun illumination is filtered, the pixels are classified between cloud and sky pixels, and the sky behind the clouds is reconstructed. After having been smoothed, the intensity of the clouds is used to create vertices on a hemisphere, and their radius coordinate is modulated by the intensity value of the corresponding pixel. Finally, the mesh is generated by triangulation of the vertices. Additionally, the use of the opacity of the clouds to simulate their transparency and render them is proposed. The results show that our method can be used to produce a realistic full sky populated...

Seasons in Drift: A Long-Term Thermal Imaging Dataset for Studying Concept Drift (2021)

The time dimension of datasets and the long-term performance of machine learning models have received little attention. With extended deployments in the wild, models are bound to encounter novel scenarios and concept drift that cannot be accounted for during development and training. In order for long-term patterns and cycles to appear in datasets, the datasets must cover long periods of time. Since this is rarely the case, it is difficult to explore how computer vision algorithms cope with changes in data distribution occurring across long-term cycles such as seasons. Video surveillance is an application area clearly affected by concept drift. For this reason, we publish the Long-term Thermal Drift (LTD) dataset. LTD consists of thermal surveillance imaging from a single location across 8 months. Along with thermal images we provide relevant metadata such as weather, the day/night cycle, and scene activity. In this paper, we use the metadata for in-depth analysis of the causal and correlational relationships between environmental variables and the performance of selected computer vision algorithms used for anomaly and object detection. Long-term performance is shown to be most correlated with temperature, humidity, the day/night cycle, and scene activity level. This suggests that the coverage of these variables should be prioritised when building datasets for similar applications. As a baseline, we propose to mitigate the impact of concept drift by first detecting points in time where drift occurs. At this point, we collect additional data that is used to retraining the models. This improves later performance by an average of 25% across all tested algorithms.

Sewer Defect Classification using Synthetic Point Clouds (2021)

Sewer pipes are currently manually inspected by trained inspectors, making the process prone to human errors, which can be potentially critical. There is therefore a great research and industry interest in automating the sewer inspection process. Previous research have been focused on working with 2D image data, similar to how inspections are currently conducted. There is, however, a clear potential for utilizing recent advances within 3D computer vision for this task. In this paper we investigate the feasibility of applying two modern deep learning methods, DGCNN and PointNet, on a new publicly available sewer point cloud dataset. As point cloud data from real sewers is scarce, we investigate using synthetic data to bootstrap the training process. We investigate four data scenarios, and find that training on synthetic data and fine-tune on real data gives the best results, increasing the metrics by 6-10 percentage points for the best model.

Quantifying Wind Turbine Blade Surface Roughness using Sandpaper Grit Sizes: An Initial Exploration (2021)

Surface inspection of wind turbine blades is a necessary step, to ensure longevity and sustained high energy output. The detection of accumulation of damages and increased surface roughness of in-use blades, is one of the main objectives of inspections in the wind energy industry. Creating 3D scans of the leading edges of blade surfaces has been more and more used for capturing the roughness profile of blades. An important part in analysing these surface 3D scans is the standardization of the captured data across different blade surfaces, types and sizes. In this paper we propose an initial exploration of using sandpaper grit sizes to provide this standardization. Sandpaper has been widely used for approximating different levels of blade surface roughness and its standardized nature can be used to easily describe and compare blade surfaces. We reconstruct a number of different sandpaper grit sizes-from coarser P40 to a finer P180. We extract a number of 3D surface features from them and use them to train a random forest classification method. This method is then used to segment the surfaces of wind turbine blades in areas of different surface roughness. We test our proposed solution on a variety of blade surfaces-from smooth to course and damaged and show that it manages to classify them depending on their roughness

How Image Capturing Setups Influence the Quality of SfM
Reconstructions for Wind Turbine Blade Inspection (2020)

Increased leading edge roughness (LER) is one of the main causes for wind turbine blade performance degradation. To ensure a consistently high energy output, the surface erosion of wind turbine blades, needs to be monitored regularly, so preventive measurements can be done. Capturing 3D surface data is becoming a more and more popular way to extract and quantify roughness on a micro level. In this paper we want to test the possibility of using Structure from Motion (SfM) 3D reconstructions for extracting surface roughness information from wind turbine blades. We test various capturing scenarios with varying horizontal and vertical image overlap, as well as varying distances to the blade, using a real blade in outdoor conditions. We analyze the quality of the reconstructions and provide a benchmark, as well as guidelines to what are the best possible capturing conditions for ensuring a high quality and noise free 3D surface results. All data from the experiments is made publicly accessible.

Rough or Noisy? Metrics for Noise Estimation in SfM Reconstructions (2020)

Structure from Motion (SfM) can produce highly detailed 3D reconstructions, but distinguishing real surface roughness from reconstruction noise and geometric inaccuracies has always been a difficult problem to solve. Existing SfM commercial solutions achieve noise removal by a combination of aggressive global smoothing and the reconstructed texture for smaller details, which is a subpar solution when the results are used for surface inspection. Other noise estimation and removal algorithms do not take advantage of all the additional data connected with SfM. We propose a number of geometrical and statistical metrics for noise assessment, based on both the reconstructed object and the capturing camera setup. We test the correlation of each of the metrics to the presence of noise on reconstructed surfaces and demonstrate that classical supervised learning methods, trained with these metrics can be used to distinguish between noise and roughness with an accuracy above 85%, with an additional 5–6% performance coming from the capturing setup metrics...

Calculating Absolute Scale and Scale Uncertainty for SfM Using Distance Sensor Measurements (2020)

Capturing details of objects and surfaces using structure from motion (SfM) 3D reconstruction has become an important part of data gathering in geomapping, medicine, cultural heritage, and the energy and production industries. One inherent problem with SfM, due to its reliance on 2D images, is the ambiguity of the reconstruction's scale. Absolute scale can be calculated by using the data from additional sensors. This chapter demonstrates how distance sensors can be used to calculate the scale of a reconstructed object. In addition, the authors demonstrate that the uncertainty of the calculated scale can be computed and how it depends on the precision of the used sensors. The provided methods are straightforward and easy to integrate into the workflow of commercial SfM solutions.

High-Resolution Structure-from-Motion for Quantitative Measurement of Leading-Edge Roughness (2020)

Over time, erosion of the leading edge of wind turbine blades increases the leading-edge roughness (LER). This may reduce the aerodynamic performance of the blade and hence the annual energy production of the wind turbine. As early detection is key for cost-effective maintenance, inspection methods are needed to quantify the LER of the blade. The aim of this proof-of-principle study is to determine whether high-resolution Structure-from-Motion (SfM) has the sufficient resolution and accuracy for quantitative inspection of LER. SfM provides 3D reconstruction of an object geometry using overlapping images of the object acquired with an RGB camera. Using information of the camera positions and orientations, absolute scale of the reconstruction can be achieved. Combined with a UAV platform, SfM has the potential for remote blade inspections with a reduced downtime. The tip of a decommissioned blade with an artificially enhanced erosion was used for the measurements...

Performance Characterization of Absolute Scale Computation for 3D Structure from Motion Reconstruction (2019)

Structure from Motion (SfM) 3D reconstruction of objects and environments has become a go-to process, when detailed digitization and visualization is needed in the energy and production industry, medicine, culture and media. A successful reconstruction must properly capture the 3D information and it must scale everything to the correct scale. SfM has an inherent ambivalence to the scale of the scanned objects, so additional data is needed. In this paper we propose a lightweight solution for computation of absolute scale of 3D reconstructions by using only a real-time kinematic (RTK) GPS, in comparison to other custom solutions, which require multiple sensor fusion. Additionally, our solution estimates the noise sensitivity of the calculated scale, depending on the precision of the positioning sensor. We first test our solution with synthetic data to find how the results depend on changes to the capturing setup...

LiDAR-based 2D localization and mapping system using elliptical distance correction models for UAV wind turbine blade inspection (2017)

The wind energy sector faces a constant need for annual inspections of wind turbine blades for damage, erosion and cracks. These inspections are an important part of the wind turbine life cycle and can be very costly and hazardous to specialists. This has led to the use of automated drone inspections and the need for accurate, robust and inexpensive systems for localization of drones relative to the wing. Due to the lack of visual and geometrical features on the wind turbine blade, conventional SLAM algorithms have a limited use. We propose a cost-effective, easy to implement and extend system for on-site outdoor localization and mapping in low feature environment using the inexpensive RPLIDAR and an 9-DOF IMU. Our algorithm geometrically simplifies the wind turbine blade 2D cross-section to an elliptical model and uses it for distance and shape correction...

Benchmarking Close-Range Structure from Motion 3D Reconstruction Software under Varying Capturing Conditions (2016)

Structure from Motion 3D reconstruction has become widely used in recent years in a number of fields such as industrial surface inspection, archeology, cultural heritage preservation and geomapping. A number of software solutions have been released using variations of this technique. In this paper we analyse the state of the art of these software applications, by comparing the resultant 3D meshes qualitatively and quantitatively. We propose a number of testing scenarios using different lighting conditions, camera positions and image acquisition methods for the best in-depth analysis and discuss the results, the overall performance and the problems present in each software. We employ distance and roughness metrics for evaluating the final reconstruction results.

Circular Hough Transform and Local Circularity Measure for Weight Estimation of a
Graph-Cut based Wood Stack Measurement (2015)

One of the time consuming tasks in the timber industry is the manually measurement of features of wood stacks. Such features include, but are not limited to, the number of the logs in a stack, their diameters distribution, and their volumes. Computer vision techniques have recently been used for solving this real-world industrial application. Such techniques are facing many challenges as the task is usually performed in outdoor, uncontrolled, environments. Furthermore, the logs can vary in texture and they can be occluded by different obstacles. These all make the segmentation of the wood logs a difficult task. Graph-cut has shown to be good enough for such a segmentation. However, it is hard to find proper graph weights. This is exactly the contribution of this paper to propose a method for setting the weights of the graph. To do so, we use Circular Hough Transform (CHT) for obtaining information about the foreand background regions of a stack image, and then use this together with a Local Circularity Measure (LCM) to modify the weights of the graph to segment the wood logs from the rest of the image...

UNDER CONSTRUCTION

IMAGE AND 3D MODELS DATASETS

Long-term Thermal Drift Dataset

Thermal Mannequin Fall Image Dataset

GGG-BenchmarkSfM: Dataset for Benchmarking Close-range SfM Software Performance under Varying Capturing Conditions

GGG-BenchmarkSfM- Secondary: Secondary Dataset for Benchmarking Close-range SfM Software Performance Captured with Different Cameras

GGG-PositioningSfMScale: Dataset for Testing Scaling and Uncertainty Propagation Through Camera Positioning Data

Wind Turbine Blade Surfaces Dataset

Sandpaper Wind Turbine Blade Benchmark Dataset

GGG - Rough or Noisy? Metrics for Noise Detection in SfM Reconstructions

An educational game, made for explaining the government pandemic response of closing and opening infrastructure for limiting infection spread

An educational game, made to put people in the shoes of a law maker at a time of pandemic - choosing what to close and when, so people can get through it

A Unity based mesh annotation tool

Resource Hunting Game - Game Jam Prototype

Unity Raycast Raytracing