Resarch interest: Computer Vision, Robotics, Machine Learning

Visual SLAM

ROVO: Robust Omnidirectional Visual Odometry

oem In this paper we propose a robust visual odometry system for a wide-baseline camera rig with wide field-of-view (FOV) fisheye lenses, which provides full omnidirectional stereo observations of the environment. For more robust and accurate ego-motion estimation we adds three components to the standard VO pipelines, 1) the hybrid projection model for improved feature matching, 2) multi-view p3p ransanc algorithm for pose estimation, and 3) online update of rig extrinsic parameters. The proposed system is extensively evaluated with synthetic datasets with ground-truth and real sequences of highly dynamic environment, and its superior performance is demonstrated.
  • Hochang Seok, Jongwoo Lim*, “ROVO: Robust Omnidirectional Visual Odometry for Wide-baseline Wide-FOV Camera Systems,” in ICRA 2019(accepted) [link]

Online Environment Mapping (Visual SLAM)

oem Building the map of environment for localization and navigation is critical for scene understanding and robot operation. We propose a metric-topological mapping which holds the benefits of both metric maps and topological maps.
  • Jongwoo Lim*, jan-Michael Frahm, Marc Pollefeys, “Online Environment Mapping using Metric-topological Maps,” in IJRR, Vol. 31 Issue 12, Page 1394-1408, October 2012 [link]
    • Earlier version: Online Environment Mapping, in CVPR 2011 [link] [pdf]

Real-Time 6-DOF Monocular Visual SLAM in a Large-Scale Environment

realtimeslam Real-time approach for monocular visual simultaneous localization and mapping (SLAM) within a large-scale environment is proposed. From a monocular video sequence, the proposed method continuously computes the current 6-DOF camera pose and 3D landmarks position. The proposed method successfully builds consistent maps from challenging outdoor sequences using a monocular camera as the only sensor, while existing approaches have utilized additional structural information such as camera height from the ground.
  • Hyon Lim, Jongwoo Lim, H. Jin Kim, “Real-Time 6-DOF Monocular Visual SLAM in a Large-Scale Environment,” in ICRA 2014 [link]
    • Hyon Lim, Jongwoo Lim, H. Jin Kim, Online 3D Reconstruction and 6-DoF Pose Estimation for RGB-D Sensors, in ECCV 2014 [link]

Camera Motion Estimation

triquat linesfm Points are commonly used for structure from motion and ego-motion estimation. We investigated more robust and fast ways to use line features for motion estimation of a stereo camera rig.
  • Vivek Pradeep, Jongwoo Lim, “Egomotion Estimation Using Assorted Features,” in IJCV, Vol. 98, Issue 2, Page 202-216, 2012 [link]
    • Earlier version: Egomotion using Assorted Features, in CVPR 2010 [link] [pdf]
  • Manmohan Chandraker, Jongwoo Lim, David Kriegman, “Moving in Stereo: Efficient Structure and Motion Using Lines,” in ICCV 2009 [link] [pdf]

Visual Inertial Odometry

Visual Inertial Odometry Using Coupled Nonlinear Optimization

vio Visual inertial odometry (VIO) gained lots of interest recently for efficient and accurate ego-motion estimation of robots and automobiles. With a monocular camera and an inertial measurement unit (IMU) rigidly attached, VIO aims to estimate the 3D pose trajectory of the device in a global metric space. We propose a novel visual inertial odometry algorithm which directly optimizes the camera poses with noisy IMU data and visual feature locations.
  • Euntae Hong, Jongwoo Lim, “Visual inertial odometry using coupled nonlinear optimization,” in IROS 2017, [link]

Depth Estimation

Robust stereo matching using adaptive random walk with restart algorithm

robuststereo In this paper, we propose a robust dense stereo reconstruction algorithm using a random walk with restart. The pixel-wise matching costs are aggregated into superpixels and the modified random walk with restart algorithm updates the matching cost for all possible disparities between the superpixels.
  • Sehyung Lee, Jin Han Lee, Jongwoo Lim, Il Hong Suh, “Robust Stereo Matching using Adaptive Random Walk with Restart Algorithm,” in Image and Vision Computing 2015 [link]

Visual Tracking

Visual Tracking Benchmark

benchmark13 For decades many visual trackers have been proposed, but there was little effort to quantitatively measure and compare their performance. In this work we provide a dataset which contains common test videos with hand-labeled groundtruth. The tracker library with standardized interface for massive evaluation enables the researchers to easily test and compare their trackers with the state-of-the-art trackers.
  • Yi Wu, Jonbwoo Lim*, Ming-Hsuan Yang, “Online Object Tracking: A Benchmark,” in CVPR 2013, [link] [pdf] [project page , code]

Tracking Persons-of-Interest via Adaptive Discriminative Features

trackingpoi Multi-face tracking in unconstrained videos is a challenging problem as faces of one person often appear drastically different in multiple shots due to significant variations in scale, pose, expression, illumination, and make-up. Low- level features used in existing multi-target tracking methods are not effective for identifying faces with such large appearance variations. In this paper, we tackle this problem by learning discriminative, video-specific face features using convolutional neural networks (CNNs).
  • Shun Zhang, Yihong Gong, Jia-Bin Huang, Jongwoo Lim, Jinjun Wang, Narendra Ahuja, Ming-Hsuan Yang, “Tracking Persons-of-Interest via Adaptive Discriminative Features,” in ECCV 2016, [link]

Deep Learning

DFT-based Transformation Invariant Pooling Layer for Visual Classification

vioWe propose a DFT based pooling layer for convolutional neural networks. The proposed DFT magnitude pooling satisfies translation invariance and shape preserving properties. It pools DFT magnitude of last convolution feature map based on shift theorem. Convolutional neural networks with the proposed method improve the performance of various visual classification tasks. We validate the ability of the transformation invariance by sufficient experiments of the paper.
  • Jongbin Ryu, Ming-Hsuan Yang, Jongwoo Lim*, “DFT-based Transformation Invariant Pooling Layer for Visual Classification,” in ECCV 2018, p. 84-99 [project]

Research Funds

  • [2017.09 - 2018.05] 비전 모델 기반 공간 상황 인지 원천기술 연구 (한국연구재단)
  • [2017.09 - 2018.05] 인간 수준의 종합적 비디오 이해를 통한 상황인지 및 예측 원천기술 연구 (한국연구재단)
  • [2017.09 - 2017.12] 로봇에 적용 가능한 6DOF SLAM 기술 개발 (LG전자)
  • [2017.07 - 2018.12] 컴퓨터 비전 영상인식 소프트웨어에 관한 개발 (아마노코리아)
  • [2017.06 - 2017.12] 다양한 실제 환경에서 활용 가능한 AR 실용화 원천 기술 개발 (정보통신기술진흥센터)
  • [2017.05 - 2017.12] 광각 카메라 영상 기반 수직 물체 감지 알고리즘 개발 (현대모비스)
  • [2017.03 - 2018.02] 영상 기반 동적 환경의 기하학 및 의미론적 장면 이해 방법론 개발 (한국연구재단)
  • [2017.01 - 2017.12] AVM 카메라를 이용한 3D 주차장 환경 모델링 및 차량 위치 인식 (LG전자)
  • [2016.09 - 2018.08] 도심의 혼잡한 환경에서의 자율 주행을 위한 전방향 비전 기반 지능형 상황 인식 기술 (삼성전자)
  • [2016.04 - 2016.12] Visual-Inertial Odometry for Smart Phones (LG전자)
  • [2016.04 - 2016.11] AVM 영상을 이용한 주차공간 인식 (LG전자)
  • [2016.04 - 2016.11] 스테레오 카메라를 이용한 노면 상황 인식 (LG전자)
  • [2013.03 - 2019.03] (BK21 PLUS)지능형 소프트웨어 전문인력 양성 (한국연구재단)
  • [2016.03 - 2016.08] Stereo Camera를 이용한 협로 및 전방 구조물 인식 (현대엔지비)
  • [2015.06 - 2018.03] 미술품 원색 기록 보존 및 복원을 위한 Multi-spectral 이미징 기술 개발(한국콘텐츠진흥원)
  • [2015.10 - 2016.03] Depth sensor-based Visual SLAM (SKT)
  • [2015.05 - 2015.10] 주변 환경 감지를 통한 장애물 회피 경로 생성 알고리즘 (현대엔지비)
  • [2015.04 - 2015.11] 스테레오 카메라를 이용한 주변환경 인식 (LG전자)
  • [2015.04 - 2015.12] Realtime SLAM 원천 기술 개발 (LG전자)
  • [2014.11 - 2017.04] 웨어러블 디바이스를 위한 광역/실시간 시각기반 위치인식 기술 (한국연구재단)
  • [2014.04 - 2015.02] 예지형 시각지능 원천 기술 개발 (미래부)
  • [2014.04 - 2014.11] Online Stereo-camera Calibration (LG전자)
  • [2014.01 - ~] C-ITRC <>
  • [2013.09 - 2016.08] 다시점 블랙박스 영상을 이용한 교통사고현장 3차원 재구성 기술개발 (미래부)
  • [2012.04 - 2017.09] SLAM 기반 과제 (HONDA Research Institute)

Past Topics

Google Business Photos

photo Googel Business Photos is a service which brings the StreetView inside of local businesses. It requires automatic panorama stitching, panorama localization using structure from motion, and manual editing for misplaced panoramas.
  • Mark Colbert, Jean-Yves Bouguet, Jeff Beis, Spudde Childs, Daniel Filip, Luc Vincent, Jongwoo Lim, Scott Satkin, “Building indoor multi-panorama experiences at scale,” in ACM SIGGRAPH 2012 Talks, Article No. 24 [link]

Human Robot Interaction with ASIMO

robot In Honda Research Institute USA, inc. I worked on a human-robot interaction project. We built a system which makes ASIMO to play a memory game (card matching) with a child player. All sensing is done using the onboard stereo camera.
  • Victor Ng-Thow-Hing, Jongwoo Lim, Joel Wormer, Ravi Kiran Sarvadevabhatla, Carlos Rocha, Kikuo Fujimura, Yoshiaki Sakagami, “The memory game: Creating a human-robot interactive scenario for ASIMO,” in IROS 2008, pp. 779-786 [link] [pdf]

Incremental Visual Tracking

ivt01 ivt02 ivt03 The incremental subspace update algorithm with mean update is proposed in the paper. The simple but effective visual tracking algorithm is proposed which uses the incremental subspace update algorithm for appearance modeling and the particle filtering for motion estimation.
  • David Ross, Jongwoo Lim, Ruei-Sung Lin, Ming-Hsuan Yang, “Incremental Learning for Robust Visual Tracking,” in IJCV, Vol. 77, No. 1-3, Pg. 125-141, May 2008 [link] [pdf] [project page]

Image Clustering

cluster Image clustering is to group images according to the identity or class of the objects in the images. Ideally the affinity measure must be insensitive to illumination variation or viewing direction changes in the images. We have proposed a few affinity measures for this purpose, and a general framework for hypergraph approximation.
  • Sameer Agarwal, Jongwoo Lim, Lihi Zelnik-Manor, Pietro Perona, David Kriegman, Serge Belongie, “Beyond Pairwise Clustering,” in CVPR 2005, vol. 2, pp. 838-845 [link] [abstract] [pdf]
  • Jongwoo Lim, Jeffrey Ho, Ming-Hsuan Yang, Kuang-Chih Lee, David Kriegman, “Image Clustering with Metric, Local Linear Structure and Affinity Symmetry,” in ECCV 2004, vol. 1, pp. 456-468 [link] [abstract] [pdf]
  • Jeffrey Ho, Ming-Hsuan Yang, Jongwoo Lim, Kuang-Chih Lee, David Kriegman, “Clustering Appearances of Objects Under Varying Illumination Conditions,” in CVPR 2003, vol 1, pp. 11-18 [link] [abstract] [pdf]