Marcos V. Conde

I'm a PhD Researcher in Artificial Intelligence and Computer Vision at the University of Würzburg, advised by Prof. Radu Timofte. My work is supported by Sony PlayStation (FTG) where I'm a Computer Vision Researcher working on AI Enhancement and Super-Resolution.
I'm also Senior Data Scientist (Kaggle Grandmaster) at H2O.ai.

During 2020-2021, I worked at Huawei Noah’s Ark Lab (London) supervised by Dr. Eduardo Pérez-Pellitero and Prof. Aleš Leonardis, and received the best intern award for my work on neural camera ISPs.

I received my M.Sc. in Computer Vision from the Autonomous University of Barcelona (UAB) with honours for my work on Real-time Photography Enhancement with Michael S. Brown and Javi Vazquez-Corral.

I lead a small team of crazy students at CIDAUT (Spain) working on Vision for Autonomous Driving -- 18M $ budget.

Email  /  Google Scholar  /  GitHub  /  Kaggle  /  Medium  /  Hugging Face 🤗

井の中の蛙大海を知らず  |  井鼃不可以語於海者，拘於虛也
"I learned very early the difference between knowing the name of something and knowing something." - Richard Feynmann

News

[Oct 2024] ECCV 2024 InstructIR: High-Quality Image Restoration Following Human Instructions is trending on HF 🤗. Read the preprint | Watch the demo Video | Our demo is available Try it now! (click)

[June 2024] We organize the 1st AI for Streaming Workshop at CVPR 2024. Check the awesome speakers and challenges in the website. The workshop is sponsored by Sony PlayStation, Meta and Netflix.

[Jan 2024] I co-organize the New Trends in Image Restoration and Enhancement (NTIRE) Workshop at CVPR 2024.
I organize the RAW SR and Portrait IQA Challenges.

Highlights

Research

My current research interests include neural networks, deep learning, low-level computer vision, inverse problems, computational photography and photorealism. How can we improve/change/enhance cameras and vision using AI?
Representative papers are highlighted.

(Past/Present) Collaborators: Radu Timofte  /  Michael S. Brown (York University, Samsung)  /  Tom E. Bishop (Glass Imaging)  /  Javier Vazquez-Corral (CVC, UAB)  /  Sira Ferradans (DXOMARK)  /  Ioannis Katsavounidis (Meta, AOMedia)

InstructIR takes as input a degraded image and a human-written instruction for how to improve that image. The (single) neural model performs all-in-one image restoration. We achieve state-of-the-art results on several restoration tasks including image denoising, deraining, deblurring, dehazing, and (low-light) image enhancement.

We present a novel analysis on compressing neural fields, with the focus on images. We also introduce Adaptive Neural Images (ANI), an efficient neural representation that enables adaptation to different inference or transmission requirements.

Implicit Neural Representations (INRs) are a novel paradigm for signal representation that have attracted considerable interest for image compression. We explore the critical yet overlooked limiting factors of INRs, such as computational cost, unstable performance, and robustness

We tackle image restoration directly in the RAW domain (yes, it is tricky and a bit crazy).

First, we propose a novel module that can be integrated into any neural ISP to capture the global context information from the full RAW images. Second, we propose an efficient and simple neural ISP that utilizes our proposed module.

NILUTs are neural representations of real 3D LUTs for controllable photo-realistic image enhancement and color manipulation. Moreover, a NILUT can be extended to incorporate multiple styles into a single network with the ability to blend styles implicitly.

Collaboration with Meta. This paper introduces a novel benchmark as part of the AIS 2024 Real-Time Image Super-Resolution (RTSR) Challenge, which aims to upscale compressed images from 540p to 4K resolution (4x factor) in real-time on commercial GPUs. The images are compressed using the modern AVIF codec, instead of JPEG.

Collaboration with DXOMARK. This paper reviews the NTIRE 2024 Portrait Quality Assessment Challenge, highlighting the proposed solutions and results. This challenge aims to obtain an efficient deep neural network capable of estimating the perceptual quality of real portrait photos.

New methods for RAW Super-Resolution could be essential in modern Image Signal Processing (ISP) pipelines, however, this problem is not as explored as in the RGB domain. The goal of this challenge is to upscale RAW Bayer images by 2x, considering unknown degradations such as noise and blur.

We advance RAW sensor images up-scaling (Super-Resolution). We explore diverse image degradations (e.g. Noise, Blur) to emulate a low-resolution RAW image, and we train a neural network to upsample it.

EBokehNet, an efficient state-of-the-art solution for Bokeh effect transformation and rendering. We can render Bokeh from all-in-focus images, or transform the Bokeh of one lens to the effect of another lens without harming the sharp foreground in the image.

The paper presents an exhaustive study of baseline methods for real-time image SR. The methods allow 60 FPS and even 120 FPS.

The paper gauges the state-of-the-art methods for real-time 4K upscaling using our new dataset and benchmark protocol. Ovr 100 participants joined the challenge and 15 teams proposed diverse solutions. The methods allow 60 FPS and even 120 FPS on regular GPUs.

We propose LPIENet, a lightweight network for perceptual image enhancement, with the focus on deploying it on smartphones. The model was tested for image denoising, deblurring, and HDR correction.

Super-resolution of compressed images using transformers. We use the Swin Transformer V2, to improve SwinIR for image super-resolution, and in particular, the compressed input scenario. Using this method we can tackle the major issues in training transformer vision models, such as training instability, resolution gaps between pre-training and fine-tuning.

This paper introduces the AIM 2022 Challenge on Reversed Image Signal Processing and RAW Reconstruction. We aim to recover raw sensor images from the corresponding RGBs without metadata and, by doing this, "reverse" the ISP transformation.

Hybrid model-based and data-driven approach for modelling ISPs using learnable dictionaries. We explore RAW image reconstruction and improve downstream tasks like RAW Image Denoising via raw data augmentation-synthesis.

We were one of the 1st attempts to use CLIP (Contrastive Language-Image Pre-Training) for training a neural network on a variety of art images and descriptions, being able to learn directly from raw descriptions about images, or if available, curated labels.

Academic Service

Teaching: Image Processing and Computational Photography (IPCP), Computer Vision (CV)

Reviewer: Outstanding Reviewer at ECCV 2024. CVPR 2022-2024, ECCV 2022-24, ICCV 2023, AAAI 2023-24, SIGGRAPH 2024, ACCV 2024, IEEE Transactions on Image Processing, IEEE Transactions on Computational Imaging

Workshops: AIM 2024 ECCV  /  AI for Streaming (AIS 2024) CVPR  /  NTIRE 2024 CVPR  /  VQEG 2023  /  NTIRE 2023 CVPR  /  AIM 2022 ECCV

Other Projects

We introduce a complete open-source ecosystem for developing and testing LLMs. The goal of this project is to boost open alterna- tives to closed-source approaches.
Authors: Arno Candel, Jon McKinney, Philipp Singer, Pascal Pfeiffer, Maximilian Jeblick, Chun Ming Lee, Marcos V. Conde

We proposed multiple methods for motion prediction in Autonomous Driving. The methods were presented at CVPR Workshops, ICRA, and IEEE Transactions on Intelligent Transportation Systems (T-ITS, Q1, IF:8.5).
Authors: Carlos Gómez-Huélamo, Marcos V. Conde

This survey reviews the AIS 2024 Event-Based Eye Tracking (EET) Challenge. The task of the challenge focuses on processing eye movement recorded with event cameras and predicting the pupil center of the eye.
Authors: Zuowen Wang, Chang Gao, Zongwei Wu, Marcos V. Conde, Radu Timofte, Shih-Chii Liu, Qinyu Chen

I was invited by Japan’s National Institute of Informatics (NII) and ROIS-DS Center for Open Data in the Humanities (CODH) to present a novel solution for the Kuzushiji Recognition Challenge at the Japanese Culture and AI Symposium 2019 in Tokyo.
Organizers: Dr. Asanobu Kitamoto, Tarin Clanuwat, Alex Lamb.

Design and source code from Jon Barron's website.