🔥 News

April 27, 2024 Our team will attend ICLR 2024 to present our paper in Halle B on Tuesday, May 7, from 10:45 a.m. to 12:45 p.m. CEST (local time in Austria). Your local time: May 7, 2024 4:35 AM (EDT). Session link: here. Full ICLR 2024 calendar: here.
January 31, 2024 Accepted by ICLR 2024 as a spotlight presentation.

🔥 Table of Content

🔥 TLDR

This paper formalizes and addresses signal delay in deep reinforcement learning, introducing effective strategies that maintain high performance in robotic control tasks despite substantial delays.

🔥 Overview

Despite the notable advancements in deep reinforcement learning (DRL) in recent years, a prevalent issue that is often overlooked is the impact of signal delay. Signal delay occurs when there is a lag between an agent's perception of the environment and its corresponding actions. In this paper, we first formalize delayed-observation Markov decision processes (DOMDP) by extending the standard MDP framework to incorporate signal delays. Next, we elucidate the challenges posed by the presence of signal delay in DRL, showing that trivial DRL algorithms and generic methods for partially observable tasks suffer greatly from delays. Lastly, we propose effective strategies to overcome these challenges. Our methods achieve remarkable performance in continuous robotic control tasks with large delays, yielding results comparable to those in non-delayed cases. Overall, our work contributes to a deeper understanding of DRL in the presence of signal delays and introduces novel approaches to address the associated challenges.