We present a platform to foster research in active scene understanding, consisting of high-fidelity simulated environments and a simple yet powerful API that controls a mobile robot in simulation and reality. In contrast to static, pre-recorded datasets that focus on the perception aspect of scene understanding, agency is a top priority in our work. We provide three levels of robot agency, allowing users to control a robot at varying levels of difficulty and realism. While the most basic level provides pre-defined trajectories and ground-truth localisation, the more realistic levels allow us to evaluate integrated behaviours comprising perception, navigation, exploration and SLAM. In contrast to existing simulation environments, we focus on robust scene understanding research using our environment interface (BenchBot) that provides a simple API for seamless transition between the simulated environments and real robotic platforms. We believe this scaffolded design is an effective approach to bridge the gap between classical static datasets without any agency and the unique challenges of robotic evaluation in reality. Our BenchBot Environments for Active Robotics (BEAR) consist of 25 indoor environments under day and night lighting conditions, a total of 1443 objects to be identified and mapped, and ground-truth 3D bounding boxes for use in evaluation. BEAR website: https://qcr.github.io/dataset/benchbot-bear-data/.
AmmiratoPPoirsonPParkE, et al. (2017) A dataset for developing and benchmarking active vision. In: IEEE International Conference on Robotics and Automation (ICRA), Singapore, 29 May–3 June 2017.
2.
BistaSRHallDTalbotB, et al. (2021) Evaluating the impact of semantic segmentation and pose estimation on dense semantic slam. In: 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Prague, Czech Republic, 27 September–1 October 2021. arXiv preprint arXiv:2109.07748.
3.
ChangADaiAFunkhouserT, et al. (2017) Matterport3d: Learning from rgb-d data in indoor environments. In: International Conference on 3D Vision (3DV), Qingdao, China, 10–12 October 2017.
4.
CordtsMOmranMRamosS, et al. (2016) The cityscapes dataset for semantic urban scene understanding. In: Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016.
5.
DeitkeMHanWHerrastiA, et al. (2020) Robothor: An open simulation-to-real embodied ai platform. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 13-19 June 2020, pp. 3164–3174.
6.
DosovitskiyARosGCodevillaF, et al. (2017) CARLA: An open urban driving simulator. In: Proceedings of the 1st Annual Conference on Robot Learning, Mountain View, California, USA, 13-15 November, pp. 1–16.
7.
GeigerALenzPStillerC, et al. (2013) Vision meets robotics: The kitti dataset. International Journal of Robotics Research (IJRR)32(11): 1231–1237.
8.
GuerraWTalEMuraliV, et al. (2019) Flightgoggles: Photorealistic sensor simulation for perception-driven robotics using photogrammetry and virtual reality. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Macau, China, 3–8 November 2019, pp. 6941–6948. IEEE,.
9.
HallDTalbotBBistaSR, et al. (2020) The robotic vision scene understanding challenge. arXiv preprint arXiv:2009.05246.
10.
JamesSMaZArrojoDR, et al. (2020) RLBench: the robot learning benchmark learning environment. IEEE Robotics and Automation Letters5(2): 3019–3026. DOI: 10.1109/LRA.2020.2974707.
11.
KadianATruongJGokaslanA, et al. (2020) Sim2real predictivity: Does evaluation in simulation predict real-world performance?IEEE Robotics and Automation Letters (RA-L)5(4): 6670–6677.
12.
KolveEMottaghiRHanW, et al. (2017) Ai2-thor: an interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474.
13.
LinTYMaireMBelongieS, et al. (2014) Microsoft coco: common objects in context. In: European conference on computer vision, Zurich, Switzerland, 6-12 September, pp. 740–755. Springer.
14.
MaddernWPascoeGLinegarC, et al. (2017) 1 year, 1000km: the oxford robotcar dataset. The International Journal of Robotics Research (IJRR)36(1): 3–15. DOI: 10.1177/0278364916679498.
15.
SavvaMKadianAMaksymetsO, et al. (2019) Habitat: a platform for embodied AI research. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea, 27 October–2 November 2019.
16.
McCormacJClarkRBloeschM, et al. (2018) Fusion++: volumetric object-level slam. In: 2018 international conference on 3D vision (3DV), Verona, Italy, 5–8 September 2018, pp. 32–41. IEEE.
17.
McCormacJHandaALeuteneggerS, et al. (2017) Scenenet rgb-d: can 5m synthetic images beat generic imagenet pre-training on indoor segmentation? In: 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 22–29 October 2017.
18.
NaritaGSenoTIshikawaT, et al. (2019) Panopticfusion: online volumetric semantic mapping at the level of stuff and things. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Macau, China, 3–8 November 2019, pp. 4205–4212. IEEE.
RavichandranZGriffithJDSmithB, et al. (2020) Bridging scene understanding and task execution with flexible simulation environments. arXiv preprint arXiv:2011.10452.
21.
RosinolAAbateMChangY, et al. (2020a) Kimera: an open-source library for real-time metric-semantic localization and mapping. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France, 31 May–31 August 2020, pp. 1689–1696. DOI: 10.1109/ICRA40945.2020.9196885.
22.
RosinolAGuptaAAbateM, et al. (2020b) 3d dynamic scene graphs: actionable spatial perception with places, objects, and humans. arXiv preprint arXiv:2002.06289.
23.
ShahSDeyDLovettC, et al. (2017) Airsim: High-fidelity visual and physical simulation for autonomous vehicles. In: Field and Service Robotics, Zurich, Switzerland, 13-15 September. https://arxiv.org/abs/1705.05065.
24.
ShenBXiaFLiCet al. (2020) igibson, a simulation environment for interactive tasks in large realistic scenes. arXiv preprint arXiv:2012.02924.
25.
SongSLichtenbergSPXiaoJ (2015) Sun rgb-d: a rgb-d scene understanding benchmark suite. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Boston, MA, USA, 7–12 June 2015, pp. 567–576.
26.
StraubJWhelanTMaL, et al. (2019) The Replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv:1906.05797.
27.
TalbotBHallDZhangH, et al. (2020) Benchbot: evaluating robotics research in photorealistic 3d simulation and on real robots. arXiv preprint arXiv:2008.00635.
28.
WijmansEDattaSMaksymetsO, et al. (2019) Embodied question answering in photorealistic environments with point cloud perception. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 15–20 June 2019.
29.
XiaFZamirARHeZ, et al. (2018) Gibson env: real-world perception for embodied agents. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018, pp. 9068–9079.
30.
ZhengTZhangGHanL, et al (2020) Building fusion: semantic-aware structural building-scale 3d reconstruction. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, DOI: 10.1109/TPAMI.2020.3042881.
