Overview

A robot has different means to actively explore and better understand its environment. It can look around and fixate on interesting areas, it can ask someone for more information or it can physically interact with the environment to explore and better understand its structure.

Feedback received in this manner provides informative sensory signals that would otherwise not be present and are especially important for manipulation. Furthermore, understanding the signal in the context of the action that has produced it facilitates interpretation and prediction of the signal.

Check out the papers below to get an idea on how we implemented this concept on robots. Read more about Interactive Perception in this comprehensive survey!

Fusing Vision, Touch and Motion

Lee, M.*, Zhu, Y.*, Zachares, P., Tan, M., Srinivasan, K., Savarese, S., Fei-Fei, L., Garg, A., Bohg, J. Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks. Transactions in Robotics (Accepted for Publication). 2019.

Lee, M., Zhu, Y., Srinivasan, K., Shah, P., Savarese, S., Fei-Fei, L., Garg, A., Bohg, J. Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks. Accepted at ICRA '19. Best Paper Award and Finalist for Best Paper Award on Cognitive Robotics.

Shao, L., Shah, P., Dwaracherla, V., Bohg, J. Motion-based Object Segmentation based on Dense RGB-D Scene Flow IEEE Robotics and Automation Letters, 3(4):3797-3804, IEEE, IEEE/RSJ International Conference on Intelligent Robots and Systems, October 2018

Ilonen, J., Bohg, J., Kyrki, V. Fusing visual and tactile sensing for 3-D object reconstruction while grasping In IEEE International Conference on Robotics and Automation (ICRA), pages: 3547-3554, 2013

Illonen, J., Bohg, J., Kyrki, V. 3-D Object Reconstruction of Symmetric Objects by Fusing Visual and Tactile Sensing The International Journal of Robotics Research, 33(2):321-341, Sage, October 2013.

Bohg, J., Johnson-Roberson, M., Björkman, M., Kragic, D. Strategies for multi-modal scene exploration In Intelligent Robots and Systems (IROS), 2010 IEEE/RSJ International Conference on, pages: 4509-4515, October 2010.

Top-Down and Bottom-Up Visual Attention

Kloss, A., Kappler, D., Lensch, H. P. A., Butz, M. V., Schaal, S., Bohg, J. Learning Where to Search Using Visual Attention Proceedings of the IEEE/RSJ Conference on Intelligent Robots and Systems, IEEE, IROS, October 2016

Gratal, X., Bohg, J., Björkman, M., Kragic, D. Scene Representation and Object Grasping Using Active Vision In IROS’10 Workshop on Defining and Solving Realistic Perception Problems in Personal Robotics, October 2010

Johnson-Roberson, M., Bohg, J., Björkman, M., Kragic, D. Attention-based active 3D point cloud segmentation In Intelligent Robots and Systems (IROS), 2010 IEEE/RSJ International Conference on, pages: 1165-1170, October 2010.

Active Learning and Information Gathering

Shao, L., Migimatsu, T., Bohg, J. Learning to Scaffold the Development of Robotic Manipulation Skills Submitted to ICRA. 2020.

Marco, A., Hennig, P., Bohg, J., Schaal, S., Trimpe, S. Automatic LQR Tuning Based on Gaussian Process Global Optimization In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), IEEE International Conference on Robotics and Automation, May 2016

Toussaint, M., Ratliff, N., Bohg, J., Righetti, L., Englert, P., Schaal, S. Dual Execution of Optimized Contact Interaction Trajectories In Proceedings of the International Conference on Intelligent Robots and Systems, Chicago, IL, October 2014.

Johnson-Roberson, M., Bohg, J., Skantze, G., Gustafson, J., Carlson, R., Rasolzadeh, B., Kragic, D. Enhanced visual scene understanding through human-robot dialog In Intelligent Robots and Systems (IROS), 2011 IEEE/RSJ International Conference on, pages: 3342-3348, 2011.

Cognitive Science

Bohg, J., Hausman, K., Sankaran, B., Brock, O., Kragic, D., Schaal, S., Sukhatme, G. Interactive Perception: Leveraging Action in Perception and Perception in Action IEEE Transactions on Robotics, 33, pages: 1273-1291, December 2017.

Dominey, P. F., Prescott, T. J., Bohg, J., Engel, A. K., Gallagher, S., Heed, T., Hoffmann, M., Knoblich, G., Prinz, W., Schwartz, A. Implications of Action-Oriented Paradigm Shifts in Cognitive Science In The Pragmatic Turn - Toward Action-Oriented Views in Cognitive Science, 18, pages: 333-356, 20, Strüngmann Forum Reports, vol. 18, J. Lupp, series editor, (Editors: Andreas K. Engel and Karl J. Friston and Danica Kragic), The MIT Press, 18th Ernst Strungmann Forum, May 2016.

Bohg, J., Kragic, D. Learning Action-Perception Cycles in Robotics: A Question of Representations and Embodiment. In The Pragmatic Turn - Toward Action-Oriented Views in Cognitive Science, 18, pages: 333-356, 20, Strüngmann Forum Reports, vol. 18, J. Lupp, series editor, (Editors: Andreas K. Engel and Karl J. Friston and Danica Kragic), The MIT Press, 18th Ernst Strungmann Forum, May 2016.