Communications Biology

Non-invasive eye tracking and retinal view reconstruction in free swimming schooling fish

R. Wu, O. Deussen, I. D. Couzin, L. Li

Material

Abstract

Eye tracking has emerged as a key method for understanding how animals process visual information, identifying crucial elements of perception and attention. Traditional fish eye tracking often alters animal behavior due to invasive techniques, while non-invasive methods are limited to either 2D tracking or restricting animals after training. Our study introduces a non-invasive technique for tracking and reconstructing the retinal view of free-swimming fish in a large 3D arena without behavioral training. Using 3D fish bodymeshes reconstructed by DeepShapeKit, our method integrates multiple camera angles, deep learning for 3D fish posture reconstruction, perspective transformation, and eye tracking. We evaluated our approach using data from two fish swimming in a flow tank, captured from two perpendicular viewpoints, and validated its accuracy using human-labeled and synthesized ground truth data. Our analysis of eye movements and retinal view reconstruction within leader-follower schooling behavior reveals that fish exhibit negatively synchronised eye movements and focus on neighbors centered in the retinal view. These findings are consistent with previous studies on schooling fish, providing a further, indirect, validation of our method. Our approach offers new insights into animal attention in naturalistic settings and potentially has broader implications for studying collective behavior and advancing swarm robotics.

BibTeX

@article{Wu2024Noninvasiveeye,
  abstract  = {Eye tracking has emerged as a key method for understanding how animals process visual information, identifying crucial elements of perception and attention. Traditional fish eye tracking often alters animal behavior due to invasive techniques, while non-invasive methods are limited to either 2D tracking or restricting animals after training. Our study introduces a non-invasive technique for tracking and reconstructing the retinal view of free-swimming fish in a large 3D arena without behavioral training. Using 3D fish bodymeshes reconstructed by DeepShapeKit, our method integrates multiple camera angles, deep learning for 3D fish posture reconstruction, perspective transformation, and eye tracking. We evaluated our approach using data from two fish swimming in a flow tank, captured from two perpendicular viewpoints, and validated its accuracy using human-labeled and synthesized ground truth data. Our analysis of eye movements and retinal view reconstruction within leader-follower schooling behavior reveals that fish exhibit negatively synchronised eye movements and focus on neighbors centered in the retinal view. These findings are consistent with previous studies on schooling fish, providing a further, indirect, validation of our method. Our approach offers new insights into animal attention in naturalistic settings and potentially has broader implications for studying collective behavior and advancing swarm robotics.},
  author    = {R. Wu, O. Deussen, I. D. Couzin, L. Li},
  day       = {12},
  doi       = {10.1038/s42003-024-07322-y},
  issn      = {2399-3642},
  journal   = {Communications Biology},
  month     = {Dec},
  number    = {1},
  pages     = {1636},
  title     = {Non-invasive eye tracking and retinal view reconstruction in free swimming schooling fish},
  url       = {https://doi.org/10.1038/s42003-024-07322-y},
  volume    = {7},
  year      = {2024}
}