Publications - Thematic Area C

Publications project C1 - Crossmodal speech representations and their implications in speech learning

Publications of the current project

Hu X, Wang F, Zhang D (2022). Similar brains blend emotion in similar ways: Neural representations of individual difference in emotion profiles, Neuroimage 247, 118819.
https://doi.org/10.1016/j.neuroimage.2021.118819
Li Z, Hong B, Wang D, Nolte G, Engel AK, Zhang D (2022). Speaker–listener neural coupling reveals a right-lateralized mechanism for non-native speech-in-noise comprehension, Cereb Cortex, bhac302, online published.
https://doi.org/10.1093/cercor/bhac302
Li J, Hong B, Nolte G, Engel AK, Zhang D (2022). Preparatory delta phase response is correlated with naturalistic speech comprehension performance, Cognitive Neurodynamics, 16, 337–352.
https://doi.org/10.1007/s11571-021-09711-z
Liu D, Xu X, Li D, Li J, Yu X, Ling Z, Hong B (2022). Intracranial brain-computer interface spelling using localized visual motion response, NeuroImage, 119363.
https://doi.org/10.1016/j.neuroimage.2022.119363
Chen J, Wang Y, Maye A, Hong B, Gao X, Engel AK, Zhang D (2021). A Spatially-coded Visual Brain-Computer Interface for Flexible Visual Spatial Information Decoding, IEEE Transactions on Neural Systems and Rehabilitation Engineering, 29: 926–933.
https://doi.org/10.1109/TNSRE.2021.3080045
Guo N, Si X, Zhang Y, Ding Y, Zhou W, Zhang D, Hong B (2021). Speech frequency-following response in human auditory cortex is more than a simple tracking, NeuroImage, 226, 117545.
https://doi.org/10.1016/j.neuroimage.2020.117545
Li Z, Li J, Hong B, Nolte G, Engel AK, Zhang D (2021). Speaker-Listener Neural Coupling Reveals an Adaptive Mechanism for Speech Comprehension in a Noisy Environment, Cerebral Cortex, 31(10), 4719–4729.
https://doi.org/10.1093/cercor/bhab118
Liu D, Liu C, Chen J, Zhang D, Hong B (2021). Doubling the Speed of N200 Speller via Dual-Directional Motion Encoding, IEEE Transactions on Biomedical Engineering, 68, 204-213.
https://doi.org/10.1109/TBME.2020.3005518
Yan Y, Qian T, Xu X, Han H, Ling Z, Zhou W, Liu H, Hong B (2020). Human cortical networking by probabilistic and frequency-specific coupling, NeuroImage, 207, 116363.
https://doi.org/10.1016/j.neuroimage.2019.116363
Chen J, Maye A, Engel AK, Wang Y, Gao X, Zhang D (2019). Simultaneous Decoding of Eccentricity and Direction Information for a Single-Flicker SSVEP BCI, Electronics, 8(12), 1554.
https://doi.org/10.3390/electronics8121554

Publications of the phase-1 project

Zhang Q, Hu X, Hong B, Zhang B. (2019). A hierarchical sparse coding model predicts acoustic feature encoding in both auditory midbrain and cortex. PLoS Computational Biology.
Chen, J., Li, Z., Hong, B., Maye, A., Engel, A. K., & Zhang, D. (2019). A Single-Stimulus, Multitarget BCI Based on Retinotopic Mapping of Motion-Onset VEPs. IEEE Transactions on Biomedical Engineering, 66(2), 464-470.
Si X, Zhou W, Hong B. (2017). Cooperative cortical network for categorical processing of Chinese lexical tone. Proceedings of the National Academy of Sciences. 114(46):12303-12308.

Relevant previous work

Ewald, A., Aristei, S., Nolte, G., & Rahman, R. A. (2012). Brain oscillations and functional connectivity during overt language production. Frontiers in psychology, 3.
Guo, J., Gao, S., & Hong, B. (2010). An auditory brain–computer interface using active mental response. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 18(3), 230-235.
Hong, B., Guo, F., Liu, T., Gao, X., & Gao, S. (2009). N200-speller using motion-onset visual response. Clinical neurophysiology, 120(9), 1658-1666.
Lou, B., Hong, B., Gao, X., & Gao, S. (2008). Bipolar electrode selection for a motor imagery based brain–computer interface. Journal of Neural Engineering, 5(3), 342.
Marzetti, L., Del Gratta, C., & Nolte, G. (2008). Understanding brain connectivity from EEG data by identifying systems composed of interacting sources. Neuroimage, 42(1), 87-98.
Nolte, G., Bai, O., Wheaton, L., Mari, Z., Vorbach, S., & Hallett, M. (2004). Identifying true brain interaction from EEG data using the imaginary part of coherency. Clinical neurophysiology, 115(10), 2292-2307.
Nolte, G., Ziehe, A., Nikulin, V. V., Schlögl, A., Krämer, N., Brismar, T., & Müller, K. R. (2008). Robustly estimating the flow direction of information in complex physical systems. Physical review letters, 100(23), 234101.
Qian, T., Zhou, W., Ling, Z., Gao, S., Liu, H., & Hong, B. (2013). Fast presurgical functional mapping using task-related intracranial high gamma activity: Laboratory investigation. Journal of neurosurgery, 119(1), 26-36.
Zhang, D., Gong, E., Wu, W., Lin, J., Zhou, W., & Hong, B. (2012, August). Spoken sentences decoding based on intracranial high gamma response using dynamic time warping. In Engineering in Medicine and Biology Society (EMBC), 2012 Annual International Conference of the IEEE (pp. 3292-3295). IEEE.
Zhang, D., Song, H., Xu, R., Zhou, W., Ling, Z., & Hong, B. (2013). Toward a minimally invasive brain–computer interface using a single subdural channel: a visual speller study. Neuroimage, 71, 30-41.

Publications project C2 - Crossmodal learning of language – mechanisms and facilitation

Publications of the phase 1 project

Huang, X., Zhang, H., Chen, C., Xue, G., & He, Q. (2019). The neuroanatomical basis of the Gambler's fallacy: A univariate and multivariate morphometric study. Human brain mapping, 40(3), 967-975.
Feng, K., Zhao, X., Liu, J., Cai, Y., Ye, Z., Chen, C., & Xue, G. (2019). Spaced learning enhances episodic memory by increasing neural pattern similarity across repetitions. Journal of Neuroscience, 39(27), 5351-5360.
Zhu, B., Chen, C., Shao, X., Liu, W., Ye, Z., Zhuang, L., ... & Xue, G. (2019). Multiple interactive memory representations underlie the induction of false memory. Proceedings of the National Academy of Sciences, 116(9), 3466-3475.
Xue, G. (2018). The neural representations underlying human episodic memory. Trends in cognitive sciences, 22(6), 544-561.
Qu, J., Qian, L., Chen, C., Xue, G., Li, H., Xie, P., & Mei, L. (2017). Neural pattern similarity in the left IFG and fusiform is associated with novel word learning. Frontiers in human neuroscience, 11, 424.
Xue, H., Zhao, L., Wang, Y., Dong, Q., Chen, C., & Xue, G. (2017). Anodal transcranial direct current stimulation over the left temporoparietal cortex facilitates assembled phonology. Trends in neuroscience and education, 8, 10-17.
Zhao, L., Chen, C., Shao, L., Wang, Y., Xiao, X., Chen, C., Yang, J., Zevin, Jason., & Xue, G. (2016). Orthographic and phonological representations in the fusiform cortex. Cerebral Cortex, 27(11), 5197-5210.

Relevant previous work

He, Q., Xue, G., Chen, C., Chen, C., Lu, Z. L., & Dong, Q. (2013). Decoding the neuroanatomical basis of reading ability: A multimoxel morphometric study. Journal of Neuroscience, 33, 12835-12843.
Helfrich, R. F., Knepper, H., Nolte, G., Strüber, D., Rach, S., Herrmann, C. S., Schneider, T. R., & Engel, A. K. (2014). Selective modulation of interhemispheric functional connectivity by HD-tACS shapes perception. PLoS Biology, 12, e1002031.
Hipp, J. F., Engel, A. K., & Siegel, M. (2011). Oscillatory synchronization in large-scale cortical networks predicts perception. Neuron, 69, 387-396.
Xue, G., Chen, C., Zhen, J., & Dong, Q. (2006). Language experience shapes fusiform activation in processing a logographic artificial language: an fMRI Training Study. Neuroimage, 31, 1315-1326.
Xue, G., Dong, Q., Chen, C., Lu, Z. L., Mumford, J. A., & Poldrack, R. A. (2013). Complementary role of frontoparietal activity and cortical pattern similarity in successful episodic memory encoding. Cerebral Cortex, 23, 1562-1571.
Xue, G., Dong, Q., Chen, C., Lu, Z. L., & Poldrack, R. A. (2010). Greater neural pattern similarity across repetitions is associated with better later memory. Science, 330, 97-101.
Xue, G., Mei, L., Chen, C., Lu, Z. L., Poldrack, R. A., & Dong, Q. (2010). Facilitating memory for novel characters by reducing the neural repetition suppression in the left fusiform cortex. PLoS One, 5, e13204.
Xue, G., Juan, C. H., Chang, C. F., Lu, Z. L., & Dong, Q. (2012). Lateral prefrontal cortex contributes to maladaptive decisions. Proceedings of the National Academy of Sciences U. S. A., 109, 4401-4406.
Lu, Y., Wang, C., Chen, C., & Xue, G. (2015). Spatiotemporal neural pattern similarity supports episodic memory. Current Biology, 25, 780-785.
Womelsdorf, T., Schoffelen, J.-M., Oostenveld, R., Singer, W., Desimone, R., Engel, A. K., & Fries, P. (2007). Modulation of neuronal interactions through neuronal synchronization. Science, 316, 1609-1612.

Publications project C4 - Vision- and action-embodied language learning

Publications of the current project

Ahrens K, Kerzel M, Lee JH, Weber C, Wermter S (2022). Knowing earlier what right means to you: A comprehensive VQA dataset for grounding relative directions via multi-task learning., IJCAI Workshop on Spatio-Temporal Reasoning and Learning - Jul 2022,.
not assigned
Lee JH, Yao Y, Özdemir O, Li M, Weber C, Liu Z, Wermter S (2022). Spatial relation learning in complementary scenarios with deep neural networks, Frontiers in Neurorobotics,.
https://doi.org/10.3389/fnbot.2022.844753
Özdemir O, Kerzel M, Weber C, Lee JH, Wermter S (2022). Language Model-Based Paired Variational Autoencoders for Robotic Language Learning, IEEE Transactions on Cognitive and Developmental Systems, early access.
https://doi.org/10.1109/TCDS.2022.3204452
Özdemir O, Kerzel M, Weber C, Lee JH, Wermter S (2022). Learning Flexible Translation Between Robot Actions and Language Descriptions, Proc. Artificial Neural Networks and Machine Learning – ICANN 2022, pp 246-257, Lecture Notes in Computer Science, vol 13530..
https://doi.org/10.1007/978-3-031-15931-2_21
Lee JH, Kerzel M, Ahrens K, Weber C, Wermter S (2022). What is right for me is not yet right for you: A dataset for grounding relative directions via multi-task learning., Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022..
https://doi.org/10.24963/ijcai.2022/145
Lysa Y, Weber C, Becker D, Wermter S (2022). Word-by-Word Generation of Visual Dialog using Reinforcement Learning., Proceedings 31st ICANN 2022, LNCS 13531, pp. 123-135, 2022..
https://doi.org/10.1007/978-3-031-15931-2_11
Sun X, Weber C, Kerzel M, Weber T, Li M, Wermter S (2022). Learning Visually Grounded Human-Robot Dialog in a Hybrid Neural Architecture., Proceedings 31st ICANN 2022, LNCS 13531, pp. 258-269, 2022..
https://doi.org/10.1007/978-3-031-15931-2_22
Volquardsen C, Lee JH, Weber C, Wermter S (2022). More Diverse Training, Better Compositionality! Evidence from Multimodal Language Learning., Proceedings 31st ICANN 2022, LNCS 13531, pp. 417–428, 2022..
https://doi.org/10.1007/978-3-031-15934-3_3
Yao Y, Dong B, Zhang A, Zhang Z, Xie R, Liu Z, Lin L, Sun M, Wang J (2022). Prompt Tuning for Discriminative Pre-trained Language Models., Findings of the Association for Computational Linguistics: ACL 2022, pages 3468–3473, Dublin, Ireland..
https://doi.org/10.18653/v1/2022.findings-acl.273
Yuan Yao, Ao Zhang, Xu Han, Mengdi Li, Cornelius Weber, Zhiyuan Liu, Stefan Wermter, Maosong Sun (2021). Visual Distant Supervision for Scene Graph Generation. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 15816-15826.
Mengdi Li, Cornelius Weber, Matthias Kerzel, Jae Hee Lee, Zheni Zeng, Zhiyuan Liu, Stefan Wermter (2021). Robotic Occlusion Reasoning for Efficient Object Existence Prediction. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Oct 2021.
Huajian Fang, Guillaume Carbajal, Stefan Wermter, Timo Gerkmann (2021). Variational Autoencoder for Speech Enhancement with a Noise-Aware Encoder. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 676-680, doi: 10.1109/ICASSP39728.2021.9414060.
Ozan Özdemir, Matthias Kerzel, Stefan Wermter (2021). Embodied Language Learning with Paired Variational Autoencoders. IEEE International Conference on Development and Learning (ICDL), pages 1-6, doi:10.1109/ICDL49984.2021.9515668, Aug 2021.
Hugo Carneiro, Cornelius Weber, Stefan Wermter (2021). FaVoA: Face-Voice Association Favours Ambiguous Speaker Detection. Proceedings of the 30th International Conference on Artificial Neural Networks (ICANN 2021), Volume LNCS 12891, pages 439-450, doi:10.1007/978-3-030-86362-3_36, Sep 2021.
Julien Scholz, Cornelius Weber, Burhan Hafez, Stefan Wermter (2021). Improving Model-Based Reinforcement Learning with Internal State Representations through Self-Supervision. Proceedings of the International Joint Conference on Neural Networks (IJCNN 2021), Jul 2021.
Aaron Eisermann, Jae Hee Lee, Cornelius Weber, Stefan Wermter (2021). Generalization in Multimodal Language Learning from Simulation. Proceedings of the International Joint Conference on Neural Networks (IJCNN 2021), Jul 2021.
Frank Röder, Ozan Özdemir, Phuong D. H. Nguyen, Stefan Wermter and Manfred Eppe (2021). The Embodied Crossmodal Self Forms Language and Interaction: A Computational Cognitive Review. Frontiers in Psychology, Vol. 12, doi: 10.3389/fpsyg.2021.716671
Matthias Möller, Johannes Twiefel, Cornelius Weber, Stefan Wermter (2021). Controlling the Noise Robustness of End-to-End Automatic Speech Recognition Systems. Proceedings of the International Joint Conference on Neural Networks (IJCNN 2021), Jul 2021.
Victor Uc-Cetina, Nicolás Navarro-Guerrero, Anabel Martin-Gonzalez, Cornelius Weber, Stefan Wermter (2021). Survey on reinforcement learning for language processing. arXiv: 2104.05565, Apr 2021.
Vadym Gryshchuk, Cornelius Weber, Chu Kiong Loo, Stefan Wermter (2021). Lifelong Learning from Event-based Data. European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN '21), Oct 2021.
Christoper Wulur, Cornelius Weber, Stefan Wermter (2021). Planning-integrated Policy for Efficient Reinforcement Learning in Sparse-reward Environments. Proceedings of the International Joint Conference on Neural Networks (IJCNN 2021), Jul 2021.
Jie Zhou, Shengding Hu, Xin Lv, Cheng Yang, Zhiyuan Liu, Wei Xu, Jie Jiang, Juanzi Li, Maosong Sun (2021). KACC: A Multi-task Benchmark for Knowledge Abstraction, Concretization and Completion. Findings of the Association for Computational Linguistics (ACL-IJCNLP 2021), August 2021.

Publications of the phase 1 project

Springenberg S., Lakomkin E., Weber C., Wermter S.(2019) Predictive Auxiliary Variational Autoencoder for Representation Learning of Global Speech Characteristics. Proc. International Conference on Spoken Language Processing (INTERSPEECH), 2019.
Yu, N., Weber, C., & Hu, X. (2019) Learning sparse hidden states in Long Short-Term Memory. Proc. 28th International Conference on Artificial Neural Networks and Machine Learning, ICANN 2019.
Xie, R., Heinrich, S., Liu, Z., Weber, C., Yao, Y., Wermter, S., & Sun, M. (2019). Integrating Image-based and Knowledge-based Representation Learning. IEEE Transactions on Cognitive and Developmental Systems, early access.
Heinrich, S., Springstübe, P., Knöppler, T., Kerzel, M., & Wermter, S. (2019). Continuous convolutional object tracking in developmental robot scenarios. Neurocomputing. 342, 137-144.
Weber, C. (2019). Lernverfahren fur neuronale Netze und Anwendungen in der Robotik. Mitt. Math. Ges. Hamburg, 39, 27-58
Hinz, T., Heinrich, S., & Wermter, S. (2019). Generating multiple objects at spatially distinct locations. International Conference on Learning Representations (ICLR), arxiv.org/abs/1901.00686
Zhang Z., Han X., Liu Z., Jiang X., Sun M., Liu Q. (2019). ERNIE: Enhanced Language Representation with Informative Entities. Proc. of the 57th Annual Meeting of the Association for Computational Linguistics (ACL).
Wang, X., Han, X., Lin, Y., Liu, Z., & Sun, M. (2018). Adversarial multi-lingual neural relation extraction. Proceedings of the 27th International Conference on Computational Linguistics, 1156-1166.
Han, X., Zhu, H., Yu, P., Wang, Z., Yao, Y., Liu, Z., & Sun, M. (2018). Fewrel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. arXiv preprint arXiv:1810.10147.
Han, X., Yu, P., Liu, Z., Sun, M., & Li, P. (2018). Hierarchical Relation Extraction with Coarse-to-Fine Grained Attention. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2236-2245.
Shen, S. Q., Chen, Y., Yang, C., Liu, Z. Y., & Sun, M. S. (2018). Zero-Shot Cross-Lingual Neural Headline Generation. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 26(12), 2319-2327.
Heinrich, S., & Wermter, S. (2018). Interactive natural language acquisition in a multi-modal recurrent neural architecture. Connection Science, 30(1), 99-133.
Heinrich, S., Alpay, T., Wermter, S. (2018). Adaptive and Variational Continuous Time Recurrent Neural Networks. Proc. IEEE ICDL-EpiRob 2018.
Heinrich, S., Kerzel, M., Strahl, E., Wermter, S. (2018). Embodied Multi-modal Interaction in Language learning: the EMIL data collection. Proc. ICDL-EpiRob Workshop on Active Vision, Attention, and Learning 2018.
Springstübe, P., Heinrich, S., Wermter, S. (2018). Continuous convolutional object tracking. Proc. ESANN, 2018.
Lin Y., Liu Z., Sun M. (2017). Neural Relation Extraction with Multi-lingual Attention. The 55th Annual Meeting of the Association for Computational Linguistics (ACL), 34-43.
Niu Y., Xie R., Liu Z., Sun M. (2017). Improved Word Representation Learning with Sememes. The 55th Annual Meeting of the Association for Computational Linguistics (ACL), 2049-2058.
Ng, H.G. et al., (2017). Hey Robot, Why Don't You Talk to Me? Proc. 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN 2017).
Xie, R., Liu Z., Luan H., Sun M. (2017). Image-embodied Knowledge Representation Learning. Proceedings of International Joint Conference on Artificial Intelligence.
Heinrich, S., Weber, C., Wermter, S., Xie, R., Lin, Y., Liu, Z. (2016). Crossmodal language grounding, learning, and teaching. NIPS2016 Workshop on Cognitive Computation, pp. 62-68, Barcelona, Spain.

Relevant previous work

Chen, X., Liu, Z., & Sun, M. (2014). A Unified Model for Word Sense Representation and Disambiguation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP'14) (pp.1025-1035). Association for Computational Linguistics.
Heinrich, S., Magg, S., & Wermter, S. (2015). Analysing the Multiple Timescale Recurrent Neural Network for Embodied Language Understanding. In Artificial Neural Networks - Methods and Applications (pp. 149-174). Springer International Publishing.
Heinrich, S., & Wermter, S. (2014). Interactive Language Understanding with Multiple Timescale Recurrent Neural Networks. In Proceedings of the 24th International Conference on Artificial Neural Networks (ICANN 2014) (pp. 193-200), Springer International Publishing.
Kleesiek, J., Engel, A. K., Weber, C., & Wermter, S. (2011, August). Reward-driven learning of sensorimotor laws and visual features. In Proceedings of the International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob) (pp. 1-6). IEEE Xpress.
Lin, Y., Liu, Z., Sun, M., Liu, Y., & Zhu, X. (2015, February). Learning Entity and Relation Embeddings for Knowledge Graph Completion. In Proceedings of the29th AAAI Conference on Artificial Intelligence (AAAI'15) (pp. 2181-2187). Association for the Advancement of Artificial Intelligence.
Saeb, S., Weber, C., & Triesch, J. (2009). Goal-directed learning of features and forward models. Neural Networks, 22(5-6), 586-592, Elsevier
Weber, C., Wermter, S., & Elshaw, M. (2006). A hybrid generative and predictive model of the motor cortex. Neural Networks, 19(4), 339-353.
Wermter, S., Weber, C., Elshaw, M., Gallese, V., & Pulvermüller, F. (2005). Grounding neural robot language in action. In Wermter S, Palm G, & Elshaw M. (Eds.), Biomimetic Neural Learning for Intelligent Robots (pp.162-181). Springer Berlin Heidelberg.
Zhao, Y., Liu, Z., & Sun, M. (2015, February). Phrase Type Sensitive Tensor Indexing Model for Semantic Composition. In Proceedings of the 29th Conference on Artificial Intelligence (AAAI 2015) (pp. 2195-2201).
Zhong, J., Weber, C., & Wermter, S. (2012). A predictive network architecture for a robust and smooth robot docking behavior. Paladyn. Journal of Behavioral Robotics, 3(4), 172-180.

Publications project C5 - Combining active vision with multimodal speech comprehension

Publications of the phase 1 project

Alaçam, Ö. (2019). Enhancing natural language understanding through cross-modal interaction: Meaning recovery from acoustically noisy speech. In Proceedings of the 22nd Nordic Conference on Computational Linguistics (NoDaLiDa), Association for Computational Linguistics, Turku, Finland.
Alaçam, Ö., Menzel, W., and Staron, T. (2019-accepted). How does visual complexity influence predictive language processing in a situated context? In Proceedings of the 15th KONVENS – Bridging the gap between NLP and human understanding, Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany.
Alaçam, Ö., Menzel, W. and Staron, T. (2019). Syntactic and Semantic Disambiguation in a Situated Language Setting. EuroCogsci. 2019. Bochum.
Qu, Q., & Damian, M. F. (2019). Orthographic effects in Mandarin spoken language production. Memory & cognition, 47(2), 326-334.
Lou, Y., Cai, H., Liu, X., & Li, X. (2019). Effects of self-enhancement on eye movements during reading. Frontiers in psychology, 10.
Qu, Q., & Damian, M. F. (2019). The role of orthography in second-language spoken word production: Evidence from Tibetan Chinese bilinguals. Quarterly Journal of Experimental Psychology.
Xu, P., Qu, Q., Shen, W., & Li, X. (2019). Co-activation of taxonomic and thematic relations in spoken word comprehension: Evidence from eye movements. Frontiers in psychology, 10, 964.
韩海宾, 许萍萍, 屈青青, 程茜, & 李兴珊. (2019). 语言加工过程中的视听跨通道整合. 心理科学进展, 27(3), 475-489.
Chen, L., Zhou, X., Müller, H. J., & Shi, Z. (2018). What you see depends on what you hear: Temporal averaging and crossmodal integration. Journal of Experimental Psychology: General, 147(12), 1851.
Salama, A. R., Alaçam, Ö and Menzel, W. (2018) Text Completion using Context-Integrated Dependency Parsing. In the Proceedings of 3rd Workshop on Representation Learning for NLP, ACL 2018.
Qu, Q., Cui, Z. L., and Damian, M. F(2018). Orthographic effects in second-language spoken word recognition. Journal of Experimental Psychology: Learning, Memory, and Cognition, 44(8), 1325 1332.
Alaçam, Ö., Staron, T., and Menzel, W. (2018). A multi-modal data-set for systematic analyses of linguistic ambiguities in situated contexts. In Annual International Symposium on Information Management and Big Data (pp. 106-122). Springer.
Staron, T., Alaçam, Ö., and Menzel, W. (2018). Incorporating Contextual Information for Language-Independent, Dynamic Disambiguation Tasks. In Proceedings of the 11th Language Resources and Evaluation Conference (LREC), Miyazaki, Japan.
屈青青*，刘维琳，李兴珊 (2018). 汉语言语产生的语音加工单元—基于音位的研究. 心理科学进展, 26(9), 1535-1544.
Alaçam, Ö., Staron, T. and Menzel, W. (2017). Towards a Systematic Analysis of Linguistic and Visual Complexity in Disambiguation and Structural Prediction. In the Proceedings of the 5th International Conference on Information Management and Big Data 4-6 September 2017. Lima, Peru.
Alaçam, Ö. and Habel, C. (2017). Negation in Gesture Accompanying Speech: A Cross-Modal Study. Journal of Multimodal Communication Studies, Vol. 4, issue 1-2.
Shen, W., Qu, Q., Ni, A., Zhou, J., & Li, X. (2017). The time course of morphological processing during spoken word recognition in Chinese. Psychonomic Bulletin & Review, 1-7. doi:10.3758/s13423-017-1274-z.
Qu, Q. Q., & Damian, M. F. (2017). Orthographic effects in spoken word recognition: Evidence from Chinese, Psychonomic Bulletin & Review, 1-6.

Relevant previous work

Baumgärtner, C., Beuck, N., & Menzel, W. (2012). An Architecture for Incremental Information Fusion of Cross-Modal Representations. In Proc. IEEE Conference on Multisensor Fusion and Integration for Intelligent Systems, MFI-2012 (pp. 498-503). IEEE
Beuck, N., & Menzel, W. (2013). Structural prediction in Incremental Dependency Parsing, In Proc. 14th Int. Conf. in Intelligent Text Processing and Computational Linguistics, CICLing-2013 (pp. 245-257). Springer
Foth, K. A., & Menzel, W. (2006). Hybrid Parsing: Using Probabilistic Models as Predictors for a Symbolic Parser, In Proc. 21st Int. Conf. on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, Coling-ACL-2006 (pp. 321-328). IEEE.
Köhn, A., & Menzel, W. (2014). Incremental Predictive Parsing with TurboParser, In Proc. 52nd Annual Meeting of the Association for Computational Linguistics (pp. 803-808).
Li, X., Bicknell, K., Liu, P., Wei, W., & Rayner, K. (2014). Reading is fundamentally similar across disparate writing systems: A systematic characterization of how words and characters influence eye movements in Chinese
reading. Journal of Experimental Psychology: General, 143, 895-913.
Li, X., Gu, J., Liu, P., & Rayner, K. (2013). The advantage of word-based processing in Chinese reading: Evidence from eye movements. Journal of Experimental Psychology: Learning, Memory and Cognition, 39, 879-889.
Li, X., & Logan, G. D. (2008). Object-based attention in Chinese readers of Chinese words: Beyond Gestalt principles. Psychonomic Bulletin & Review, 15, 945-949.
McCrae, P., Menzel, W., & Sun, M. (2011). A computational model of concept generalization in cross-modal reference. Tsinghua Science & Technology, 16, 113-120.
Menzel, W. (2009). Towards radically incremental parsing of natural language. In Nicolov, N., & Angelova, G. (Eds.), Proc. 5th Int. Conf. on Recent Advances in Natural Language Processing (pp. 41-56). Amsterdam &
Philadelphia: John Benjamin's Publishers.
Rayner, K., Li, X., & Pollatsek, A. (2007). Extending the E-Z Reader model of eye movement control to Chinese reading. Cognitive Science, 31, 1021-1033.

Publications project C6 - Multisensory perception, learning and action for crossmodal tele-operations

Publications of the phase 1 project

Zeng, H., & Chen, L. (2019). Robust temporal averaging of time intervals between action and sensation. Frontiers in psychology, 10, 511.
Chen, L. (2019). Discrimination of empty and filled intervals marked by auditory signals with different durations and directions of intensity change. PsyCh journal, 8(2), 187-202.
Shen, L., Han, B., Chen, L., & Chen, Q. (2019). Perceptual inference employs intrinsic alpha frequency to resolve perceptual ambiguity. PLoS biology, 17(3), e3000025.
Zheng, W., & Chen, L. (2019). Illusory perception of auditory filled duration is task‐and context‐dependent. British Journal of Psychology.
Li, B., Chen, L., & Fang, F. (2019). Somatotopic representation of tactile duration: evidence from tactile duration aftereffect. Behavioural brain research, 371, 111954.
He, D., Wang, Y., & Fang, F. (2019). The Critical Role of V2 Population Receptive Fields in Visual Orientation Crowding. Current Biology.
Katzakis, N., Chen,L., Ariza, O., Teather,R., and Steinicke, F.(2019).Evaluation of 3D Pointing Accuracy in the Fovea and Periphery in Immersive Head-Mounted Display Environments, IEEE Transaction on visualization and computer graphics
Wang, Y., Zhu, Z., Chen, B., & Fang, F. (2019). Perceptual learning and recognition confusion reveal the underlying relationships among the six basic emotions. Cognition and Emotion, 33(4), 754-767.
Chen, L.*, Zhou, X., Müller, H.J., Shi, Z. (in press) What you see depends on what you hear: temporal averaging and crossmodal integration, Journal of Experimental Psychology: General.
Zheng,W.* & Chen, L.* (2018). The Roles of Attentional Shifts and Attentional Reengagement in Resolving The Spatial Compatibility Effect in Tactile Simon-like Tasks. Scientific Reports, 8:8760. doi:10.1038/s41598-018-27114-9.
Wan, Y. & Chen, L.* (2018). Temporal Reference, Attentional Modulation, and Crossmodal Assimilation. Front. Comput. Neurosci. 12:39. doi: 10.3389/fncom.2018.00039.
Na R., Bi T., Tjan B.S., Liu Z. and Fang F. (2018). Effect of task difficulty on blood-oxygen-level-dependent signal: a functional magnetic resonance imaging study in a motion discrimination task. PLoS ONE 13(6): e0199440.
Wang Y., Zhu Z., Chen B. and Fang F. (2018) Perceptual learning and recognition confusion reveal the underlying relationships among the six basic emotions. Cognition and Emotion. https://doi.org/10.1080/02699931.2018.1491831
Yiltiz, H. & Chen, L.*（2018）. Emotional cues and social anxiety resolve ambiguous perception of biological motion. Exp Brain Res. 236：1409-1420
Tian,Y. & Chen, L.* (2018). Cross-modal attention modulates tactile subitizing but not tactile numerosity estimation. Attention, Perception, & Psychophysics, https://doi.org/10.3758/s13414-018-1507-x
Mo C.*, He D.* and Fang F. (2018). Attention priority map of face images in human early visual cortex. Journal of Neuroscience. 38(1), 149-157
Zhang, J., Langbehn, E., Krupke, D., Katzakis, N., & Steinicke, F. (2018). Detection Thresholds for Rotation and Translation Gains in 360° Video-Based Telepresence Systems. IEEE transactions on visualization and computer graphics, 24(4), 1671-1680.
Zhang, J., Langbehn, E., Krupke, D., Katzakis, N., & Steinicke, F. (2018). A 360 video-based robot platform for telepresent redirected walking.
Freiwald, J. P., Katzakis, N., & Steinicke, F. (2018). Camera time warp: compensating latency in video see-through head-mounted-displays for reduced cybersickness effects. In Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology. 9.
Ariza, O., Bruder, G., Katzakis, N., & Steinicke, F. (2018). Analysis of Proximity-Based Multimodal Feedback for 3D Selection in Immersive Virtual Environments. In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). 327-334.
Katzakis, N., & Steinicke, F. (2018). Excuse me! perception of abrupt direction changes using body cues and paths on mixed reality avatars. arXiv preprint arXiv:1801.05085.
Chen N., Lu J., Shao H., Weng X. and Fang F. (2017). Neural mechanisms of motion perceptual learning in noise. Human Brain Mapping. 38(12), 6029-6042
Chen, L., Guo, L., Bao,M. (2017). "Sleep-dependent consolidation benefits fast transfer of time interval training". Experimental Brain Research, 235(3),661-672.
Guo,L., Bao, M., Guan, L., Chen, L.. (2017). Cognitive Styles Differentiate Crossmodal Correspondences Between Pitch Glide and Visual Apparent Motion, Multisensory Research.30,363-385.
He D., Mo C. and Fang F. (2017). Predictive feature remapping before saccadic eye movements. Journal of Vision 17(5):14, 1-12.

Relevant previous work

Bruder, G., Steinicke, F., & Sturzlinger, W. (2013, July). To touch or not to touch?: comparing 2D touch and 3D mid-air interaction on stereoscopic tabletop surfaces. In Proceedings of the 1st symposium on Spatial user interaction (pp. 9-16). ACM.
Bruder, G., Lubos, P., & Steinicke, F. (2015). Cognitive Resource Demands of Redirected Walking. IEEE Trans. on Visualization and Computer Graphics (TVCG), 4, 539-544.
Chen, L., Wang, Q., & Bao, M. (2014). Spatial references and audio-tactile interaction in cross-modal dynamic capture. Multisensory Research, 27, 55-70.
Chen, L., & Vroomen, J (2013). Intersensory Binding across Space and Time: A Tutorial Review. Attention, Perception, & Psychophysics, 75, 790-811.
Chen, L., Shi, Z., & Müller, H. J. (2011). Interaction of Perceptual Grouping and Crossmodal Temporal Capture in Tactile Apparent-Motion. PLoS ONE, 6(2), e17130.
Chen, J., He, Y., Zhu, Z., Zhou, T., Peng, Y., Zhang, X., & Fang, F. (2014). Attention-dependent early Cortical Suppression contributes to Crowding. Journal of Neuroscience. 34(32), 10465-10474.
Daiber, F., Valkov, D., Steinicke, F., Hinrich, K., & Krüger, A. (2012). Towards Object Prediction based on Hand Postures for Reach to Grasp Interaction. In Proceedings of the ACM, the 3rd Dimension of CHI (3DCHI) (pp.99-106). ACM.
Fang, F., & He, S. (2005). Cortical Responses to Invisible Objects in Human Dorsal and Ventral Pathways. Nature Neuroscience. 8(10), 1380-1385.
Steinicke, F., Bruder, G., Jerald, J., Frenz, H., & Lappe, M. (2010). Estimation of Detection Thresholds for Redirected Walking Techniques. IEEE Trans. on Visualization and Computer Graphics (TVCG), 16(1), 17-27
Zhang, X., Zhaoping, L., Zhou, T., & Fang, F. (2012). Neural Activities in V1 create a Bottom-Up Saliency Map. Neuron 73(1), 183-192.

Publications project C7 - Cross-modal Learning for Improving Human Reading

Publications of the current project

Li X, Huang L, Yao P, Hyönä J (2022). Universal and specific reading mechanisms across different writing systems., Nature Reviews Psychology..
https://doi.org/10.1038/s44159-022-00022-6
Gregg J, Inhoff AW, Li X (2022). Lexical competition influences correct and incorrect visual word recognition., Quarterly Journal of Experimental Psychology. Advance online publication..
http://doi.org/10.1177/17470218221102878
Zhang G, Yao P, Ma G, et al. (2022). The database of eye-movement measures on words in Chinese reading, Scientific Data, 9(1), 411.
https://doi.org/10.1038/s41597-022-01464-6
Wang X, Schneider S, Alaçam Ö, Chaudhury P, Biemann C (2022). MOTIF: Contextualized Images for Complex Words to Improve Human Reading, Proceedings of the 13th Edition of its Language Resources and Evaluation Conference (LREC 2022). Pages 2468-2477.
Wiehe A, Schneider S, Blank S, Wang X, Zorn H, Biemann C (2022). Language over Labels: Contrastive Language Supervision Exceeds Purely Label-Supervised Classification Performance on Chest X-Rays., AACL 2022.
Yao, P., Slattery, T. J., & Li, X. (in press). Sentence context modulates the neighborhood frequency effect: Evidence from eye movements while reading Chinese. Journal of Experimental Psychology: Learning, Memory, and Cognition.
Yao, P., Staub, A., & Li, X. (2021). Predictability Eliminates Neighborhood Effects During Chinese Sentence. Psychonomic Bulletin & Review. https://doi.org/10.3758/s13423-021-01966-1
Huang, L., Staub, A., & Li, X. (2021). Prior context influences lexical competition when segmenting Chinese overlapping ambiguous strings. Journal of Memory and Language, 118
Florian Schneider, Özge Alaçam, Xintong Wang, Chris Biemann (2021). Towards Multi-Modal Text-Image Retrieval to improve Human Reading. NAACL SRW 2021.
Özge Alaçam, Ganeshan Malhotra, Eugen Ruppert, Chris Biemann (2021). Gaze-based Multimodal Meaning Recovery for Noisy / Complex Environments. ICMI 2021: 673-681
Li, X., & Pollatsek, A. (2020). An integrated model of word processing and eye-movement control during Chinese reading. Psychological Review, 127(6), 1139-1162.
Han, H. & Li, X.(2020). Degree of conceptual overlap affects eye movements in visual world paradigm. Language, Cognition and Neuroscience, 35(10), 1456-1464.
Huang, L., & Li, X. (2020). Early, but not overwhelming: The effect of prior context on segmenting overlapping ambiguous strings when reading Chinese. Quarterly Journal of Experimental Psychology. 73(9), 1382-1395. https://doi.org/10.1177/1747021820926012

Publications project C8 - Crossmodal bindings and plasticity during visual-haptic interactions for novel forms of therapy

Publications of the current project

Wang H, Wei K (2022). Transcranial Direct-Current Stimulation Does Not Affect Implicit Sensorimotor Adaptation: A Randomized Sham-Controlled Trial, Brain Sciences. 29;12(10):1325.
https://doi.org/10.3390/brainsci12101325
Hartfill J, Gabel J, Kruse L, Schmidt S, Riebandt K, Kühn S, Steinicke F (2021). Analysis of Detection Thresholds for Hand Redirection during Mid-Air Interactions in Virtual Reality, Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology (VRST 2021), pp 1-10.
https://doi.org/10.1145/3489849.3489866
Liu Y, Jiang W, Bi Y, Wei K (2021). Sensorimotor knowledge from task-irrelevant feedback contributes to motor learning., Journal of Neurophysiology. 2021 Sep 1;126(3):723-35.
https://doi.org/10.1152/jn.00174.2021
Chen L, Feng Y, Chen B, Wang Q, Wei K (2021). Improving postural stability among people with lower-limb amputations by tactile sensory substitution., Journal of NeuroEngineering and Rehabilitation 2021, 18(1):1-4.
https://doi.org/10.1186/s12984-021-00952-x
Wang W, Lei X, Gong W, Liang K, Chen L (2022). Facilitation and inhibition effects of anodal and cathodal tDCS over areas MT+ on the flash-lag effect, Journal of Neurophysiology 2022, 128(1):239-48.
https://doi.org/10.1152/jn.00091.2022
Lou C, Zeng H, Chen L (2022). Asymmetric switch cost between subitizing and estimation in tactile modality, Current Psychology, 2022, https://doi.org/10.1007/s12144-022-02858-w.
https://doi.org/10.1007/s12144-022-02858-w,
Katzakis, Chen, Ariza, Teacher, & Steinicke (2021). Evaulation of 3D Pointing Accuracy in the Fovea and Periphery in Immersive Head-Mounted Display Environments. IEEE Transaction on Visualization and Computer Graphics, 27:3,1929-1936.
Gong, Gu, Wang & Chen (2021). Interoception visualization relieves acute pain. Biological Psychology, in minor revision.
Katzakis, Chen & Steinicke (2020). Visual-Haptic Size Estimation in Peripersonal Space. Frontiers in Neurorobotics, 14:18.

Publications project C9 - The role of mental models and sense of agency in learning crossmodal communicative acts

Publications of the current project

Zhang L, Khandil F, Zhao K, Fu X, Hilgetag H, Gläscher J (2022). A causal role of the human left temporo-parietal junction in computing simulated social influence during goal-directed learning, biorxiv Preprint.
https://doi.org/10.1101/2022.06.13.495824
Korn C, Redzepovic S, Gläscher J, Kerzel M, Barros P, Heinrich S, Wermter S (2021). Cross-modal emotion recognition: How similar are patterns between DNNs and human fMRI data?, ICLR2021 Workshop on How Can Findings About The Brain Improve AI Systems? (Brain2AI@ICLR2021), Number 18 - May 2021.
Möhring L, Gläscher J (2022). Common and distinct changes in brain activation patterns modulated by two different types of prediction errors, Conference on Cognitive Computational Neuroscience San Francisco, August 25 - 28, 2022, paper P-1.13.
Li Y, Gu J, Zhao K, Fu X (2022). Developmental trajectory of time perception from childhood to adolescence., Current Psychology..
https://doi.org/10.1007/s12144-022-03526-9
Cui Q, Liu M, Liu CH, Long Z, Zhao K, Fu X (2021). Unpredictable fearful stimuli disrupt timing activities: Evidence from event-related potentials, Neuropsychologia, 163, 108057.
https://doi.org/10.1016/j.neuropsychologia.2021.108057
Liu M, Liu CH, Zheng S, Zhao K, Fu X (2021). Reexamining the Neural Network Involved in Perception of Facial Expression: A Meta-analysis, Neuroscience and Biobehavioral Reviews, 131，179–191.
https://doi.org/10.1016/j.neubiorev.2021.09.024
Gu JJ, Li YY, Zhao K, Fu X (2021). Disappearing and Appearing: Temporal Binding Effects are Consistent Across Situations, Consciousness and Cognition. 93, 103166..
https://doi.org/10.1016/j.concog.2021.103166
Zhao K, Gu JJ, Huang G, Zheng S (2021). The spatiotemporal markers for the sense of agency in the human brain, Advances in Psychological Science, 29(11), 1901-1910.
https://doi.org/10.3724/SP.J.1042.2021.01901