Skip to main content
SpringerLink
Log in

Multiple Instance Neuroimage Transformer

  • Conference paper
  • First Online:
Predictive Intelligence in Medicine (PRIME 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13564))

Included in the following conference series:

Abstract

For the first time, we propose using a multiple instance learning based convolution-free transformer model, called Multiple Instance Neuroimage Transformer (MINiT), for the classification of T1-weighted (T1w) MRIs. We first present several variants of transformer models adopted for neuroimages. These models extract non-overlapping 3D blocks from the input volume and perform multi-headed self-attention on a sequence of their linear projections. MINiT, on the other hand, treats each of the non-overlapping 3D blocks of the input MRI as its own instance, splitting it further into non-overlapping 3D patches, on which multi-headed self-attention is computed. As a proof-of-concept, we evaluate the efficacy of our model by training it to identify sex from T1w-MRIs of two public datasets: Adolescent Brain Cognitive Development (ABCD) and the National Consortium on Alcohol and Neurodevelopment in Adolescence (NCANDA). The learned attention maps highlight voxels contributing to identifying sex differences in brain morphometry. The code is available at https://github.com/singlaayush/MINIT.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
eBook
USD 39.99
Price excludes VAT (USA)
Softcover Book
USD 54.99
Price excludes VAT (USA)

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Abnar, S., Zuidema, W.: Quantifying attention flow in transformers. arXiv:2005.00928 (2020)

  2. Adeli, E., et al.: Deep learning identifies morphological determinants of sex differences in the pre-adolescent brain. Neuroimage 223, 117293 (2020)

    Google Scholar 

  3. Agarap, A.F.: Deep learning using rectified linear units (ReLU). arXiv:1803.08375 (2018)

  4. Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučić, M., Schmid, C.: ViViT: a video vision transformer. In: ICCV, pp. 6836–6846 (2021)

    Google Scholar 

  5. Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. arXiv:1607.06450 (2016)

  6. Brown, S.A., et al.: The national consortium on alcohol and neurodevelopment in adolescence (NCANDA): a multisite study of adolescent development and substance use. JSAD 76(6), 895–908 (2015)

    Google Scholar 

  7. Carbonneau, M.A., Cheplygina, V., Granger, E., Gagnon, G.: Multiple instance learning: a survey of problem characteristics and applications. Pattern Recogn. 77, 329–353 (2018)

    Article  Google Scholar 

  8. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 213–229. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_13

    Chapter  Google Scholar 

  9. Casey, B., et al.: The adolescent brain cognitive development (ABCD) study: imaging acquisition across 21 sites. Dev. Cogn. Neurosci. 32, 43–54 (2018)

    Article  Google Scholar 

  10. Chen, J., et al.: TransUNet: transformers make strong encoders for medical image segmentation. arXiv:2102.04306 (2021)

  11. Chen, X., Hsieh, C.J., Gong, B.: When vision transformers outperform ResNets without pre-training or strong data augmentations. arXiv:2106.01548 (2021)

  12. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805 (2018)

  13. Dosovitskiy, A., et al.: An image is worth \(16 \times 16\) words: transformers for image recognition at scale. arXiv:2010.11929 (2020)

  14. Efraimidis, P.S., Spirakis, P.G.: Weighted random sampling with a reservoir. Inf. Process. Lett. 97(5), 181–185 (2006)

    Article  MathSciNet  Google Scholar 

  15. Goyal, P., et al.: Accurate, large minibatch SGD: training ImageNet in 1 hour. arXiv:1706.02677 (2017)

  16. Han, K., Xiao, A., Wu, E., Guo, J., Xu, C., Wang, Y.: Transformer in transformer. In: NeurIPS, vol. 34 (2021)

    Google Scholar 

  17. Hänggi, J., Buchmann, A., Mondadori, C.R., Henke, K., Jäncke, L., Hock, C.: Sexual dimorphism in the parietal substrate associated with visuospatial cognition independent of general intelligence. JoCN 22(1), 139–155 (2010)

    Google Scholar 

  18. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)

    Google Scholar 

  19. Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: ICML, pp. 448–456. PMLR (2015)

    Google Scholar 

  20. Jun, E., Jeong, S., Heo, D.W., Suk, H.I.: Medical transformer: universal brain encoder for 3D MRI analysis. arXiv:2104.13633 (2021)

  21. Kaczkurkin, A.N., Raznahan, A., Satterthwaite, T.D.: Sex differences in the developing brain: insights from multimodal neuroimaging. Neuropsychopharmacology 44(1), 71–85 (2019)

    Article  Google Scholar 

  22. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv:1412.6980 (2014)

  23. Larrazabal, A.J., Nieto, N., Peterson, V., Milone, D.H., Ferrante, E.: Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis. Proc. Natl. Acad. Sci. 117(23), 12592–12594 (2020)

    Article  Google Scholar 

  24. Liu, M., Zhang, J., Adeli, E., Shen, D.: Landmark-based deep multi-instance learning for brain disease diagnosis. Med. Image Anal. 43, 157–168 (2018)

    Article  Google Scholar 

  25. Loshchilov, I., Hutter, F.: SGDR: stochastic gradient descent with warm restarts. arXiv:1608.03983 (2016)

  26. Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv:1711.05101 (2017)

  27. Malkiel, I., Rosenman, G., Wolf, L., Hendler, T.: Pre-training and fine-tuning transformers for FMRI prediction tasks. arXiv:2112.05761 (2021)

  28. Ouyang, J., et al.: Longitudinal pooling & consistency regularization to model disease progression from MRIs. IEEE J. Biomed. Health Inform. 25(6), 2082–2092 (2020)

    Article  Google Scholar 

  29. Pohl, K.M., et al.: The ‘NCANDA_PUBLIC_6Y_STRUCTURAL_V01’ data release of the National Consortium on Alcohol and NeuroDevelopment in Adolescence (NCANDA). Sage Bionetworks Synapse (2022). https://doi.org/10.7303/syn32773308

  30. Pramono, R.R.A., Chen, Y.T., Fang, W.H.: Hierarchical self-attention network for action localization in videos. In: ICCV, pp. 61–70 (2019)

    Google Scholar 

  31. Sacher, J., Neumann, J., Okon-Singer, H., Gotowiec, S., Villringer, A.: Sexual dimorphism in the human brain: evidence from neuroimaging. JMRI 31(3), 366–375 (2013)

    Google Scholar 

  32. Shazeer, N.: GLU variants improve transformer. arXiv:2002.05202 (2020)

  33. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. JMLR 15(1), 1929–1958 (2014)

    MathSciNet  MATH  Google Scholar 

  34. Steiner, A., Kolesnikov, A., Zhai, X., Wightman, R., Uszkoreit, J., Beyer, L.: How to train your ViT? data, augmentation, and regularization in vision transformers. arXiv:2106.10270 (2021)

  35. Su, J., Lu, Y., Pan, S., Wen, B., Liu, Y.: RoFormer: enhanced transformer with rotary position embedding. arXiv:2104.09864 (2021)

  36. Van Putten, M.J., Olbrich, S., Arns, M.: Predicting sex from brain rhythms with deep learning. Sci. Rep. 8(1), 1–7 (2018)

    Google Scholar 

  37. Vaswani, A., et al.: Attention is all you need. In: NeurIPS, vol. 30 (2017)

    Google Scholar 

  38. Wang, H., Zhu, Y., Adam, H., Yuille, A., Chen, L.C.: MaX-DeepLab: end-to-end panoptic segmentation with mask transformers. In: CVPR, pp. 5463–5474 (2021)

    Google Scholar 

  39. Xin, J., Zhang, Y., Tang, Y., Yang, Y.: Brain differences between men and women: evidence from deep learning. Front. Neurosci. 13, 185 (2019)

    Article  Google Scholar 

  40. Yun, S., Han, D., Oh, S.J., Chun, S., Choe, J., Yoo, Y.: CutMix: regularization strategy to train strong classifiers with localizable features. In: ICCV, pp. 6023–6032 (2019)

    Google Scholar 

  41. Zhang, B., et al.: Co-training transformer with videos and images improves action recognition. arXiv:2112.07175 (2021)

  42. Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: Mixup: beyond empirical risk minimization. arXiv:1710.09412 (2017)

  43. Zhao, H., Jiang, L., Jia, J., Torr, P.H., Koltun, V.: Point transformer. In: ICCV, pp. 16259–16268 (2021)

    Google Scholar 

  44. Zhao, Q., Adeli, E., Pfefferbaum, A., Sullivan, E.V., Pohl, K.M.: Confounder-aware visualization of ConvNets. In: Suk, H.-I., Liu, M., Yan, P., Lian, C. (eds.) MLMI 2019. LNCS, vol. 11861, pp. 328–336. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32692-0_38

    Chapter  Google Scholar 

Download references

Acknowledgements

This work was partially supported by the NIH grants AA021697 and AA028840, and the Stanford Institute for Human-centered Artificial Intelligence (HAI) Google Cloud Credits (GCP) credits.

Author information

Authors and Affiliations

  1. Stanford University, Stanford, CA, 94305, USA

    Ayush Singla, Qingyu Zhao, Daniel K. Do, Yuyin Zhou, Kilian M. Pohl & Ehsan Adeli

  2. University of California Santa Cruz, Santa Cruz, CA, 95064, USA

    Yuyin Zhou

Authors
  1. Ayush Singla
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qingyu Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Daniel K. Do
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yuyin Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kilian M. Pohl
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ehsan Adeli
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ayush Singla .

Editor information

Editors and Affiliations

  1. Istanbul Technical University, Istanbul, Turkey

    Islem Rekik

  2. Stanford University, Stanford, CA, USA

    Ehsan Adeli

  3. Daegu Gyeongbuk Institute of Science and Technology, Daegu, Korea (Republic of)

    Sang Hyun Park

  4. IBM Research - Africa, Nairobi, Kenya

    Celia Cintas

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Singla, A., Zhao, Q., Do, D.K., Zhou, Y., Pohl, K.M., Adeli, E. (2022). Multiple Instance Neuroimage Transformer. In: Rekik, I., Adeli, E., Park, S.H., Cintas, C. (eds) Predictive Intelligence in Medicine. PRIME 2022. Lecture Notes in Computer Science, vol 13564. Springer, Cham. https://doi.org/10.1007/978-3-031-16919-9_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-16919-9_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-16918-2

  • Online ISBN: 978-3-031-16919-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation