Marek S, Tervo-Clemmens B, Calabro FJ, Montez DF, Kay BP, Hatoum AS, Donohue MR, Foran W, Miller RL, Hendrickson TJ, Malone SM, Kandala S, Feczko E, Miranda-Dominguez O, Graham AM, Earl EA, Perrone AJ, Cordova M, Doyle O, Moore LA, Conan GM, Uriarte J, Snider K, Lynch BJ, Wilgenbusch JC, Pengo T, Tam A, Chen J, Newbold DJ, Zheng A, Seider NA, Van AN, Metoki A, Chauvin RJ, Laumann TO, Greene DJ, Petersen SE, Garavan H, Thompson WK, Nichols TE, Yeo BT, Barch DM, Luna B, Fair DA, Dosenbach NU. Reproducible brain-wide association studies require thousands of individuals. Nature. 2022 Mar 16; PubMed.

Recommends

Please login to recommend the paper.

Comments

  1. This is an interesting paper. One overall goal of our field is to determine how we can use brain MRI information, together with other information about the participants, to gain knowledge concerning the brain regions which are involved in specific functions, and how they are affected by disease, and by treatments. Many of the reports made over the years have not been replicated. This very careful study demonstrates that the more data that is available, the more robust the findings are. This confirms, supports, and emphasizes the suspicion that many investigators have had that the results of small studies might lead to chance findings, or findings limited to a population subset, which would not stand the test of time. Therefore more large-scale studies, and data-sharing which facilitates amalgamation of data from separate studies, are to be encouraged.

    An important issue, not directly addressed by this paper, is the “generalizability” of such findings. The vast majority of brain-imaging studies come from projects in which the participants volunteer to participate. Such studies tend to over-select from populations who have the interest and time to participate in research, that is, people with higher socioeconomic status, who are more affluent. To obtain truly generalizable findings, we need more large-scale brain imaging studies from participants from epidemiologically sampled populations such as the Health and Retirement study, the Framingham study, or the Mayo Clinic Study of Aging.

    View all comments by Michael Weiner

  2. Marek et al. are to be commended on their timely reminder of the large sample sizes often required to obtain reliable and reproducible findings in brain imaging studies. With this in mind, we formed the ENIGMA Consortium over a decade ago, and a recent special issue of Human Brain Mapping (Thompson et al., 2022) showcases more than 30 papers pooling brain-imaging data from across the world, in diverse populations from over 45 countries, often yielding the largest neuroimaging studies of over 30 major brain diseases and conditions. What we learned regarding reproducibility may offer some relief from the reproducibility crisis, and the concerns expressed by Marek et al. ENIGMA has benefited from the power of global collaboration to put the neuroimaging of disease on a firmer footing. It offers a solution for those collecting smaller cohort studies to team up with others worldwide to identify how robust their findings are, but also to discover crucial modulating factors that influence disease risk and progression in the brain. Some examples of what we learned by pooling tens of thousands of brain scans worldwide are below.

    1. There are robust brain correlates of major brain diseases. By pooling brain disease data from more than 45 countries and more than 30 disorders, ENIGMA published the largest neuroimaging studies of Parkinson’s disease, epilepsy, schizophrenia, bipolar disorder, major depressive disorder, and PTSD (reviewed in Thompson et al., 2020). The general message is that the characteristic signatures of brain disease are highly robust and reproducible worldwide. Independent consortia in Japan are finding almost the exact same patterns of disease effects on the brain, and even the same rankings for brain biomarkers from MRI and DTI (Koshiyama et al., 2022). To first order, this agreement is remarkable and has brought robustness and rigor to fields where small effects are the rule, and where modulators of disease are important to identify.
    2. Do we always need thousands of subjects to find anything reliable? No. ENIGMA’s studies allowed us to go back and see which of the findings in the largest studies of each disease would have been found if groups had just published on their own, and what factors affect the discoverability and consistency of disease biomarkers worldwide. In neurological disorders such as Parkinson’s disease, epilepsy, and ataxia, extremely robust effects are found, replicating and extending effects found in smaller studies of a few hundred people, but often revealing stage-specific patterns and unsuspected subtypes. Similarly, ENIGMA’s studies show huge effects on the brain for many rare genetic variants—cohorts were able to discover these effects in a few dozen cases, but larger studies were able to verify and confirm them. In the psychiatric neuroimaging field, ENIGMA’s largest neuroimaging studies to date of anorexia, schizophrenia, bipolar disorder, and PTSD all show robust effects on brain MRI, DTI, and resting state fMRI. Pooling tens of thousands of MRIs across sites and countries helped to both discover and verify patterns that were once hotly debated or controversial. By pooling data from hundreds of labs, we found that the patterns of structural brain asymmetry are consistent worldwide (Kong et al., 2022), but are subtly altered in schizophrenia (Schijven et al., 2022) and autism (Postema et al., 2019)—supporting a hypothesis that had been controversial since the 1970s. Other hypotheses that have recently been resolved with very large datasets include the presence of subtle brain differences in schizotypy (Kirschner et al., 2021), a mysterious protective pattern seen in unaffected siblings of people with bipolar disorder (de Zwarte et al., 2022), and the association of robust brain differences in hippocampal volume with epigenetic variations such as methylation (Jia et al., 2021). 
    3. Modulators of disease and ancestral diversity. Sometimes disease effects do differ in important ways across populations—depending on prevailing risk factors and a person’s ancestry, effects found in one population may not always be found in another. In ENIGMA’s studies of OCD, large-scale data pooling (Piras et al., 2021) resolved medication, duration of illness, and age of onset as largely accounting for the differences seen in the reported brain effects across cohorts worldwide—a field that had once been regarded as rife with inconsistencies. In ADHD, we found a subtle and shifting pattern of effects on the brain that varies with age—in a remarkable study pooling data from over 150 cohorts worldwide and comparing effects across OCD, ASD and ADHD (Boedhoe et al. 2020). 

    As noted by Marek et al., special design considerations are required—including very large samples (thousands or tens of thousands) when very high dimensional datasets are screened for associations. To avoid the risk of spurious associations when large sets of biomarkers are screened, a split sample—or an independent replication sample—is often used. Before anyone complains about the cost, great success has been achieved by pooling existing datasets across the world, after careful coordination and harmonization. This process is eased in ENIGMA by using standardized workflows that process datasets consistently, avoiding the methodological heterogeneity that often leads to discrepancies, known and unknown, in the published literature.

    When thousands or even millions of biomarkers are screened for associations at the same time, the risk of false positive associations increases; as Smith and Nichols (2018) note, this case is widely recognized in brain mapping, and there are already many widely used methods to mitigate it. Methods to meta-analyze brain maps and aggregate evidence across multiple brain mapping studies have been highly successful (BrainMap, Neurosynth, SDM, and meta-TBM). In genetics, the same multiple testing issues have led to the coordinated analysis of large samples to detect associations between genomic variation and key brain biomarkers. In our own GWAS studies, we screened over a million genomic markers to identify common genetic variants associated with brain measures derived from MRI (Grasby et al., 2020), and even with the rates of brain atrophy over time (Brouwer et al., 2022), offering key new genomic targets for drug discovery. These genetic risk effects are individually small but highly robust—much like the genetic risk loci for Alzheimer’s disease beyond APOE, which required thousands of subjects to discover, but much smaller samples to verify (when the penalty of multiple testing is reduced in a follow-up, independent replication phase). One great success of Alzheimer’s research and degenerative disease research in general is the quest to identify disease-related markers in the genome, and in brain images. Such efforts have often disentangled the heterogeneity of disease, and discovered disease subtypes that can be dealt with in a more personalized and targeted way.

    We propose the following remedies for the need for large samples in neuroimaging research. First of all, large samples are not always required: They are only required when the effects are small, or if a large number of associations are being tested at the same time, or if key modulators of the disease effect are suspected. All these situations will typically lead to different findings across cohorts, when cohorts are small (under 100 subjects). If the effects are small, or multiple tests are performed, or modulators are important (e.g., sex differences, or ancestry effects), then clubbing together as consortia will offer the power to discover and verify effects, and find key modulators. We have shown this repeatedly, in over 100 papers by ENIGMA over the last decade.

    The neurology and psychiatry fields have been immensely successful in this regard, with vast international efforts in genomics (ADSP, ADGC, PGC), and neuroimaging (ENIGMA). We thank Marek et al. for their call to action for better reproducibility in brain imaging, and we encourage teams worldwide to unite to discover and verify factors that resist the devastating diseases that affect us all today.

    References:

    Thompson PM, Jahanshad N, Schmaal L, Turner JA, Winkler AM, Thomopoulos SI, Egan GF, Kochunov P. The Enhancing NeuroImaging Genetics through Meta-Analysis Consortium: 10 Years of Global Collaborations in Human Brain Mapping. Hum Brain Mapp. 2022 Jan;43(1):15-22. Epub 2021 Oct 6 PubMed.

    Thompson PM, Jahanshad N, Ching CR, Salminen LE, Thomopoulos SI, Bright J, Baune BT, Bertolín S, Bralten J, Bruin WB, Bülow R, Chen J, Chye Y, Dannlowski U, de Kovel CG, Donohoe G, Eyler LT, Faraone SV, Favre P, Filippi CA, Frodl T, Garijo D, Gil Y, Grabe HJ, Grasby KL, Hajek T, Han LK, Hatton SN, Hilbert K, Ho TC, Holleran L, Homuth G, Hosten N, Houenou J, Ivanov I, Jia T, Kelly S, Klein M, Kwon JS, Laansma MA, Leerssen J, Lueken U, Nunes A, Neill JO, Opel N, Piras F, Piras F, Postema MC, Pozzi E, Shatokhina N, Soriano-Mas C, Spalletta G, Sun D, Teumer A, Tilot AK, Tozzi L, van der Merwe C, Van Someren EJ, van Wingen GA, Völzke H, Walton E, Wang L, Winkler AM, Wittfeld K, Wright MJ, Yun JY, Zhang G, Zhang-James Y, Adhikari BM, Agartz I, Aghajani M, Aleman A, Althoff RR, Altmann A, Andreassen OA, Baron DA, Bartnik-Olson BL, Marie Bas-Hoogendam J, Baskin-Sommers AR, Bearden CE, Berner LA, Boedhoe PS, Brouwer RM, Buitelaar JK, Caeyenberghs K, Cecil CA, Cohen RA, Cole JH, Conrod PJ, De Brito SA, de Zwarte SM, Dennis EL, Desrivieres S, Dima D, Ehrlich S, Esopenko C, Fairchild G, Fisher SE, Fouche JP, Francks C, Frangou S, Franke B, Garavan HP, Glahn DC, Groenewold NA, Gurholt TP, Gutman BA, Hahn T, Harding IH, Hernaus D, Hibar DP, Hillary FG, Hoogman M, Hulshoff Pol HE, Jalbrzikowski M, Karkashadze GA, Klapwijk ET, Knickmeyer RC, Kochunov P, Koerte IK, Kong XZ, Liew SL, Lin AP, Logue MW, Luders E, Macciardi F, Mackey S, Mayer AR, McDonald CR, McMahon AB, Medland SE, Modinos G, Morey RA, Mueller SC, Mukherjee P, Namazova-Baranova L, Nir TM, Olsen A, Paschou P, Pine DS, Pizzagalli F, Rentería ME, Rohrer JD, Sämann PG, Schmaal L, Schumann G, Shiroishi MS, Sisodiya SM, Smit DJ, Sønderby IE, Stein DJ, Stein JL, Tahmasian M, Tate DF, Turner JA, van den Heuvel OA, van der Wee NJ, van der Werf YD, van Erp TG, van Haren NE, van Rooij D, van Velzen LS, Veer IM, Veltman DJ, Villalon-Reina JE, Walter H, Whelan CD, Wilde EA, Zarei M, Zelman V, ENIGMA Consortium. ENIGMA and global neuroscience: A decade of large-scale studies of the brain in health and disease across more than 40 countries. Transl Psychiatry. 2020 Mar 20;10(1):100. PubMed.

    Koshiyama D, Miura K, Nemoto K, Okada N, Matsumoto J, Fukunaga M, Hashimoto R. Neuroimaging studies within Cognitive Genetics Collaborative Research Organization aiming to replicate and extend works of ENIGMA. Hum Brain Mapp. 2022 Jan;43(1):182-193. Epub 2020 Jun 5 PubMed.

    Kong XZ, Postema MC, Guadalupe T, de Kovel C, Boedhoe PS, Hoogman M, Mathias SR, van Rooij D, Schijven D, Glahn DC, Medland SE, Jahanshad N, Thomopoulos SI, Turner JA, Buitelaar J, van Erp TG, Franke B, Fisher SE, van den Heuvel OA, Schmaal L, Thompson PM, Francks C. Mapping brain asymmetry in health and disease through the ENIGMA consortium. Hum Brain Mapp. 2022 Jan;43(1):167-181. Epub 2020 May 18 PubMed.

    Schijven D, Postema MC, Fukunaga M, Matsumoto J, Miura K, de Zwarte SMC, van Haren NEM, Cahn W, Pol HEH, Kahn RS, Ayesa-Arriola R, Ortiz-García de la Foz V, Tordesillas-Gutierrez D, Vázquez-Bourgon J, Crespo-Facorro B, Alnæs D, Dahl A, Westlye LT, Agartz I, Andreassen OA, Jönsson EG, Kochunov P, Bruggemann JM, Catts SV, Michie PT, Mowry BJ, Quidé Y, Rasser PE, Schall U, Scott RJ, Carr VJ, Green MJ, Henskens FA, Loughland CM, Pantelis C, Weickert CS, Weickert TW, de Haan L, Brosch K, Pfarr JK, Ringwald KG, Stein F, Jansen A, Kircher TTJ, Nenadić I, Krämer B, Gruber O, Satterthwaite TD, Bustillo J, Mathalon DH, Preda A, Calhoun VD, Ford JM, Potkin SG, Chen J, Tan Y, Wang Z, Xiang H, Fan F, Ehrlich S, Fuentes-Claramonte P, Garcia-Leon MA, Guerrero-Pedraza A, Salvador R, Sarró S, Pomarol-Clotet E, Ciullo V, Piras F, Vecchio D, Banaj N, Spalletta G, Michielse S, van Amelsvoort T, Dickie EW, Voineskos AN, Sim K, Ciufolini S, Dazzan P, Murray RM, Kim WS, Chung YC, Andreou C, Schmidt A, Borgwardt S, McIntosh AM, Whalley HC, Lawrie SM, du Plessis S, Luckhoff HK, Scheffler F, Emsley R, Grotegerd D, Lencer R, Dannlowski U, Edmond JT, Rootes-Murdy K, Stephen JM, Mayer AR, Antonucci LA, Fazio L, Pergola G, Bertolino A, Díaz-Caneja CM, Janssen J, Lois NG, Arango C, Tomyshev AS, Lebedeva I, Cervenka S, Sellgren CM, Georgiadis F, Kirschner M, Kaiser S, Hajek T, Skoch A, Spaniel F, Kim M, Kwak YB, Oh S, Kwon JS, James A, Bakker G, Knöchel C, Stäblein M, Oertel V, Uhlmann A, Howells FM, Stein DJ, Temmingh HS, Diaz-Zuluaga AM, Pineda-Zapata JA, López-Jaramillo C, Homan ,S Ji E, Surbeck W, Homan P, Fisher SE, Franke B, Glahn DC, Gur RC, Hashimoto R, Jahanshad N, Luders E, Medland SE, Thompson PM, Turner JA, van Erp TGM, Francks C. Large-scale analysis of structural brain asymmetries in schizophrenia via the ENIGMA consortium. medRxiv, March 11, 2022. medRxiv

    Postema MC, van Rooij D, Anagnostou E, Arango C, Auzias G, Behrmann M, Filho GB, Calderoni S, Calvo R, Daly E, Deruelle C, Di Martino A, Dinstein I, Duran FL, Durston S, Ecker C, Ehrlich S, Fair D, Fedor J, Feng X, Fitzgerald J, Floris DL, Freitag CM, Gallagher L, Glahn DC, Gori I, Haar S, Hoekstra L, Jahanshad N, Jalbrzikowski M, Janssen J, King JA, Kong XZ, Lazaro L, Lerch JP, Luna B, Martinho MM, McGrath J, Medland SE, Muratori F, Murphy CM, Murphy DG, O'Hearn K, Oranje B, Parellada M, Puig O, Retico A, Rosa P, Rubia K, Shook D, Taylor MJ, Tosetti M, Wallace GL, Zhou F, Thompson PM, Fisher SE, Buitelaar JK, Francks C. Author Correction: Altered structural brain asymmetry in autism spectrum disorder in a study of 54 datasets. Nat Commun. 2021 Dec 8;12(1):7260. PubMed.

    Kirschner M, Hodzic-Santor B, Antoniades M, Nenadic I, Kircher T, Krug A, Meller T, Grotegerd D, Fornito A, Arnatkeviciute A, Bellgrove MA, Tiego J, Dannlowski U, Koch K, Hülsmann C, Kugel H, Enneking V, Klug M, Leehr EJ, Böhnlein J, Gruber M, Mehler D, DeRosse P, Moyett A, Baune BT, Green M, Quidé Y, Pantelis C, Chan R, Wang Y, Ettinger U, Debbané M, Derome M, Gaser C, Besteher B, Diederen K, Spencer TJ, Fletcher P, Rössler W, Smigielski L, Kumari V, Premkumar P, Park HR, Wiebels K, Lemmers-Jansen I, Gilleen J, Allen P, Kozhuharova P, Marsman JB, Lebedeva I, Tomyshev A, Mukhorina A, Kaiser S, Fett AK, Sommer I, Schuite-Koops S, Paquola C, Larivière S, Bernhardt B, Dagher A, Grant P, van Erp TG, Turner JA, Thompson PM, Aleman A, Modinos G. Cortical and subcortical neuroanatomical signatures of schizotypy in 3004 individuals assessed in a worldwide ENIGMA study. Mol Psychiatry. 2021 Oct 27; PubMed.

    de Zwarte SM, Brouwer RM, Agartz I, Alda M, Alonso-Lana S, Bearden CE, Bertolino A, Bonvino A, Bramon E, Buimer EE, Cahn W, Canales-Rodríguez EJ, Cannon DM, Cannon TD, Caseras X, Castro-Fornieles J, Chen Q, Chung Y, De la Serna E, Del Mar Bonnin C, Demro C, Di Giorgio A, Doucet GE, Eker MC, Erk S, Fatjó-Vilas M, Fears SC, Foley SF, Frangou S, Fullerton JM, Glahn DC, Goghari VM, Goikolea JM, Goldman AL, Gonul AS, Gruber O, Hajek T, Hawkins EL, Heinz A, Hidiroglu Ongun C, Hillegers MH, Houenou J, Hulshoff Pol HE, Hultman CM, Ingvar M, Johansson V, Jönsson EG, Kane F, Kempton MJ, Koenis MM, Kopecek M, Krämer B, Lawrie SM, Lenroot RK, Marcelis M, Mattay VS, McDonald C, Meyer-Lindenberg A, Michielse S, Mitchell PB, Moreno D, Murray RM, Mwangi B, Nabulsi L, Newport J, Olman CA, van Os J, Overs BJ, Ozerdem A, Pergola G, Picchioni MM, Piguet C, Pomarol-Clotet E, Radua J, Ramsay IS, Richter A, Roberts G, Salvador R, Saricicek Aydogan A, Sarró S, Schofield PR, Simsek EM, Simsek F, Soares JC, Sponheim SR, Sugranyes G, Toulopoulou T, Tronchin G, Vieta E, Walter H, Weinberger DR, Whalley HC, Wu MJ, Yalin N, Andreassen OA, Ching CR, Thomopoulos SI, van Erp TG, Jahanshad N, Thompson PM, Kahn RS, van Haren NE. Intelligence, educational attainment, and brain structure in those at familial high-risk for schizophrenia or bipolar disorder. Hum Brain Mapp. 2022 Jan;43(1):414-430. Epub 2020 Oct 7 PubMed.

    Jia T, Chu C, Liu Y, van Dongen J, Papastergios E, Armstrong NJ, Bastin ME, Carrillo-Roa T, den Braber A, Harris M, Jansen R, Liu J, Luciano M, Ori AP, Roiz Santiañez R, Ruggeri B, Sarkisyan D, Shin J, Sungeun K, Tordesillas Gutiérrez D, Van't Ent D, Ames D, Artiges E, Bakalkin G, Banaschewski T, Bokde AL, Brodaty H, Bromberg U, Brouwer R, Büchel C, Burke Quinlan E, Cahn W, de Zubicaray GI, Ehrlich S, Ekström TJ, Flor H, Fröhner JH, Frouin V, Garavan H, Gowland P, Heinz A, Hoare J, Ittermann B, Jahanshad N, Jiang J, Kwok JB, Martin NG, Martinot JL, Mather KA, McMahon KL, McRae AF, Nees F, Papadopoulos Orfanos D, Paus T, Poustka L, Sämann PG, Schofield PR, Smolka MN, Stein DJ, Strike LT, Teeuw J, Thalamuthu A, Trollor J, Walter H, Wardlaw JM, Wen W, Whelan R, Apostolova LG, Binder EB, Boomsma DI, Calhoun V, Crespo-Facorro B, Deary IJ, Hulshoff Pol H, Ophoff RA, Pausova Z, Sachdev PS, Saykin A, Wright MJ, Thompson PM, Schumann G, Desrivières S. Epigenome-wide meta-analysis of blood DNA methylation and its association with subcortical volumes: findings from the ENIGMA Epigenetics Working Group. Mol Psychiatry. 2021 Aug;26(8):3884-3895. Epub 2019 Dec 6 PubMed.

    Piras F, Piras F, Abe Y, Agarwal SM, Anticevic A, Ameis S, Arnold P, Banaj N, Bargalló N, Batistuzzo MC, Benedetti F, Beucke JC, Boedhoe PS, Bollettini I, Brem S, Calvo A, Cho KI, Ciullo V, Dallaspezia S, Dickie E, Ely BA, Fan S, Fouche JP, Gruner P, Gürsel DA, Hauser T, Hirano Y, Hoexter MQ, Iorio M, James A, Reddy YC, Kaufmann C, Koch K, Kochunov P, Kwon JS, Lazaro L, Lochner C, Marsh R, Nakagawa A, Nakamae T, Narayanaswamy JC, Sakai Y, Shimizu E, Simon D, Simpson HB, Soreni N, Stämpfli P, Stern ER, Szeszko P, Takahashi J, Venkatasubramanian G, Wang Z, Yun JY, ENIGMA OCD Working Group, Stein DJ, Jahanshad N, Thompson PM, van den Heuvel OA, Spalletta G. White matter microstructure and its relation to clinical features of obsessive-compulsive disorder: findings from the ENIGMA OCD Working Group. Transl Psychiatry. 2021 Mar 17;11(1):173. PubMed.

    Boedhoe PS, van Rooij D, Hoogman M, Twisk JW, Schmaal L, Abe Y, Alonso P, Ameis SH, Anikin A, Anticevic A, Arango C, Arnold PD, Asherson P, Assogna F, Auzias G, Banaschewski T, Baranov A, Batistuzzo MC, Baumeister S, Baur-Streubel R, Behrmann M, Bellgrove MA, Benedetti F, Beucke JC, Biederman J, Bollettini I, Bose A, Bralten J, Bramati IE, Brandeis D, Brem S, Brennan BP, Busatto GF, Calderoni S, Calvo A, Calvo R, Castellanos FX, Cercignani M, Chaim-Avancini TM, Chantiluke KC, Cheng Y, Cho KI, Christakou A, Coghill D, Conzelmann A, Cubillo AI, Dale AM, Dallaspezia S, Daly E, Denys D, Deruelle C, Di Martino A, Dinstein I, Doyle AE, Durston S, Earl EA, Ecker C, Ehrlich S, Ely BA, Epstein JN, Ethofer T, Fair DA, Fallgatter AJ, Faraone SV, Fedor J, Feng X, Feusner JD, Fitzgerald J, Fitzgerald KD, Fouche JP, Freitag CM, Fridgeirsson EA, Frodl T, Gabel MC, Gallagher L, Gogberashvili T, Gori I, Gruner P, Gürsel DA, Haar S, Haavik J, Hall GB, Harrison NA, Hartman CA, Heslenfeld DJ, Hirano Y, Hoekstra PJ, Hoexter MQ, Hohmann S, Høvik MF, Hu H, Huyser C, Jahanshad N, Jalbrzikowski M, James A, Janssen J, Jaspers-Fayer F, Jernigan TL, Kapilushniy D, Kardatzki B, Karkashadze G, Kathmann N, Kaufmann C, Kelly C, Khadka S, King JA, Koch K, Kohls G, Konrad K, Kuno M, Kuntsi J, Kvale G, Kwon JS, Lázaro L, Lera-Miguel S, Lesch KP, Hoekstra L, Liu Y, Lochner C, Louza MR, Luna B, Lundervold AJ, Malpas CB, Marques P, Marsh R, Martínez-Zalacaín I, Mataix-Cols D, Mattos P, McCarthy H, McGrath J, Mehta MA, Menchón JM, Mennes M, Martinho MM, Moreira PS, Morer A, Morgado P, Muratori F, Murphy CM, Murphy DG, Nakagawa A, Nakamae T, Nakao T, Namazova-Baranova L, Narayanaswamy JC, Nicolau R, Nigg JT, Novotny SE, Nurmi EL, Weiss EO, O'Gorman Tuura RL, O'Hearn K, O'Neill J, Oosterlaan J, Oranje B, Paloyelis Y, Parellada M, Pauli P, Perriello C, Piacentini J, Piras F, Piras F, Plessen KJ, Puig O, Ramos-Quiroga JA, Reddy YC, Reif A, Reneman L, Retico A, Rosa PG, Rubia K, Rus OG, Sakai Y, Schrantee A, Schwarz L, Schweren LJ, Seitz J, Shaw P, Shook D, Silk TJ, Simpson HB, Skokauskas N, Soliva Vila JC, Solovieva A, Soreni N, Soriano-Mas C, Spalletta G, Stern ER, Stevens MC, Stewart SE, Sudre G, Szeszko PR, Tamm L, Taylor MJ, Tolin DF, Tosetti M, Tovar-Moll F, Tsuchiyagaito A, van Erp TG, van Wingen GA, Vance A, Venkatasubramanian G, Vilarroya O, Vives-Gilabert Y, von Polier GG, Walitza S, Wallace GL, Wang Z, Wolfers T, Yoncheva YN, Yun JY, Zanetti MV, Zhou F, Ziegler GC, Zierhut KC, Zwiers MP, ENIGMA ADHD working group, ENIGMA ASD working group, ENIGMA OCD working group, Thompson PM, Stein DJ, Buitelaar J, Franke B, van den Heuvel OA. Subcortical Brain Volume, Regional Cortical Thickness, and Cortical Surface Area Across Disorders: Findings From the ENIGMA ADHD, ASD, and OCD Working Groups. Am J Psychiatry. 2020 Sep 1;177(9):834-843. Epub 2020 Jun 16 PubMed.

    Smith SM, Nichols TE. Statistical Challenges in "Big Data" Human Neuroimaging. Neuron. 2018 Jan 17;97(2):263-268. PubMed.

    Grasby KL, Jahanshad N, Painter JN, Colodro-Conde L, Bralten J, Hibar DP, Lind PA, Pizzagalli F, Ching CR, McMahon MA, Shatokhina N, Zsembik LC, Thomopoulos SI, Zhu AH, Strike LT, Agartz I, Alhusaini S, Almeida MA, Alnæs D, Amlien IK, Andersson M, Ard T, Armstrong NJ, Ashley-Koch A, Atkins JR, Bernard M, Brouwer RM, Buimer EE, Bülow R, Bürger C, Cannon DM, Chakravarty M, Chen Q, Cheung JW, Couvy-Duchesne B, Dale AM, Dalvie S, de Araujo TK, de Zubicaray GI, de Zwarte SM, den Braber A, Doan NT, Dohm K, Ehrlich S, Engelbrecht HR, Erk S, Fan CC, Fedko IO, Foley SF, Ford JM, Fukunaga M, Garrett ME, Ge T, Giddaluru S, Goldman AL, Green MJ, Groenewold NA, Grotegerd D, Gurholt TP, Gutman BA, Hansell NK, Harris MA, Harrison MB, Haswell CC, Hauser M, Herms S, Heslenfeld DJ, Ho NF, Hoehn D, Hoffmann P, Holleran L, Hoogman M, Hottenga JJ, Ikeda M, Janowitz D, Jansen IE, Jia T, Jockwitz C, Kanai R, Karama S, Kasperaviciute D, Kaufmann T, Kelly S, Kikuchi M, Klein M, Knapp M, Knodt AR, Krämer B, Lam M, Lancaster TM, Lee PH, Lett TA, Lewis LB, Lopes-Cendes I, Luciano M, Macciardi F, Marquand AF, Mathias SR, Melzer TR, Milaneschi Y, Mirza-Schreiber N, Moreira JC, Mühleisen TW, Müller-Myhsok B, Najt P, Nakahara S, Nho K, Olde Loohuis LM, Orfanos DP, Pearson JF, Pitcher TL, Pütz B, Quidé Y, Ragothaman A, Rashid FM, Reay WR, Redlich R, Reinbold CS, Repple J, Richard G, Riedel BC, Risacher SL, Rocha CS, Mota NR, Salminen L, Saremi A, Saykin AJ, Schlag F, Schmaal L, Schofield PR, Secolin R, Shapland CY, Shen L, Shin J, Shumskaya E, Sønderby IE, Sprooten E, Tansey KE, Teumer A, Thalamuthu A, Tordesillas-Gutiérrez D, Turner JA, Uhlmann A, Vallerga CL, van der Meer D, van Donkelaar MM, van Eijk L, van Erp TG, van Haren NE, van Rooij D, van Tol MJ, Veldink JH, Verhoef E, Walton E, Wang M, Wang Y, Wardlaw JM, Wen W, Westlye LT, Whelan CD, Witt SH, Wittfeld K, Wolf C, Wolfers T, Wu JQ, Yasuda CL, Zaremba D, Zhang Z, Zwiers MP, Artiges E, Assareh AA, Ayesa-Arriola R, Belger A, Brandt CL, Brown GG, Cichon S, Curran JE, Davies GE, Degenhardt F, Dennis MF, Dietsche B, Djurovic S, Doherty CP, Espiritu R, Garijo D, Gil Y, Gowland PA, Green RC, Häusler AN, Heindel W, Ho BC, Hoffmann WU, Holsboer F, Homuth G, Hosten N, Jack CR Jr, Jang M, Jansen A, Kimbrel NA, Kolskår K, Koops S, Krug A, Lim KO, Luykx JJ, Mathalon DH, Mather KA, Mattay VS, Matthews S, Mayoral Van Son J, McEwen SC, Melle I, Morris DW, Mueller BA, Nauck M, Nordvik JE, Nöthen MM, O'Leary DS, Opel N, Martinot MP, Pike GB, Preda A, Quinlan EB, Rasser PE, Ratnakar V, Reppermund S, Steen VM, Tooney PA, Torres FR, Veltman DJ, Voyvodic JT, Whelan R, White T, Yamamori H, Adams HH, Bis JC, Debette S, Decarli C, Fornage M, Gudnason V, Hofer E, Ikram MA, Launer L, Longstreth WT, Lopez OL, Mazoyer B, Mosley TH, Roshchupkin GV, Satizabal CL, Schmidt R, Seshadri S, Yang Q, Alzheimer’s Disease Neuroimaging Initiative, CHARGE Consortium, EPIGEN Consortium, IMAGEN Consortium, SYS Consortium, Parkinson’s Progression Markers Initiative, Alvim MK, Ames D, Anderson TJ, Andreassen OA, Arias-Vasquez A, Bastin ME, Baune BT, Beckham JC, Blangero J, Boomsma DI, Brodaty H, Brunner HG, Buckner RL, Buitelaar JK, Bustillo JR, Cahn W, Cairns MJ, Calhoun V, Carr VJ, Caseras X, Caspers S, Cavalleri GL, Cendes F, Corvin A, Crespo-Facorro B, Dalrymple-Alford JC, Dannlowski U, de Geus EJ, Deary IJ, Delanty N, Depondt C, Desrivières S, Donohoe G, Espeseth T, Fernández G, Fisher SE, Flor H, Forstner AJ, Francks C, Franke B, Glahn DC, Gollub RL, Grabe HJ, Gruber O, Håberg AK, Hariri AR, Hartman CA, Hashimoto R, Heinz A, Henskens FA, Hillegers MH, Hoekstra PJ, Holmes AJ, Hong LE, Hopkins WD, Hulshoff Pol HE, Jernigan TL, Jönsson EG, Kahn RS, Kennedy MA, Kircher TT, Kochunov P, Kwok JB, Le Hellard S, Loughland CM, Martin NG, Martinot JL, McDonald C, McMahon KL, Meyer-Lindenberg A, Michie PT, Morey RA, Mowry B, Nyberg L, Oosterlaan J, Ophoff RA, Pantelis C, Paus T, Pausova Z, Penninx BW, Polderman TJ, Posthuma D, Rietschel M, Roffman JL, Rowland LM, Sachdev PS, Sämann PG, Schall U, Schumann G, Scott RJ, Sim K, Sisodiya SM, Smoller JW, Sommer IE, St Pourcain B, Stein DJ, Toga AW, Trollor JN, Van der Wee NJ, van 't Ent D, Völzke H, Walter H, Weber B, Weinberger DR, Wright MJ, Zhou J, Stein JL, Thompson PM, Medland SE, Enhancing NeuroImaging Genetics through Meta-Analysis Consortium (ENIGMA)—Genetics working group. The genetic architecture of the human cerebral cortex. Science. 2020 Mar 20;367(6484) PubMed.

    Brouwer RM, Klein M, Grasby KL, Schnack HG, Jahanshad N, Teeuw J, Thomopoulos SI, Sprooten E, Franz CE, Gogtay N, Kremen WS, Panizzon MS, Loohuis LMO, Whelan CD, Aghajani M, Alloza C, Alnæs D, Artiges E, Ayesa-Arriola R, Barker JG, Bastin ME, Blok E, Bøen E, Breukelaar IA, Bright JK, Buimer EEL, Bülow R, Cannon DM, Ciufolini S, Crossley NA, Damatac CG, Dazzan P, de Mol CL, de Zwarte SMC, Desrivières S, Díaz-Caneja CM, Doan NT, Dohm K, Fröhner JH, Goltermann J, Grigis A, Grotegerd D, Han LKM, Harris MA, Hartman CA, Heany SJ, Heindel W, Heslenfeld DJ, Hohmann S, Ittermann B, Jansen PR, Janssen J, Jia T, Jiang J, Jockwitz C, Karali T, Keeser D, Koevoets MGJC, Lenroot RK, Malchow B, Mandl RCW, Medel V, Meinert S, Morgan CA, Mühleisen TW, Nabulsi L, Opel N, de la Foz VOG, Overs BJ, Martinot MLP, Quinlan EB, Redlich R, Marques TR, Repple J, Roberts G, Roshchupkin GV, Setiaman N, Shumskaya E, Stein F, Sudre G, Takahashi S, Thalamuthu A, Tordesillas-Gutiérrez D, van der Lugt A, van Haren NEM, Wardlaw JM, Wen W, Westeneng HJ, Wittfeld K, Zhu AH, Zugman A, Armstrong NJ, Bonfiglio G, Bralten J, Dalvie S, Davies G, Di Forti M, Ding L, Donohoe G, Forstner AJ, Gonzalez-Peñas J, Guimaraes JPOFT, Homuth G, Hottenga JJ, Knol MJ, Kwok JBJ, Le Hellard S, Mather KA, Milaneschi Y, Morris DW, Nöthen MM, Papiol S, Rietschel M, Santoro ML, Steen VM, Stein JL, Streit F, Tankard RM, Teumer A, van ‘t Ent D, van der Meer D, van Eijk KR, Vassos E, Vázquez-Bourgon J, Witt SH, Alzheimer’s Disease Neuroimaging Initiative, the IMAGEN consortium, Adams HHH, Agartz I, Ames D, Amunts K, Andreassen OA, Arango C, Banaschewski T, Baune BT, Belangero SI, Bokde ALW, Boomsma DI, Bressan RA, Brodaty H, Buitelaar JK, Cahn W, Caspers S, Cichon S, Facorro BC, Cox SR, Dannlowski U, Elvsåshagen T, Espeseth T, Falkai PG, Fisher SE, Flor H, Fullerton JM, Garavan H, Gowland PA, Grabe HJ, Hahn T, Heinz A, Hillegers M, Hoare J, Hoekstra PJ, Ikram MA, Jackowski AP, Jansen A, Jönsson EG, Kahn RS, Kircher T, Korgaonkar MS, Krug A, Lemaitre H, Malt UF, Martinot JL, McDonald C, Mitchell PB, Muetzel RL, Murray RM, Nees F, Nenadic I, Oosterlaan J, Ophoff RA, Pan PM, Penninx BWJH, Poustka L, Sachdev PS, Salum GA, Schofield PR, Schumann G, Shaw P, Sim K, Smolka MN, Stein DJ, Trollor J, van den Berg LH, Veldink JH, Walter H, Westlye LT, Whelan R, White T, Wright MJ, Medland SE, Franke B, Thompson PM, Pol HEH. Age-dependent genetic variants associated with longitudinal changes in brain structure across the lifespan. bioRxiv, November 8, 2021 bioRxiv

    View all comments by Paul Thompson

  3. This study neatly and systematically evaluates the boundaries of effect sizes that can be expected for varying sample sizes in Brain Wide Association Studies (BWAS) with cognitive or mental health measures by making use of three large, publicly available datasets (ABCD, HPC UKB) on mostly normal individuals.

    They find that when sample sizes are small, effect sizes are probably an exaggeration of the real effect, since effect sizes decreased when replicated in larger samples. Importantly, applying a stringent p-value threshold will not solve this problem, and will lead to bias for inflated effects. This indicates that when sample sizes are small, all results should be reported, and researchers should not fixate on p-values only. 

    The authors further compared effect sizes for small subsets of the data with effects observed in the total large data. Striking from those analyses is that the false-negative rate remained very high for very large sample sizes that included more than 2,000 individuals. This puts us between Scylla and Charybdis: Small and robust effect sizes may help understanding brain-phenotype relationships at a group level, but may not apply to single participants; whereas high effect sizes in small samples may exaggerate potential clinical meaningfulness. 

    In general, effect sizes in this study were small, which may reflect that investigating brain-phenotype relationships in normal individuals is a hard thing to do. It can be questioned whether performance on different cognitive tests would map onto the same brain areas, which is here assumed by the use of a composite score. Individuals show complex differences in brain-phenotype relationships, which can change with older age, and it is conceivable that small effect sizes echo this inter-individual heterogeneity. Here the ground truth was assumed be the effect observed in the largest group, but in fact this is unknown.

    Care should be taken to generalize these findings to research on Alzheimer’s disease, where effects might be more pronounced and do not require large sample sizes. Hippocampal atrophy can be seen with the naked eye and has been associated with memory decline in many studies. It would be interesting if this exercise could be repeated using a known, possibly artificial, ground truth to find ideal sample sizes for a given effect size.

    View all comments by Betty Tijms

  4. This is a very interesting article and a long-needed study and a great reference for the future. It demonstrates that the laws of (sample size and effect size in) statistics are universal and make no difference between the type of data. Thus, BWAS suffers from the same problems as GWAS and likely the remedies are similar: more data and replications in out-of-sample cohorts.

    We knew that small sample sizes are more likely, due to sampling bias, to produce more extreme effect sizes, i.e., significant results. These typically fail to replicate. However, what surprised me was that our common strategy to limit false-positives in the first place, that is, to apply a more rigorous statistical threshold, makes things even worse!

    For studies of AD, the takeaway is twofold. Firstly, we can assume that effect sizes of dementia on the brain are stronger than the cognitive phenotype examined here, therefore the sample-size requirements may be in the hundreds instead of thousands. Indeed, studies in the past have reliably mapped hallmark features of AD such as regional atrophy. However, subtle, or secondary, cognitive effects of the disease that are more varied between patients may indeed require thousands of participants to reliably map in order to better understand the disease’s heterogeneity. Secondly, the fundamental takeaway is that our study designs should include replication cohorts, besides aiming to maximize the expected effect sizes by focusing on longitudinal designs and the most promising imaging modalities.

    View all comments by Andre Altmann

  5. This important paper highlights both the benefits and challenges of open science. The benefits are clear—many samples that are widely available provide statistical power and opportunities for replication. However, there are challenges, mostly associated with using many exploratory correlations across multiple metrics both within and between the brain and behavior. The many aspects to measure, and ways to do so, make it difficult for the field to rely on a handful of large studies, each making its own choices about what to measure. As an alternative, researchers can aggregate smaller samples through meta-analyses, keeping the variety of brain and behavior phenotypes.

    As a side note, I argue against imposing committee-driven harmonization when studying brain and behavior because of the complexity of both that, in turn, requires a wide variety of approaches. Different forms of meta-analyses, such as the Activation Likelihood Estimates method and the ENIGMA and CHARGE consortia, have been very helpful in this regard.

    It is noteworthy that, although important, statistical power alone does not replace the need for asking the right question. Researchers must thoughtfully use large datasets to test specific hypotheses grounded in biology (brain) and psychology (behavior), taking advantage of existing knowledge gathered at multiple levels in various species, including humans, over the past 100+ years.

    Conceptually, brain-wide association studies are quite different from genome-wide association studies. The GWAS approach has a solid foundation: It is easily assessed in a highly standardized manner (e.g., SNP arrays), and inter-individual variations in just four nucleotides across the genome are correlated with various phenotypes. On the other hand, one can argue that BWAS is a correlation between a series of open-ended constructs on both sides (e.g., various metrics of “functional connectivity” against various, often latent, metrics of cognition or mental health). Furthermore, unlike the reasonably noise-free genomic variations of genotyping, phenotyping of the brain and behavior is noisy.

    View all comments by Tomas Paus

  6. This study is important because it shows that the concept of BWAS for complex phenotypes such as psychopathology and cognitive screening test composites may be principally flawed. In my view, increasing funding to obtain GWAS-like MRI sample sizes for hypothesis-free research on complex phenotypes in unstratified populations does not seem to be the best investment of resources. An important value of large cohorts is the ability to stratify for specific genetic, clinical, or socio-demographic or other phenotypes for hypothesis-driven brain imaging studies. This is, in my view, the predominant approach in the Alzheimer’s brain imaging field.

    In fact, the authors have been quite balanced in making a distinction between hypothesis-driven brain imaging research in stratified samples and the problems associated with BWAS discovery studies. We should make sure that this message does not get lost while digesting the important learnings from this study.

    View all comments by Emrah Düzel

  7. This is a timely and very important study. It confirmed several of our previous findings obtained in searching for associations between brain structure and behavior in healthy populations, but the current study further shows that the concerns we previously raised similarly apply to associations between brain connectivity and behavior. In particular, effect sizes are inflated in small samples (Kharabian Masouleh et al., 2019), and examining the same question in two different samples can lead to opposite conclusions, e.g., the sign of the associations can be flipped (Genon et al., 2017). Altogether, these works highlight important challenges in neuroimaging research, especially when aiming to relate brain to behavior in healthy populations.

    First, neuroimaging data, as indirect measurements of either brain structure or brain connectivity, are particularly noisy. We can reasonably wonder to which extent the interindividual variability that we measure in brain structure is valid in healthy populations. Similar concerns apply to psychometric data in healthy populations. For instance, we may wonder whether slight differences in the score on a list-learning task between two participants, or differences in personality scores, truly reflect differences in cognitive abilities or stable differences in behavior between these participants, respectively, allowing inferences on their neural correlates.

    Data and research in clinical populations, such as patients with Alzheimer’s disease, are affected to a lesser extent by these concerns. Indeed, we have previously shown that brain structural-behavior associations have better replicability rates in these populations (Kharabian Masouleh et al., 2019). This is likely because the current methods in the field well capture atrophy, as well as behavioral deficits in these populations. This leads to greater effect sizes than those observed for associations in healthy populations, hence yielding more robust findings. Thus, improving the validity of neuroimaging markers of brain structure and function would be an important perspective for progresses in understanding brain-behavior associations.

    Second, sampling variability is a crucial issue in neuroimaging research, like in any research field. The present study well demonstrated that extremely large cohorts are needed to identify robust brain-behavior associations. With cohorts of thousands of participants, a strict cross-validation scheme can be implemented to evaluate the generalizability of the findings. However, even with a within-cohort cross-validation setting, when the data have been acquired at a single site using a unique research protocol, the question of generalizability to a completely independent dataset remains open.

    Researchers have to remain aware of the characteristics of their samples and, thus, of potential bias. For instance, most datasets in our fields are dominated by one type of population, and the model derived for these samples may not generalize to some minorities in the general populations (Li et al., 2022). So, there is still a lot of work to be done to derive brain-behavior models that will be useful and accurate for single individuals.

    References:

    Genon S, Wensing T, Reid A, Hoffstaedter F, Caspers S, Grefkes C, Nickl-Jockschat T, Eickhoff SB. Searching for behavior relating to grey matter volume in a-priori defined right dorsal premotor regions: Lessons learned. Neuroimage. 2017 Aug 15;157:144-156. Epub 2017 May 25 PubMed.

    Kharabian Masouleh S, Eickhoff SB, Hoffstaedter F, Genon S, Alzheimer's Disease Neuroimaging Initiative. Empirical examination of the replicability of associations between brain structure and psychological variables. Elife. 2019 Mar 13;8 PubMed.

    Li J, Bzdok D, Chen J, Tam A, Ooi LQ, Holmes AJ, Ge T, Patil KR, Jabbi M, Eickhoff SB, Yeo BT, Genon S. Cross-ethnicity/race generalization failure of behavioral prediction from resting-state functional connectivity. Sci Adv. 2022 Mar 18;8(11):eabj1812. Epub 2022 Mar 16 PubMed.

    View all comments by Sarah Genon

  8. The work by Marek and colleagues is an outstanding demonstration of a discussion that has been ongoing for the better part of a decade in the fields of psychology and cognitive neuroscience. Often deemed the "replication crisis," there is recognition that many behavioral and neuroimaging studies do not replicate (Open Science Collaboration, 2015; Camerer et al., 2018; Elliott et al., 2020). This lack of replication is often tied to small sample sizes leading to inadequate power (Button et al., 2013), and the ultimate result is poor science. Given how widespread and systematic such issues are across scientific specializations, it is unlikely that studies of neurodegenerative diseases such as Alzheimer's are free of such concerns. In fact, the prevalence of case studies or case series in more clinically oriented fields may exacerbate such issues.

    There are a number of factors, though, that are in our favor. The first is that the overall power in a given study is not simply due to sample size, but also the size of the effect being studied. Many disease effects are orders of magnitude larger than the subtle behavioral associations investigated by Marek and colleagues. For example, detecting atrophy in individuals with an AD diagnosis, or abnormal amyloid values in APOE E4 carriers, are robust, strong effects that can be seen in a handful of individuals. Statistical concerns become more prominent when analyses are focused on individual differences, which are likely to be subtle, as well as increasingly complex statistical models (i.e., three-way interactions).

    Another strength of our field is that the multiple large cohorts being studied in the U.S., Europe, and Asia can inherently lead to a high degree of replication and convergence. While there is often emphasis placed on being the first to show a finding, as a field we should appreciate the utility that replication and reaching a consensus bring. There is also a growing push to make data from these large cohorts more accessible so that studies do have increasingly large Ns. Still, the most important thing we can do is to ask reasonable, well-thought-out scientific questions. Larger and larger samples are not helpful if the base science being pursued is inherently flawed or ill planned.

    References:

    Open Science Collaboration. PSYCHOLOGY. Estimating the reproducibility of psychological science. Science. 2015 Aug 28;349(6251):aac4716. PubMed.

    Camerer CF, Dreber A, Holzmeister F, Ho TH, Huber J, Johannesson M, Kirchler M, Nave G, Nosek BA, Pfeiffer T, Altmejd A, Buttrick N, Chan T, Chen Y, Forsell E, Gampa A, Heikensten E, Hummer L, Imai T, Isaksson S, Manfredi D, Rose J, Wagenmakers EJ, Wu H. Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nat Hum Behav. 2018 Sep;2(9):637-644. Epub 2018 Aug 27 PubMed.

    Elliott ML, Knodt AR, Ireland D, Morris ML, Poulton R, Ramrakha S, Sison ML, Moffitt TE, Caspi A, Hariri AR. What Is the Test-Retest Reliability of Common Task-Functional MRI Measures? New Empirical Evidence and a Meta-Analysis. Psychol Sci. 2020 Jul;31(7):792-806. Epub 2020 Jun 3 PubMed.

    Button KS, Ioannidis JP, Mokrysz C, Nosek BA, Flint J, Robinson ES, Munafò MR. Power failure: why small sample size undermines the reliability of neuroscience. Nat Rev Neurosci. 2013 May;14(5):365-76. Epub 2013 Apr 10 PubMed.

    View all comments by Brian Gordon

Make a Comment

To make a comment you must login or register.

This paper appears in the following:

News

  1. Like GWAS, BWAS Need Thousands of Participants