Section: Mathematical & Computational Biology
Topic: Evolution, Population biology, Statistics

Trait evolution with incomplete lineage sorting and gene flow: the Gaussian Coalescent model

Ané, Cécile1 orcid  ; Bastide, Paul2 orcid  
1Department of Statistics, University of Wisconsin-Madison, Madison, WI 53706, United States
2Université Paris Cité, CNRS, MAP5, F-75006 Paris, France

Corresponding author(s): Ané, Cécile (cecile.ane@wisc.edu); Bastide, Paul (paul.bastide@u-pariscite.fr)

10.24072/pcjournal.789 - Peer Community Journal, Volume 6 (2026), article no. e96

Get full text PDF Peer reviewed and recommended by PCI
article image

Most phylogenetic comparative methods use a species-level phylogeny, ignoring the effect of incomplete lineage sorting (ILS) and hemiplasy on the traits of interest. We consider here a trait controlled additively by one or more unknown loci. Their gene trees may differ from the species phylogeny due to ILS. The species phylogeny can be any phylogenetic tree or network or admixture graph on any number of taxa or populations. Our process of trait evolution accounts for introgression represented in the phylogeny by hybrid (or admixture) edges, which capture gene flow, admixture or hybridization events. The process models the trait in each individual, for any number of individuals from each taxon. Our model allows for polymorphism in the ancestral population at the root of the species phylogeny, and we can derive the heritable within-population variation expected from our model. Even if each locus evolves according to a Brownian motion, the joint distribution of the trait across all measured individuals is not generally Gaussian due to ILS. We provide a Gaussian approximation, named the Gaussian Coalescent (GC), and show how to compute its variance matrix efficiently using a single traversal of the species phylogeny. In simulations, this model is much more accurate than the model ignoring ILS. In simulations and on a data set of tomato floral traits, it is favored over the standard Brownian motion model with extra within-population variance. The GC model opens new avenues for various phylogenetic methods, accounting for hemiplasy and gene flow simultaneously on any general network or admixture graph. It is implemented in phylolm v2.7.0 and in PhyloTraits v1.2.0.

Published online:
DOI: 10.24072/pcjournal.789
Type: Research article
Classification:
Keywords: phylogenetic comparative methods, phylogenetic networks, admixture graphs, polygenic trait, hemiplasy, JuliaPhylo

Ané, Cécile  1 ; Bastide, Paul  2

1 Department of Statistics, University of Wisconsin-Madison, Madison, WI 53706, United States
2 Université Paris Cité, CNRS, MAP5, F-75006 Paris, France
License: CC-BY 4.0
Copyrights: The authors retain unrestricted copyrights and publishing rights
Ané, C.; Bastide, P. Trait evolution with incomplete lineage sorting and gene flow: the Gaussian Coalescent model. Peer Community Journal, Volume 6 (2026), article  no. e96. https://doi.org/10.24072/pcjournal.789
@article{10_24072_pcjournal_789,
     author = {An\'e, C\'ecile and Bastide, Paul},
     title = {Trait evolution with incomplete lineage sorting and gene flow:~the {Gaussian} {Coalescent} model
},
     journal = {Peer Community Journal},
     eid = {e96},
     year = {2026},
     publisher = {Peer Community In},
     volume = {6},
     doi = {10.24072/pcjournal.789},
     language = {en},
     url = {https://peercommunityjournal.org/articles/10.24072/pcjournal.789/}
}
TY  - JOUR
AU  - Ané, Cécile
AU  - Bastide, Paul
TI  - Trait evolution with incomplete lineage sorting and gene flow: the Gaussian Coalescent model

JO  - Peer Community Journal
PY  - 2026
VL  - 6
PB  - Peer Community In
UR  - https://peercommunityjournal.org/articles/10.24072/pcjournal.789/
DO  - 10.24072/pcjournal.789
LA  - en
ID  - 10_24072_pcjournal_789
ER  - 
%0 Journal Article
%A Ané, Cécile
%A Bastide, Paul
%T Trait evolution with incomplete lineage sorting and gene flow: the Gaussian Coalescent model

%J Peer Community Journal
%] e96
%D 2026
%V 6
%I Peer Community In
%U https://peercommunityjournal.org/articles/10.24072/pcjournal.789/
%R 10.24072/pcjournal.789
%G en
%F 10_24072_pcjournal_789

PCI peer reviews and recommendation, and links to data, scripts, code and supplementary information: 10.24072/pci.mcb.100819

Conflict of interest of the recommender and peer reviewers:
The recommender in charge of the evaluation of the article and the reviewers declared that they have no conflict of interest (as defined in the code of conduct of PCI) with the authors or with the content of the article.

[1] Ané, C.; Fogg, J.; Justison, J.; and contributors PhyloCoalSimulations.jl, https://github.com/JuliaPhylo/PhyloCoalSimulations.jl, 2025

[2] Avise, J. C.; Robinson, T. J. Hemiplasy: A New Term in the Lexicon of Phylogenetics, Volume 57 (2008) no. 3, pp. 503-507 | DOI

[3] Bastide, P.; Ané, C. Code for "Trait evolution with incomplete lineage sorting and gene flow: the Gaussian Coalescent model", https://doi.org/10.5281/zenodo.22083886, 2026 | DOI

[4] Bastide, P.; Solís-Lemus, C.; Kriebel, R.; William Sparks, K.; Ané, C. Phylogenetic Comparative Methods on Phylogenetic Networks with Reticulations, Volume 67 (2018) no. 5, pp. 800-820 | DOI

[5] Bulmer, M. G. The Effect of Selection on Genetic Variability, Volume 105 (1971) no. 943, pp. 201-211 | DOI

[6] Clavel, J.; Escarguel, G.; Merceron, G. mvMORPH : an R package for fitting multivariate evolutionary models to morphometric data, Volume 6 (2015) no. 11, pp. 1311-1319 | DOI

[7] Clavel, J.; Morlon, H. Reliable Phylogenetic Regressions for Multivariate Comparative Data: Illustration with the MANOVA and Application to the Effect of Diet on Mandible Morphology in Phyllostomid Bats, Volume 69 (2020) no. 5, pp. 927-943 | DOI

[8] Degnan, J. H. Modeling Hybridization Under the Network Multispecies Coalescent, Volume 67 (2018) no. 5, pp. 786-799 | DOI

[9] Degnan, J. H.; Rosenberg, N. A. Gene tree discordance, phylogenetic inference and the multispecies coalescent, Volume 24 (2009) no. 6, pp. 332-340 | DOI

[10] Felsenstein, J. Maximum-likelihood estimation of evolutionary trees from continuous characters, Volume 25 (1973) no. 5, pp. 471-492

[11] Felsenstein, J. Phylogenies and the comparative method, Volume 125 (1985) no. 1, pp. 1-15 | DOI

[12] Felsenstein, J. Inferring Phylogenies, Sinauer Associates, 2004

[13] Fogg, J.; Allman, E. S.; Ané, C. PhyloCoalSimulations: A simulator for network multispecies coalescent models, including a new extension for the inheritance of gene flow, Volume 72 (2023) no. 5, pp. 1171-1179 | DOI

[14] Gautier, M.; Vitalis, R.; Flori, L.; Estoup, A. $f$-Statistics estimation and admixture graph construction with Pool-Seq or allele count data using the R package poolfstat, https://onlinelibrary.wiley.com/doi/abs/10.1111/1755-0998.13557, Volume 22 (2022) no. 4, pp. 1394-1416 | DOI

[15] Guerrero, R. F.; Hahn, M. W. Quantifying the risk of hemiplasy in phylogenetic inference, Volume 115 (2018) no. 50, pp. 12787-12792 | DOI

[16] Haak, D. C.; Ballenger, B. A.; Moyle, L. C. No evidence for phylogenetic constraint on natural defense evolution among wild tomatoes, Volume 95 (2014) no. 6, pp. 1633-1641 | DOI

[17] Hamlin, J. A. P.; Hibbins, M. S.; Moyle, L. C. Assessing biological factors affecting postspeciation introgression, Volume 4 (2020) no. 2, pp. 137-154 | DOI

[18] Harmon, L. J. Phylogenetic Comparative Methods: Learning From Trees, Center for Open Science. [EcoEvoRxiv preprint], 2019 | DOI

[19] Hibbins, M. S.; Gibson, M. J.; Hahn, M. W. Determining the probability of hemiplasy in the presence of incomplete lineage sorting and introgression, Volume 9 (2020), e63753 | DOI

[20] Hibbins, M. S.; Breithaupt, L. C.; Hahn, M. W. Phylogenomic comparative methods: Accurate evolutionary inferences in the presence of gene tree discordance, Volume 120 (2023) no. 22, e2220389120 | DOI

[21] Hibbins, M. S.; Hahn, M. W. The effects of introgression across thousands of quantitative traits revealed by gene expression in wild tomatoes, Volume 17 (2021) no. 11, e1009892 | DOI

[22] Ho, L. S. T.; Ané, C. A linear-time algorithm for Gaussian and non-Gaussian trait evolution models, Volume 63 (2014) no. 3, pp. 397-408 | DOI

[23] Karimi, N.; Grover, C. E.; Gallagher, J. P.; Wendel, J. F.; Ané, C.; Baum, D. A. Reticulate Evolution Helps Explain Apparent Homoplasy in Floral Biology and Pollination in Baobabs (Adansonia; Bombacoideae; Malvaceae), Volume 69 (2020) no. 3, pp. 462-478 | DOI

[24] Kingman, J. F. C. On the genealogy of large populations, Volume 19 (1982), pp. 27-43 | DOI

[25] Landis, M. J.; Schraiber, J. G.; Liang, M. Phylogenetic analysis using Lévy processes: finding jumps in the evolution of continuous traits, Volume 62 (2013) no. 2, pp. 193-204 | DOI

[26] Lynch, M.; Hill, W. G. Phenotypic evolution by neutral mutation, Volume 40 (1986) no. 5, pp. 915-935 | DOI

[27] Maddison, W. P. Gene Trees in Species Trees, Volume 46 (1997) no. 3, pp. 523-536 | DOI

[28] Maier, R.; Flegontov, P.; Flegontova, O.; Isildak, U.; Changmai, P.; Reich, D. On the limits of fitting complex models of population history to f-statistics, Volume 12 (2023), e85492 | DOI

[29] Mary-Huard, T.; Balding, D. Fast and accurate joint inference of coancestry parameters for populations and/or individuals, Volume 19 (2023) no. 1, e1010054 | DOI

[30] Mendes, F. K.; Hahn, M. W. Data from: Why concatenation fails near the anomaly zone (2017) | DOI

[31] Mendes, F. K.; Fuentes-González, J. A.; Schraiber, J. G.; Hahn, M. W. A multispecies coalescent model for quantitative traits, Volume 7 (2018), e36482 | DOI

[32] Mendes, F. K.; Hahn, M. W. Why Concatenation Fails Near the Anomaly Zone, Volume 67 (2018) no. 1, pp. 158-169 | DOI

[33] Mo, Y. K.; Hahn, M. W. Estimating the rate of quantitative trait evolution in the presence of gene tree discordance by calculating likelihoods across trees, Volume 5 (2026) no. 1, kzag004 | DOI

[34] Morley, L.; Lehmberg, E.; Kong, S. Evaluating Phylogenetic Comparative Methods under Reticulate Evolutionary Scenarios (2026), syag064 | DOI

[35] Nielsen, S. V.; Vaughn, A. H.; Leppälä, K.; Landis, M. J.; Mailund, T.; Nielsen, R. Bayesian inference of admixture graphs on Native American and Arctic populations, Volume 19 (2023) no. 2, pp. 1-22 | DOI

[36] Patterson, N.; Moorjani, P.; Luo, Y.; Mallick, S.; Rohland, N.; Zhan, Y.; Genschoreck, T.; Webster, T.; Reich, D. Ancient Admixture in Human History, Volume 192 (2012) no. 3, pp. 1065-1093 | DOI

[37] Pease, J. B.; Haak, D. C.; Hahn, M. W.; Moyle, L. C. Phylogenomics Reveals Three Sources of Adaptive Variation during a Rapid Radiation, Volume 14 (2016) no. 2, e1002379 | DOI

[38] Pickrell, J. K.; Pritchard, J. K. Inference of Population Splits and Mixtures from Genome-Wide Allele Frequency Data, Volume 8 (2012) no. 11, e1002967 | DOI

[39] Rannala, B.; Yang, Z. Bayes Estimation of Species Divergence Times and Ancestral Population Sizes Using DNA Sequences From Multiple Loci, Volume 164 (2003) no. 4, pp. 1645-1656 | DOI

[40] Revell, L. J.; Harmon, L. J. Phylogenetic Comparative Methods in R, Princeton University Press, 2022

[41] Rogers, A. Evolution of a Quantitative Character in a Phylogenetic Network (2026), 100819 | DOI

[42] Rose, J. P. Taxonomy and relationships within Polemonium foliosissimum (Polemoniaceae): untangling a clade of colorful and gynodioecious herbs, Volume 46 (2021) no. 3, pp. 519-537 | DOI

[43] Schraiber, J. G.; Edge, M. D.; Pennell, M. Unifying approaches from statistical genetics and phylogenetics for mapping phenotypes in structured populations, Volume 22 (2024) no. 10, e3002847 | DOI

[44] Schraiber, J. G.; Landis, M. J. Sensitivity of quantitative traits to mutational effects and number of loci, Volume 102 (2015), pp. 85-93 | DOI

[45] Schrempf, D.; Szöllösi, G. The Sources of Phylogenetic Conflicts, Phylogenetics in the Genomic Era, No commercial publisher | Authors open access book, 2020, p. 3

[46] Steel, M. Phylogeny: Discrete and Random Processes in Evolution, Society for Industrial and Applied Mathematics, 2016, p. 302 | DOI

[47] Tabatabaee, Y.; Claramunt, S.; Mirarab, S. Coalescent-Based Branch Length Estimation Improves Dating of Species Trees (2026), syag038 | DOI

[48] Teo, B.; Rose, J. P.; Bastide, P.; Ané, C. Supplementary material for: Accounting for within-species variation in continuous trait evolution on a phylogenetic network, 2022 | DOI

[49] Teo, B.; Rose, J. P.; Bastide, P.; Ané, C. Accounting for within-species variation in continuous trait evolution on a phylogenetic network, Volume 2 (2023) no. 3, pp. 1-29 | DOI

[50] Wagenmakers, E.-J.; Farrell, S. AIC model selection using Akaike weights, Volume 11 (2004) no. 1, pp. 192-196 | DOI

[51] Wang, Y.; Cao, Z.; Ogilvie, H. A.; Nakhleh, L. Phylogenomic assessment of the role of hybridization and introgression in trait evolution, Volume 17 (2021) no. 8, e1009701 | DOI

[52] Xu, J.; Ané, C. A consistent least-squares criterion for calibrating edge lengths in phylogenetic networks (2024) | DOI

Cited by Sources: