In this paper we address the problem of testing whether two observed trees $(\mathit{t},{\mathit{t}}^{\prime })$ are sampled either independently or from a joint distribution under which they are correlated. This problem, which we refer to as correlation detection in trees, plays a key role in the study of graph alignment for two correlated random graphs. Motivated by graph alignment, we investigate the conditions of existence of one-sided tests, that is, tests which have vanishing type I error and nonvanishing power in the limit of large tree depth.

For the correlated Galton–Watson model with Poisson offspring of mean $\mathit{\lambda }>0$ and correlation parameter $\mathit{s}\in (0,1)$, we identify a phase transition in the limit of large degrees at $\mathit{s}=\sqrt{\mathit{\alpha }}$, where $\mathit{\alpha }\sim 0.3383$ is Otter’s constant. Namely, we prove that no such test exists for $\mathit{s}\le \sqrt{\mathit{\alpha }}$, and that such a test exists whenever $\mathit{s}>\sqrt{\mathit{\alpha }}$, for λ large enough.

This result sheds new light on the graph alignment problem in the sparse regime (with $\mathit{O}(1)$ average node degrees) and on the performance of the $\mathtt{MPAlign}$ method studied in (Ganassali, Massoulié and Lelarge (2022), J. Stat. Mech. Theory Exp. 2022 (2022)), proving in particular the conjecture of (J. Stat. Mech. Theory Exp. 2022 (2022)) that $\mathtt{MPAlign}$ succeeds in the partial recovery task for correlation parameter $\mathit{s}>\sqrt{\mathit{\alpha }}$ provided the average node degree λ is large enough.

As a byproduct, we identify a new family of orthogonal polynomials for the Poisson–Galton–Watson measure which enjoy remarkable properties. These polynomials may be of independent interest for a variety of problems involving graphs, trees or branching processes, beyond the scope of graph alignment.