Evolutionary trajectories of social transitions: Higher social complexity is associated with lower effective population size and reduced efficacy of selection in termites

Roux, Weyna, Lode and Romiguier present dN/dS estimations for termites, non-eusocial blattodeans and outgroups that support the hypothesis that eusociality reduces selection efficiency due to a reduction in effective population size. This effect is more pronounced when considering different levels of social complexity within termites. These results build on previous findings presented for Hymenoptera, thus reinforcing support for the hypothesis.

I found the manuscript enjoyable to read and very clear. The presented results are exciting and important. However, alas, a similar study has recently been published, most likely while this manuscript was in its final stages: Ewart et al. "Pervasive relaxed selection in termite genomes." Proceedings of the Royal Society B 291.2023 (2024).
The results from the current study support the findings from Ewart et al. while adding additional important findings on the effect of social complexity. I find it is, therefore, important that that paper should be cited and results compared, e.g. that transcriptomic results on larger species set confirm genomic analyses.....

Generally, I find this study important due to its similarity in findings for Hymenoptera, offering support for important convergent genomic mechanisms associated with the evolution of eusociality (as discussed in Mikhailova et al COIS 2023: 101136). Beside the aforementioned need to incorporate Ewart et al, I have only minor comments. These are mainly related to what the data prove and what they suggest, and how this is worded in the text:

1. Overall, although I agree that high dNdS is likely caused by low Ne due to social complexity, I would suggest toning down in some places that this is proven here. High dNdS can also be caused by adaptive evolution. There were no formal tests carried out for a relaxation of selection, just dN/dS. But the results do offer strong support.

2. What is the cause and effect of eusociality and low Ne - again I agree with the main assertions in this manuscript - but there are two examples where it could be argued that a low Ne preceded the evolution of eusociality: in solitary bees and in Cryptocercus. Using ants as an example where this was not the case is difficult since it is difficult to find a suitable non-eusocial outgroup for ants.
This is discussed to some extent but then on lines 298ff the authors suggest eusociality is associated with decreases in Ne, rather than low Ne. The alternative scenarios and how they can be addressed in the future could be more openly addressed.

3. Also, please be careful when suggesting that low Ne is confirmed here, e.g. line 298. Theory predicts low Ne - and it certainly makes a lot of sense but the analyses here do not allow any direct inferences on Ne.

These are not major points and just need to be reworded to make it clear what is an interpretation and what is more directly measured by the data. For these reasons, I believe the title needs to be adapted accordingly, e.g. evolutionary rates support .... or something similar.

More detailed comments follow as they appear in the manuscript.

Intro:

- line 37: should be "effective population size"

- lines 37-40: citations and examples missing

- line 56: dS is "solely" driven by mutation rate - should be toned down to "mainly" or "predominantly" as other factors such as tRNA abundance, substituion bias play a role.

M&M:

- Blattodea dataset: were these alignments always single copy orthologues? And not always across all species?

- Social traits: Please provide a table showing how species were classified by social traits for both data sets

- line 133: Neo- and Basal Isoptera - please explain the distinction and why it is relevant

- line 134: worker type - please elaborate

Results & Discussion:

- line 157: "we observe a notable lower efficiency of selection" - this is one example of point 1 above - you observe higher dnds ratios and this can be interpreted as.... There are more examples (e.g. line 167) but I won't list them all

- line 165 and elsewhere: from here onwards the authors write "Blattodae" instead of "Blattodea"

- lines 185ff - refer to figure 2B in the text

- figure 1 caption: utilising "identical" coding genes - what is meant here?

- figure 2: what do the two colours mean? Please use a legend. Should there not be two regression lines in plot 2A - one for termites, one for others?

- Table 1: caption is a bit short and lacking details

- line 201ff: this is a good point about pyhlogenetic corrections not being necessary for dnds. But what if the cause of the differences, i.e. Ne, is phylogenetic, for example if a reduction in Ne occurred in the common ancestor of all termites and Cryptocercus? This could be tested by comparing Termites+Cryptocercus to other groups as was done in table 1

- line 210f: I'm not sure if the dnds of Cryptocercus wrighti can be considered "intermediate" - it is quite close to the mean for termites. Please provide a supplementary table of mean dnds values for all investigated species for each data set to make this clearer.

- line 219: this should be ALLOparental care

- Figure 4a: where is the dot for Cryptocercus in the left hand boxplot? It should be at around 0.125

- Social complexity: please also discuss the possible effect of indirect selection due to sterile workers in more socially complex species.

- line 287: the individual effect of worker type and nesting type is likely impossible to pull apart as they correlate. Therefore, the causes of indirect selection and Ne are also difficult to isolate. Please discuss.

- line 298: effective population size is not determined in this study.