This revision is a very different manuscript from the first submission, somewhat unexpectedly as the previous round of reviews were quite positive. The analysis has been extended in a number of ways, mostly by allowing polymorphism in parental populations, such that what used to be expressed in terms of phenotypic effects and frequencies (in hybrids) of alleles fixed in each parental population now appears as components of variance typical of quantitative genetics partitioning (additive, dominance, etc), which accounts for polymorphism. The explored evolutionary scenarios are also partly different, not only regarding directional selection, but also by including a detailed exploration of drift around a constant optimum (described as stabilizing selection). As a consequence, (almost) all the figures have changed, and so has the text. I praise the authors for this effort, which has undoubtedly made the manuscript richer, and even more interesting than it was. On the other hand, this density of results also certainly makes it a difficult read, more so than the previous version. I can’t say I’ve digested all results myself as much as I would have liked, when it comes to contrasting net evolution to total evolution for interactions between additive and dominance terms, under different scenarios of dominance at the phenotypic level (Figure 4). Still, I can’t find any obvious way around this: the authors have pursued a thorough analysis of the model while keeping it rather general (not assuming fixation in parental populations, additivity at phenotypic level, etc), so they just have lots of things to say, some of which are not straightforward. They have done some efforts to synthetize their results in Table 4, and suggest ways to empirically assess some of their predictions. Some presentation choices don’t facilitate understanding in my opinion (such as the absence of an explicit ancestral phenotype, instead using directional flow of mutations from P1 to P2), but this has been discussed previously and the authors have their arguments for doing things this way. So I will just list a few minor points below. 

Minor points: 

12: ‘making fewer assumptions’ than what? Please specify

44: perhaps insert ‘before hybrids form’ after ‘that have accumulated”

144: Perhaps write “sum of phenotypic variances among traits” instead of “sum of the trait variances”, as the latter could mean different things, including sum across loci for a given trait.

Figure 2: The cartoons on the left are far from being self-evident. For instance, what’s the difference between light and dark colors for the thick arrow? I can only guess that this relates to which population evolves, but the precise meaning isn’t obvious. And what about the small thin arrows attached to the thick ones? Why are they not described in the caption?

314: I think you mean VI and not IV here. 

Eq 42, second line for P_BB,i: one of the terms in the difference in parentheses should harbour P_1 instead of P_2, otherwise this term vanishes

652: typo ‘phenoptypic’

656: ‘positive epistasis in an optimal background’. Note that epistasis does not depend on genetic  background, but only on mutation phenotypic effects, in this classic quadratic/Gaussian version of FGM

The authors have greatly improved their clarity. Thank the authors for clarifying the epistasis in Appendix 1. In this version, my remaining comments are limited to clarifications. 

I appreciate the authors’ effort in including the segregating sites mathematically. However, some significant results come from simulations, and the author used divergence between populations to demonstrate. This makes me wonder about the reason for using shared polymorphism. How should we understand the contribution of shared polymorphism and fixed differences to hybrids or to postzygotic RI? Are they the same in two species? And how much of the shared polymorphism do we expect from two species? Since the authors and reviewer Chevin have discussed this, I hope the authors could explain a bit, like explicitly writing a paragraph in the introduction or method.

Abstract – In general, “The key quantities” include two properties of the evolution changes and their interactions, right?  This term is not obvious to me. 

Table 6 and page 21 -- I am confused with the inbreeding coefficient in the F1 population. If the two parental populations perform random mating to form the F1 hybrid population, the inbreeding coefficient is 0 in the F1 population (HWE). If the F1 population contains only hybrids of two populations, the F1 population is a result of complete assortative mating (“an excess of heterozygotes”). 

Table 6, eq. 6, eq. 32, eq. 33. -- I am confused with d_ij and delta_ij. Are they describing the same thing, the dominance in Table 5? Or is delta_ij about dominance at the population level? Is the dominance defined at a single locus (Table 5) equal to the mean dominance (eq. 33)?

Eq. 49 – Please notice that the equation is too large to show on one line.

Figure 2 – The black circle is the ancestry population, shown in the panel illustrating scenario I. I guess that the center of each panel is not the optimum (the cross in figure 1), but the two dots are two optima. I guess that the large arrow indicates the direction of additive effects, and the small arrow indicates the dominance. It might be helpful to clarify this in the figure legend.

Figure 4 – I suggest the authors keep the panels to describe two scenarios of stabilizing selection (in old Figure 2, like the left panels in new Figure 2).  It makes it easier to understand a theoretical paper.

Line 321 – “Unlike terms involving additive or dominance effects alone, the interaction terms capture differences in the evolutionary changes between two populations.” The word, “differences”, is not clear to me. Throughout the whole section, the author explains how dominance affects the fixation of mutations, Haldane’s Sieve.  Maybe, rephrase this sentence?

In this manuscript, De Sanctis et al investigate how the mode of divergence between parental populations affects the fitness of their hybrids across environments. The mode of divergence encompasses the phenotypic direction of evolution from the most recent common ancestor (MRCA), as well as the phenotypic effects of mutations that fixed, both in terms of magnitude (additive effects) and within-locus interactions (dominance effects). In line with previous work on this question, starting from Barton (2001) and followed up by a series of papers in the last decade (including several by their group), they model adaptation using Fisher’s geometrical model (FGM), whereby fitness depends on squared phenotypic distances from the optimum at multiple traits. Using this common baseline, they rely on combinatorics to generalize and extend previous findings about the way evolutionary trajectories of parental populations influence the fitness of their hybrids. 

One of their main insights is that all outcomes regarding hybrids fitness, including the relative contributions of intrinsic vs extrinsic (environment-dependent) isolation, essentially depend on two evolutionary metrics: the total amount of evolutionary change, which sums the squared magnitudes of all mutation effects, and the net effect of evolutionary change, which is the squared magnitude of the overall change (which itself sums all mutation effects). These two metrics, which can be applied to additive and dominance effects, summarize complementary aspects of the evolutionary trajectories of parental populations. When combined with summaries of the genetic composition of hybrids (hybrid index, proportion of heterozygous sites, etc), they are sufficient to predict their expected (log) fitness. The authors provide complementary interpretations of their results in terms of geometry in phenotypic space, and of selection coefficients and epistasis.

The work presented in this ms is an interesting and useful extension of previous theory on this topic, providing insightful general explanations that clarify and unify earlier findings, as well as exploring a larger diversity of evolutionary scenario. The analysis seems solid, therefore I have no major criticisms, but rather suggestions for improvement of presentation.

First, some arguments made here appeared in simpler forms in earlier work, which could perhaps be credited a bit more for completeness. For instance, the difference between the squared total phenotypic change (“net effect of evolutionary change here”) vs sum of squared changes (“total amount of evolutionary change”), and how this relates to fitness epistasis and the angle between mutations, was a key step in the derivations and overall argument in Chevin et al (2014), although perhaps a bit cryptically (between eq. 3 and appendix S1 of that paper, in a non-isotropic model where distances are less directly meaningful and are thus translated into fitness effects), and restricted to a scenario of directional selection akin to that explored in the last section and Figure 3 of the current ms. Some results about mutation-selection-drift equilibrium around a constant optimum, and the partitioning between intrinsic and extrinsic isolation, also seemed quite similar to points made here, after accounting for the fact that the “total amount of evolutionary change” is very much related to the segregation variance (including regarding the contrasted effects of a few large vs many small mutations, in fig 3 of the present ms). This is not to say that the results here are not novel or insightful, but rather that it may be worthwhile relating them a bit more explicitly to work that touched on very similar issues in a less general way, so as to clarify the improvements made here. 

Regarding the model per se, perhaps I haven’t read it carefully enough, but after several attempts I have been unable to find a proper description of what y represents in the g(x,y) functions (and S_xy in the methods). Since this function is a crucial component of hybrid fitness, it is important to be very explicit about it, to avoid any confusion or wrong interpretations. From my reading and previous understanding of these models, I concluded that x and y correspond to mutations that originate from the two different parental backgrounds, but I can’t be sure about that. 

Another point of definition that was not very clear to me was about the angle theta between substitution vectors. The first time this topic of orientation is brought up in the ms, I found it quite confusing to read (below eq. 16) that “we have cos(theta) = -1 when two substitutions point in the same phenotypic direction (such that theta = pi), […] and cos(theta) = 1 for substitutions that point in opposite directions”, because this seems to contradict basic trigonometry, if theta is the angle between two vectors. Only when looking more carefully at fig 1B did I realize that theta was in fact not the angle between the two vectors under consideration, but rather pi minus this angle (ie, the supplementary angle, as stated much later in the ms). It is unclear to me why theta was defined in this way, and I think this can be confusing to many readers used to applying geometric arguments to phenotypic spaces. If the goal was just to eliminate a minus sign before the scalar products, at the moment this seems to be done at the expense of the intuitive meaning, which is regrettable since intuitive interpretation seems to be one of the key goals of this paper. I would find it simpler to keep the minus sign in the formulas and rely on the actual angle between vectors, such that these formulas would state that something about the relative orientation of mutations is subtracted from the rest. If not using this option, then the authors should at least state clearly that theta is not the angle between vectors as usually defined, but the supplement to that angle, and provide an intuitive explanation for why that is (as it does not seem sufficient to just write that “the negative sign comes from the need to take the supplementary angle due to the directionality of the vectors”).

Minor comments

22 : “Genomic and phenotypic differentiation between populations is a major cause” -> “are”?

44-45: “strong assumptions about the distribution of the fixed effects (e.g., normality, universal pleiotropy, and independence among traits)”. Normality is often assumed for the phenotypic distribution of mutation effects, but not for fixed effects; and at least one model of hybrid fitness in FGM (the one I know best, cited above) did allow for correlations among traits.

120-121: “describes interactions between heterospecic alleles in different states”. Perhaps state here (if relevant) that you are referring to additive-by-additive or additive-by-dominance epistasis for fitness?

Eqs below 402 (methods): A key implicit step in this derivation (which is central to the results in the ms) is the requirement that the probability for a locus to be in each genetic state (heterozygote or each homozygote) does not covary with allelic phenotypic effect. I would find it useful to state this explicitly for completeness, and to explain what this implies: probably something quite general like fair segregation, or rather phenotype-independent segregation?

Eq below 409: For those curious about that (including me!), could you explain where this product of binomial coefficients comes from?

Figure S1: I can't help noticing a substantial effect of phenotypic dimensionality n, mostly on the net effect m(2a), and to a lesser extent on the total amount M(2a) of evolutionary change (comparing triangles with circles), and thus on hybrid fitness. I tend to think that this is worth mentioning briefly in the ms, but this may be a personal bias!

The manuscript “How does the mode of evolutionary divergence affect reproductive isolation?” is      an explicit, well-written manuscript on the evolution of reproductive isolation (RI) under Fisher’s geometric model. This paper aimed to investigate the connections between genotype and phenotype to explain the fitness of hybrids, from which we could gain an insight into the strength of reproductive isolation. Here, they presented the total amount (intrinsic, related to genotype divergence) and net effect (extrinsic, related to phenotype distribution) of evolutionary change caused by additive and dominance effects and their interactions. Using simulations, they showed the distribution of additive and dominance effects under different modes of evolutionary divergence, which result in predictable consequences for the hybrid fitness. They also compared the effect of intrinsic and extrinsic isolation on speciation. Within this scope, they articulate the evolutionary role of large- and weak- effect substitutions on adaptative trajectory and speciation. Based on these conclusions, they generalized their model to the effect of gene flow between species. I enjoyed reading this paper, and I found that the authors balanced the considerable number of equations and their evolutionary interpretation very well. Overall, this is interesting research and presented clearly.

My comments are extremely limited. I offer a few questions to be clarified. Hopefully, the authors could make their theoretical work not limited to being read by theoreticians but more understandable for a broader readership. 

Paragraph “Dominance effects” (line 188) – I do not understand the complementary information from additive and dominance effects. In both this paragraph and the discussion (Line 367-Lin 376), the overall dominance reflects the position of the MRCA on the graph, however, it is not related to the additive effect, like, Scenario I and II. Should I expect some explanations from g(a,d) eq. 22? It would be great if the authors could make this connection more clear in a revised manuscript.

Figure 3 – 1. I am aware that dominance is assumed to be 0 in the main text. It would be good to spell this out as well in the legend. 2. I can roughly understand the y-axis label, (E(ln wH))/m(2a) in panels A and B, to compare with M(a)/m(a). However, I do not understand why to intuitively scale log fitness by m(2a), i.e., the biological explanation. 

Equation 29—It is not obvious for me to get eq. 29 from eq. 19. It would be great if they authors could explain this in a bit more detail in a revised manuscript or provide citations if it has already been analyzed in other papers.

Line 243—Should Figure 1C be Figure 1B? Figure 1C demonstrates F1 and mid-parent, whereas Line 243 pointed to two parental phenotypes. 

Section 1.3.3 -- I might misunderstand the purpose of this section. I want to communicate more on eq. 26. I understand that epistasis was defined in eq. 25 from eq. 1 and the fitness effect was an approximation. In this section,      dominance was not considered because only homozygous substitutions are present in the background. Meanwhile, the whole manuscript focused on the fitness of hybrids (RI). I am confused with the dominance effect when using the epistatic components in m(2a). Here, the epistatic effect could come from any non-linear genotype-phenotype-fitness map. The epistasis could also be generated on the evolutionary trajectory, meaning a similar epistatic factor is in m(2d). It would be helpful if the authors could explain the meaning and relationship of epistasis in hybrids and when two homozygous substitutions are present in one genome.

Line 548 – This citation seems incomplete. It might be worth checking the reference list.  

Overall, I found this an interesting paper that will undoubtedly stimulate people to think about the strength of reproductive isolation and the evolutionary trajectory to RI during adaptation and speciation.