loos/Mountain2
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

done with bench and TBE

Browse Source
This commit is contained in:
Pierre-Francois Loos 2019-11-11 15:12:27 +01:00
parent 987f517ce0
commit 6662f5e3be
1 changed files with 68 additions and 59 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

127
Manuscript/FCI2.tex
View File

@ -1108,39 +1108,41 @@ $^h${CC3-ext.~from Ref.~\citenum{Sil10c}.}
\section{Theoretical Best Estimates}
In Table \ref{Table-tbe}, we present our TBE obtained with the {\AVTZ} basis set or further corrected for basis set effects. The details of the approach used to get all the TBE are given as well.
For all states with a dominant single-excitation character (that is when $\Td > 80\%$), we rely on CC results, using the incremental strategy to determine the TBE. For ES with $\Td$ between $70\%$ and $80\%$,
our previous works indicated that {\CCSDT} tends to overshoot the transition energies by roughly $0.05$--$0.10$ eV, but that the {\NEV} error tends to be slightly larger (on average). \cite{Loo19c} Therefore, if {\CCSDTQ} or {\FCI}
results are unavailable, it is hard to make the final call. For the other transitions, we relied either on the current or previous FCI or the {\NEV} values as reference. We indicate some transition energies
in italics in Table \ref{Table-tbe} to stress that they are (relatively) less accurate. This is the case when: i) {\NEV} results have to be selected; ii) the affordable CC calculations yield quite large changes from one expansion order to another
despite large $\Td$; and iii) there is a very large ES mixing making hard to follow a specific transition from one method (or one basis) to another. To determine the basis set corrections beyond augmented triple-$\zeta$,
we use the {\CCT}/{\AVQZ} or {\CCT}/{\AVPZ} results. For several compounds, we also provide in the SI, {\CCT}/{\DAVQZ} transition energies. However, we are not using these values as reference. This is because,
the addition of a second set of diffuse orbitals tends to modify the computed transition energies significantly only when it induces a more complex state mixing. We also stick to the
frozen-core approximation for two reasons: i) the corrections brought by ``full-correlation'' are generally trifling (typically $\pm 0.02$ eV) for the compounds under study (see the SI for many examples); and ii) it would be,
in principle, necessary to add core polarization functions in such ``full'' calculations.
Table \ref{Table-tbe} reports our two sets of TBE: a set obtained with the {\AVTZ} basis set and one set with an additional correction for the one-electron basis set incompleteness error. The details of our protocol employed to generate these TBE are also provided in Table \ref{Table-tbe}.
For all states with a dominant single-excitation character (that is when $\Td > 80\%$), we rely on CC results using an incremental strategy to generate these TBE. For ES with $\Td$ between $70\%$ and $80\%$,
our previous works indicated that {\CCSDT} tends to overshoot the transition energies by roughly $0.05$--$0.10$ eV, but that {\NEV} errors tend to be, on average, slightly larger. \cite{Loo19c} Therefore, if {\CCSDTQ} or {\FCI}
results are not available, it is extremely difficult to make the final call. For the other transitions, we relied either on the current or previous FCI data or the {\NEV} values as reference. The italicized transition energies in Table \ref{Table-tbe} are believed to be (relatively) less accurate. This is the case when: i) {\NEV} results have to be selected; ii) the CC calculations yield quite large changes in excitation energies while incrementing the excitation order by one unit despite large $\Td$;
and iii) there is a very strong ES mixing making hard to follow a specific transition from one method (or one basis) to another.
Table \ref{Table-tbe} encompasses 238 ES, all at least obtained at the {\CCSDT} level. This set that can be decomposed as follows: 144 singlet and 94 triplet transitions, or 174 valence (99 $\pi \ra \pis$, 71
$n \ra \pis$ and 4 double excitation) and 64 Rydberg transitions. Amongst the reported transition energies, fifteen can be considered as ``unsafe''. This is a significant extension of all previously
proposed ES datasets (see Introduction). The Table also reports 90 oscillator strengths, $f$ which is, to our knowledge, the largest set of {\CCT}/{\AVTZ} $f$ reported to date, the previous effort being mostly performed
at the {\CCT}/TZVP level for Thiel's set. \cite{Kan14} It should also be recalled that all these data are obtained on {\CCT}/{\AVTZ} geometries, consistently with our previous works. \cite{Loo18a,Loo19c} Taken
together they offer a consistent ensemble of transition energies of ca. 350 electronic transitions of various natures in small and medium-sized organic compounds.
To determine the basis set corrections beyond augmented triple-$\zeta$,
we use the {\CCT}/{\AVQZ} or {\CCT}/{\AVPZ} results. For several compounds, we also provide in the SI, {\CCT}/d-{\AVQZ} transition energies (\ie, with an additional set of diffuse functions). However, we do not consider such values as reference because
the addition of a second set of diffuse functions only significantly modifies the transition energies for strongly-mixing ES.
We also stick to the frozen-core approximation for two reasons: i) the effect of correlating the core electrons is generally negligible (typically $\pm 0.02$ eV) for the compounds under study (see the SI for examples); and ii) it would be,
in principle, necessary to add core polarization functions in such cases.
Table \ref{Table-tbe} encompasses 238 ES, each of them obtained, at least, at the {\CCSDT} level. This set can be decomposed as follows: 144 singlet and 94 triplet transitions, or 174 valence (99 $\pi \ra \pis$, 71
$n \ra \pis$ and 4 double excitations) and 64 Rydberg transitions. Amongst these transition energies, fifteen can be considered as ``unsafe'' and are reported in italics accordingly. This definitely corresponds to a very significant extension of our previous
ES data sets (see Introduction).
Taken all together, they offer a consistent, diverse and accurate ensemble of transition energies for approximately 350 electronic transitions of various natures in small and medium-sized organic molecules.
Table \ref{Table-tbe} also reports 90 oscillator strengths, $f$, which makes it, to the best of our knowledge, the largest set of {\CCT}/{\AVTZ} oscillator strengths reported to date, the previous effort being mostly performed
at the {\CCT}/TZVP level for Thiel's set. \cite{Kan14} It should also be pointed out that all these data are obtained on {\CCT}/{\AVTZ} geometries, consistently with our previous works. \cite{Loo18a,Loo19c}
\renewcommand*{\arraystretch}{.55}
\LTcapwidth=\textwidth
\begin{footnotesize}
\begin{longtable}{p{3.3cm}lcccccc}
\caption{\small TBE values determined for all considered states. For each state, we provide the oscillator strength and percentage of single excitations obtained at the \CCT/{\AVTZ} level,
as well TBE obtained with the same basis set together with the method used to obtain that TBE. In the right-most columns, we list the values obtained by including further basis set corrections,
that have always been obtained at the {\CCT} level. Values displayed in italics are likelt relatively less accurate. All values are in the FC approximation.} \label{Table-tbe}\\
\caption{\small TBE values (in eV) for all considered states alongside their corresponding oscillator strength, $f$, and percentage of single excitations, $\Td$, obtained at the \CCT/{\AVTZ} level.
The composite protocol to generate these TBE is also reported (see footnotes). In the right-most column, we list the TBE values obtained by including an additional correction (obtained at the {\CCT} level) for basis set incompleteness error.
Values displayed in italics are likely to be relatively less accurate. All values are obtained in the FC approximation.} \label{Table-tbe}\\
\hline
& & & & \mc{2}{l}{TBE/{\AVTZ}} & \mc{2}{l}{TBE/CBS} \\
& State & $f$ & $\Td$ & Value & Method$^a$ & Value & Corr. \\
& State & $f$ & $\Td$ & Value & Protocol$^a$ & Value & Corr. \\
\hline
\endfirsthead
\hline
& & & & \mc{2}{l}{TBE/{\AVTZ}} & \mc{2}{l}{TBE/CBS} \\
& State & $f$ & $\Td$ & Value & Method $^a$ & Value & Corr. \\
& State & $f$ & $\Td$ & Value & Protocol $^a$ & Value & Corr. \\
\hline
\endhead
\hline \mc{7}{r}{{Continued on next page}} \\
@ -1390,12 +1392,12 @@ Triazine &$^1A_1'' (\Val; n \ra \pis)$ & & 88.3 & 4.72 & {\CCSDT}/\AVTZ
\begin{flushleft}\begin{footnotesize}\begin{singlespace}
\vspace{-0.6 cm}
$^a${
Method A: {\CCSDT}/{\AVTZ} value corrected by the difference between {\CCSDTQ}/{\AVDZ} and {\CCSDT}/{\AVDZ};
Method B: {\CCSDT}/{\AVTZ} value corrected by the difference between {\CCSDTQ}/{\Pop} and {\CCSDT}/{\Pop};
Method C: {\CCT}/{\AVTZ} value corrected by the difference between {\CCSDTQ}/{\Pop} and {\CCT}/{\Pop};
Method D: {\CCT}/{\AVTZ} value corrected by the difference between {\CCSDT}/{\AVDZ} and {\CCT}/{\AVDZ};
Method E: {\CCT}/{\AVTZ} value corrected by the difference between {\CCSDT}/{\Pop} and {\CCT}/{\Pop};
Method F: {\FCI}/{\AVDZ} value (from Ref.~\citenum{Loo19c}) corrected by the difference between {\CCSDT}/{\AVTZ} and {\CCSDT}/{\AVDZ}.
Protocol A: {\CCSDT}/{\AVTZ} value corrected by the difference between {\CCSDTQ}/{\AVDZ} and {\CCSDT}/{\AVDZ};
Protocol B: {\CCSDT}/{\AVTZ} value corrected by the difference between {\CCSDTQ}/{\Pop} and {\CCSDT}/{\Pop};
Protocol C: {\CCT}/{\AVTZ} value corrected by the difference between {\CCSDTQ}/{\Pop} and {\CCT}/{\Pop};
Protocol D: {\CCT}/{\AVTZ} value corrected by the difference between {\CCSDT}/{\AVDZ} and {\CCT}/{\AVDZ};
Protocol E: {\CCT}/{\AVTZ} value corrected by the difference between {\CCSDT}/{\Pop} and {\CCT}/{\Pop};
Protocol F: {\FCI}/{\AVDZ} value (from Ref.~\citenum{Loo19c}) corrected by the difference between {\CCSDT}/{\AVTZ} and {\CCSDT}/{\AVDZ}.
}
\end{singlespace}\end{footnotesize}\end{flushleft}
@ -1440,42 +1442,49 @@ CCSDT-3 &126 &0.05 &0.05 &0.07 &0.04 &0.26 &0.00\\
\label{Fig-1}
\end{figure}
Let us analyse the global performances of all methods, starting with the most refined models. The relative accuracies of {\CCT} and {\CCSDT}-3 as compared to {\CCSDT} remains an open question in the literature. \cite{Wat13,Dem13}
Indeed, to our knowledge, the only two previous reports discussing this specific aspect are limited to tiny compounds. \cite{Kan17,Loo18a} According to the results of Table \ref{Table-bench}, it appears that {\CCT} has the edge, although
{\CCSDT}-3 is closer to {\CCSDT} in formal terms. Indeed, {\CCSDT}-3 seems to provide slightly too large transition energies (MSE of $+0.05$ eV). These conclusions are qualitatively consistent with the analyses performed for smaller derivatives,
\cite{Kan17,Loo18a} but the amplitude of {\CCSDT}-3's errors are larger with the present set. Although the performances of {\CCT} might be unduly inflated by the use of {\CCSDT} and {\CCSDTQ} reference values, it is also clear that this
method very rarely fails (Figure \ref{Fig-1}). Consequently, {\CCT} transition energies can be viewed as very solid references for all transitions with a dominant single-excitation character. This conclusion is again consistent with previous
analyses performed for smaller compounds, \cite{Kan17,Loo18a} as well as with recent comparisons between theoretical and experimental 0-0 energies that have been performed by some of us on medium-sized molecules.\cite{Loo18b,Loo19a,Sue19}
To state it more bluntly: it appears likely that {\CCT} is even more accurate than previously thought. In addition, from all the comparisons made in this work, one can conclude that {\CCT} regularly outperforms {\CASPT} and {\NEV}, even when
these methods are combined with active space chosen by specialists, a statement that seems true as long as the considered ES does not show a strong multiple excitation character, that is, when $\Td < 70\%$. The perturbative
inclusion of triples as made in CCSDR(3) offers a very small MAE ($0.05$ eV) for a much reduced computational cost as compared to {\CCSDT}. Nevertheless, as with {\CCSDT}-3, the CCSDR(3) transition energies have a clear tendency
of being too large, an error sign likely inherited from the parent {\CCSD} model. This $0.05$ eV MAE for CCSDR(3) is rather similar to the one obtained for small compounds when comparing to {\FCI} ($0.04$ eV), \cite{Loo18a} and is also inline with the
2009 benchmark of Sauer et al. \cite{Sau09} {\CCSD} provides an interesting case. The calculated MSE ($+0.11$ eV), indicating an overestimation of the transition energies, fits well many previous reports,
\cite{Sch08,Car10,Wat13,Kan14,Jac17b,Kan17,Dut18,Jac18a,Loo18a} but is larger than the one determined for smaller molecules ($+0.05$ eV), \cite{Loo18a} hinting that the performances of {\CCSD} deteriorates when larger compounds
are considered. The {\CCSD} MAE ($0.13$ eV) is much smaller than the one reported by Thiel in its original work ($0.49$ eV) \cite{Sch08} but of the same order of magnitude as in the more recent study of Kannar and Szalay performed
on Thiel's set ($0.18$ eV for transitions with $\Td > 90\%$ ). \cite{Kan14} In retrospect, the much larger value obtained by Thiel is likely related to the use of {\CASPT} reference values in the 2008 work. Indeed, as we have shown
in many of the proposed examples, {\CASPT} transitions energies tend to be significantly too low, therefore exacerbating {\CCSD}'s overestimation. The {\STEOM} approach, which received relatively less attention to date -- we are
aware of one detailed benchmark \cite{Dut18} only -- provides a smaller MSE than {\CCSD} and comparable MAE and RMSE. The spread of the error is however slightly larger as can be seen in Figure \ref{Fig-1} and from the SD values
in Table \ref{Table-bench}. These trends are the same as for smaller compounds. \cite{Loo18a} For Thiel's set using {\CCT}/TZVP results as references, Dutta and coworkers also reported a rather good performance
of {\STEOM}, though in that case a slightly negative MSE is obtained, \cite{Dut18} which could possibly be due to the different basis sets used. It should be nevertheless stressed that we consider here only ``clean'' {\STEOM} results
(see Computational details), therefore removing several difficult cases that are included in the {\CCSD} statistics, \eg, the $A_g$ excitation in butadiene, which can slightly bias the relative accuracies when comparing the two models. Finally, for the three
second-order methods, namely CIS(D), {\AD}, and {\CCD}, that are often used as reference to benchmark TD-DFT for ``real-life'' applications, we obtain clearly worse performances for the former approach than for the two latter, that show very
similar statistical behaviors. These trends were also reported in several previous works. \cite{Hat05c,Jac18a,Sch08,Sil10c,Win13,Har14,Jac15b,Kan17,Loo18a} Interestingly, the {\CCD} MAE obtained here, $0.15$ eV, is significantly
smaller than the one we found for the smaller compounds ($0.22$ eV): \cite{Loo18a} in contrast to {\CCSD}, {\CCD} seems to improve with molecular size. As above, Thiel's original MAE for {\CCD} ($0.29$ eV) was likely too large due
Let us analyse the global performances of all methods, starting with the most accurate and computationally demanding models. The relative accuracies of {\CCT} and {\CCSDT}-3 as compared to {\CCSDT} remains an open question in the literature. \cite{Wat13,Dem13}
Indeed, to the best of our knowledge, the only two previous studies discussing this specific aspect are limited to very small compounds. \cite{Kan17,Loo18a} According to the results gathered in Table \ref{Table-bench}, it appears that {\CCT} has a slight edge over {\CCSDT}-3, although
{\CCSDT}-3 is closer to {\CCSDT} in \titou{formal} terms. Indeed, {\CCSDT}-3 seems to provide slightly too large transition energies (MSE of $+0.05$ eV). These conclusions are qualitatively consistent with the analyses performed on smaller derivatives,
\cite{Kan17,Loo18a} but the amplitude of the {\CCSDT}-3 errors is larger in the present set. Although the performances of {\CCT} might be unduly inflated by the use of {\CCSDT} and {\CCSDTQ} reference values, it is also clear that this
method very rarely fails (Figure \ref{Fig-1}). Consequently, {\CCT} transition energies can be viewed as extremely solid references for any transition with a dominant single-excitation character. This conclusion is again consistent with previous
analyses performed on smaller compounds, \cite{Kan17,Loo18a} as well as with recent comparisons between theoretical and experimental 0-0 energies performed by some of us on medium-sized molecules. \cite{Loo18b,Loo19a,Sue19}
To state it more boldly: it appears likely that {\CCT} is even more accurate than previously thought. In addition, thanks to the exhaustive and detailed comparisons made in the present work, we can safely conclude that {\CCT} regularly outperforms {\CASPT} and {\NEV}, even when
these methods are combined with large active spaces. This statement seems to hold as long as the considered ES does not show a strong multiple excitation character, that is, when $\Td < 70\%$.
The perturbative inclusion of triples as in CCSDR(3) yields a very small MAE ($0.05$ eV) for a much lighter computational cost as compared to {\CCSDT}. Nevertheless, as with {\CCSDT}-3, the CCSDR(3) transition energies have a clear tendency
of being too large, an error sign likely inherited from the parent {\CCSD} model. This $0.05$ eV MAE for CCSDR(3) is rather similar to the one obtained for smaller compounds when comparing to {\FCI} ($0.04$ eV), \cite{Loo18a} and is also inline with the
2009 benchmark study of Sauer et al. \cite{Sau09}
{\CCSD} provides an interesting case. The calculated MSE ($+0.11$ eV), indicating an overestimation of the transition energies, fits well with several previous recent reports.
\cite{Sch08,Car10,Wat13,Kan14,Jac17b,Kan17,Dut18,Jac18a,Loo18a}
It is, nonetheless, larger than the one determined for smaller molecules ($+0.05$ eV), \cite{Loo18a} hinting that the performances of {\CCSD} deteriorates for larger compounds.
Moreover, {\CCSD} MAE of $0.13$ eV is much smaller than the one reported by Thiel in its original work ($0.49$ eV) \cite{Sch08} but of the same order of magnitude as in the more recent study of K\'ann\'ar and Szalay performed
on Thiel's set ($0.18$ eV for transitions with $\Td > 90\%$). \cite{Kan14} Retrospectively, it is pretty obvious that Thiel's much larger value is very likely due to the {\CASPT} reference values. \cite{Sch08}
Indeed, as we have shown several times in the present study, {\CASPT} transitions energies tend to be significantly too low, therefore exacerbating the usual {\CCSD} overestimation.
With a single detailed benchmark study to date, \cite{Dut18} the {\STEOM} approach has received relatively little attention and its overall accuracy still needs to be corroborated.
It is noteworthy that {\STEOM} provides a smaller MSE than {\CCSD} and comparable MAE and RMSE. The spread of the error is however slightly larger as evidenced by Figure \ref{Fig-1} and the SDE values
reported in Table \ref{Table-bench}. These trends are the same as for smaller compounds. \cite{Loo18a} For Thiel's set, Dutta and coworkers also reported rather good performance
for {\STEOM} with respect to the {\CCT}/TZVP reference data, though a slightly negative MSE is obtained in their case. \cite{Dut18} This could well be due to the different basis set considered by these two studies. It should be nevertheless stressed that we, here, only consider only ``clean'' {\STEOM} results
(see Computational Details), therefore removing several difficult cases that are included in the {\CCSD} statistics, \eg, the $A_g$ excitation in butadiene, which can slightly bias the relative performance of {\STEOM} and {\CCSD}.
Finally, for the three second-order methods, namely CIS(D), {\AD}, and {\CCD}, that are often used as reference to benchmark TD-DFT for ``real-life'' applications, the performance of the former method clearly deteriorates compared to the two latter which exhibit very
similar statistical behaviors. These trends were also reported in previous works. \cite{Hat05c,Jac18a,Sch08,Sil10c,Win13,Har14,Jac15b,Kan17,Loo18a} Interestingly, the {\CCD} MAE obtained here ($0.15$ eV) is significantly
smaller than the one we found for smaller compounds ($0.22$ eV). \cite{Loo18a} Therefore, in contrast to {\CCSD}, {\CCD} performance seems to improve with molecular size. As above, Thiel's original MAE for {\CCD} ($0.29$ eV) was likely too large due
to the selection of {\CASPT} reference values. \cite{Sch08} As already noticed by Szalay's group, \cite{Kan14,Kan17} although the MSE of {\CCD} is smaller than the one of {\CCSD}, the standard deviation is significantly larger
with the former model, \ie, {\CCD} is less consistent in terms of trends than {\CCSD}.
In Table \ref{Table-bench2}, we report a decomposition of the MAE for different subsets of ES. Only singlet ES could be determined with both CCSDR(3) and CCSDT-3, which is why no value appears in the triplet
column for these two methods. A few interesting conclusions emerge from the displayed data. First, the errors for the singlet and triplet transitions are rather similar with all models, but with {\CCSD} that
is significantly more effective for the triplets. Dutta and coworkers obtained the same conclusions for Thiel's set with MAE of $0.20$ eV and $0.11$ eV for the singlet and triplet ES, respectively. \cite{Dut18}
When turning to the comparison between valence and Rydberg states, it is found that {\CCD} actually performs more effectively for the former, whereas {\CCSD} (and higher order methods) yields the opposite trend.
In fact {\CCD} has a tendency to overestimate the energies of the valence ES (MSE of $+0.10$ eV), but to underestimate their Rydberg counterparts (MSE of $-0.17$ eV), whereas {\CCSD} is much more consistent
with MSE of $0.12$ and $0.09$ eV, respectively (see the SI). This relatively poorer performance of {\CCD} as compared to {\CCSD} for Rydberg ES is again consistent with other benchmarks, \cite{Kan17,Dut18} although the MAE
for {\CCD} ($0.18$ eV) reported in Table \ref{Table-bench2} remains relatively small as compared to the one given in Ref.~\citenum{Kan17}. This is likely a side effect of the consideration of (relatively) low-lying
Rydberg transitions in medium-sized molecules in the present work, whereas Kannar and Szalay (mostly) investigated higher-lying Rydberg in smaller compounds. Eventually, CIS(D), {\AD}, {\CCD}, and {\STEOM}
better describe $n\ra\pis$ transitions, whereas {\CCSD} seems more suited for $\pi\ra\pis$ transitions; the variations between the two subsets being probably not very significant for the
higher-order approaches. The former finding agrees with the results obtained for smaller compounds, \cite{Loo18a} as well for Thiel's set, \cite{Sch08,Kan14} whereas the latter, less expected conclusions, seems to be significantly
dependent on the selected subset of ES. \cite{Sch08,Kan17}
In Table \ref{Table-bench2}, we report a MAE decomposition for different subsets of ES. Note that, due to implementational limitations, only singlet ES could be computed with CCSDR(3) and CCSDT-3 which explains the lack of data for triplet ES.
A few interesting conclusions emerge from these results. First, the errors for singlet and triplet transitions are rather similar with all models, except for {\CCSD} that
is significantly more effective for triplets. Dutta and coworkers observed the same trend for Thiel's set with MAE of $0.20$ eV and $0.11$ eV for the singlet and triplet ES, respectively. \cite{Dut18}
Turning to the comparison between valence and Rydberg states, we find that {\CCD} provide a better description of the former, whereas {\CCSD} (and higher-order methods) yields the opposite trend.
In fact, {\CCD} has the clear tendency to overestimate valence ES energies (MSE of $+0.10$ eV), and to underestimate Rydberg ES energies (MSE of $-0.17$ eV).
{\CCSD} is found to be much more consistent with MSE of $0.12$ and $0.09$ eV, respectively (see the SI). This relatively poorer performance of {\CCD} as compared to {\CCSD} for Rydberg ES is again perfectly coherent with other benchmarks, \cite{Kan17,Dut18} although the MAE
for {\CCD} ($0.18$ eV) reported in Table \ref{Table-bench2} remains relatively small as compared to the one given in Ref.~\citenum{Kan17}. We believe that it is likely due to the distinct types of Rydberg states considered in these two studies. Indeed, we consider here (relatively) low-lying
Rydberg transitions in medium-sized molecules, while K\'ann\'ar and Szalay (mostly) investigated higher-lying Rydberg states in smaller compounds. Finally, CIS(D), {\AD}, {\CCD}, and {\STEOM}
better describe $n\ra\pis$ transitions, whereas {\CCSD} seems more suited for $\pi\ra\pis$ transitions.
The variations between the two subsets are probably not significant for the
higher-order approaches. The former observation agrees well with previous results on smaller compounds \cite{Loo18a} and the Thiel set, \cite{Sch08,Kan14} whereas \titou{the latter, less expected observation is likely dependent on the selected ES subset.} \cite{Sch08,Kan17}
\renewcommand*{\arraystretch}{1.0}
\begin{table}[htp]