\newcommand{\LCPQ}{Laboratoire de Chimie et Physique Quantiques, Universit\'e de Toulouse, CNRS, UPS, France}
\newcommand{\CEISAM}{Laboratoire CEISAM - UMR CNRS 6230, Universit\'e de Nantes, 2 Rue de la Houssini\`ere, BP 92208, 44322 Nantes Cedex 3, France}
\newcommand{\Pisa}{Dipartimento di Chimica e Chimica Industriale, University of Pisa, Via Moruzzi 3, 56124 Pisa, Italy}
\title{Highly-Accurate Reference Excitation Energies and Benchmarks: Medium Size Molecules}
\author{Pierre-Fran{\c c}ois Loos}
\email{loos@irsamc.ups-tlse.fr}
\affiliation[LCPQ, Toulouse]{\LCPQ}
\author{Filippo Lipparini}
\affiliation[DC, Pisa]{\Pisa}
\email{filippo.lipparini@unipi.it}
\author{Martial Boggio-Pasqua}
\affiliation[LCPQ, Toulouse]{\LCPQ}
\author{Anthony Scemama}
\affiliation[LCPQ, Toulouse]{\LCPQ}
\author{Denis Jacquemin}
\email{Denis.Jacquemin@univ-nantes.fr}
\affiliation[UN, Nantes]{\CEISAM}
\begin{document}
\begin{abstract}
Following our previous work focussing on compounds containing up to 3 non-hydrogen atoms [\emph{J. Chem. Theory Comput.}{\bfseries 14} (2018) 4360--4379], we present here highly-accurate vertical transition energies
To obtain these energies, we use equation-of-motion coupled cluster theory up to the highest technically possible excitation order for these systems (CC3, EOM-CCSDT, and EOM-CCSDTQ), selected configuration interaction (SCI) calculations (with tens of millions of determinants in the reference space),
All these approaches are applied in combination with diffuse-containing atomic basis sets. For all transitions, we report at least CC3/\emph{aug}-cc-pVQZ vertical excitation
energies as well as CC3/\emph{aug}-cc-pVTZ oscillator strengths for each dipole-allowed transition. We show that CC3 almost systematically delivers transition energies in agreement with higher-level methods with a typical deviation of $\pm0.04$ eV, except for transitions with a dominant double excitation character where the error is much larger.
Accurately describing transition energies between the electronic ground state (GS) and excited states (ES) remains an important challenge in quantum chemistry. When dealing with large compounds in complex environments,
one is typically limited to the use of time-dependent density-functional theory (TD-DFT), \cite{Cas95,Ulr12b,Ada13a} a successful yet far from flawless approach. In particular, to perform TD-DFT calculations, one must
choose an ``appropriate'' exchange-correlation functional, which is difficult yet primordial as the impact of the exchange-correlation functional is exacerbated within TD-DFT as compared to DFT. \cite{Lau13} Such selection can, of course, rely
on the intrinsic features of the various exchange-correlation functional families, \eg, it is well-known that range-separated hybrids provide a more physically-sound description of long-range charge-transfer transitions than semi-local exchange-correlation functionals. \cite{Dre04,Pea08}
However, to obtain a quantitative assessment of the accuracy that can be expected from TD-DFT calculations, benchmarks cannot be avoided. This is why so many assessments of TD-DFT performances for various properties are
While several of these benchmarks rely on experimental data as reference (typically band shapes \cite{Die04,Die04b,Avi13,Cha13,Lat15b,Mun15,Vaz15,San16b} or 0-0 energies
\cite{Die04b,Goe10a,Jac12d,Chi13b,Win13,Fan14b,Jac14a,Jac15b,Loo19b}), references from theoretical best estimates (TBE) based on state-of-the-art computational methods \cite{Sch08,Sau09,Sil10b,Sil10c,Sch17,Loo18a}
are advantageous as they allow comparisons on a perfectly equal footing (same geometry, vertical transitions, no environmental effects, etc). In such a case, the challenge is in fact to obtain accurate TBE, as these top-notch theoretical models
generally come with a dreadful scaling with system size and, in addition, typically require large atomic basis sets to deliver transition energies close to the complete basis set (CBS) limit. \cite{Gin19}
More than 20 years ago, Serrano-Andr\`es, Roos, and collaborators compiled an impressive series of reference transition energies for several typical conjugated organic molecules (butadiene, furan, pyrrole, tetrazine, \ldots).
\cite{Ful92,Ser93,Ser93b,Ser93c,Lor95b,Mer96,Mer96b,Roo96,Ser96b} To this end, they relied on experimental GS geometries and the complete-active-space second-order perturbation theory ({\CASPT}) approach with the largest active spaces and basis sets
These {\CASPT} values were later used to assess the performances of TD-DFT combined with various exchange-correlation functionals, \cite{Toz99b,Bur02} and remained for a long time the best theoretical references available on the market.
However, beyond comparisons with experiments, which are always challenging when computing vertical transition energies, \cite{San16b} there was no approach available at that time to ascertain the accuracy of these transition energies.
Nowadays, it is common knowledge that CASPT2 has the tendency of underestimating vertical excitation energies in organic molecules.
A decade ago, Thiel and coworkers defined TBE for 104 singlet and 63 triplet valence ES in 28 small and medium conjugated CNOH organic molecules. \cite{Sch08,Sil10b,Sil10c} These TBE
were computed on MP2/6-31G(d) structures with several levels of theories, notably {\CASPT} and various coupled cluster (CC) variants ({\CCD}, {\CCSD}, and {\CCT}). Interestingly, while the default theoretical protocol used by Thiel and coworkers to define their
first series of TBE was {\CASPT}, \cite{Sch08} the vast majority of their most recent TBE (the so-called ``TBE-2'' in Ref.~\citenum{Sil10c}) were determined at the {\CCT}/{\AVTZ} level of theory, often using a basis set extrapolation technique.
More specifically, CC3/TZVP values were corrected for basis set incompleteness errors by the difference between {\CCD}/{\AVTZ} and {\CCD}/TZVP results. \cite{Sil10b,Sil10c} Many works exploited Thiel's TBE for
assessing low-order methods, \cite{Sil08,Goe09,Jac09c,Roh09,Sau09,Jac10c,Jac10g,Sil10,Mar11,Jac11a,Hui11,Del11,Tra11,Pev12,Dom13,Dem13,Sch13b,Voi14,Har14,Yan14b,Sau15,Pie15,Taj16,Mai16,Ris17,Dut18,Hel19} highlighting further
their value for the electronic structure community. In contrast, the number of extensions/improvements of this original set remains quite limited.
For example, K\'ann\'ar and Szalay computed, in 2014, {\CCSDT}/TZVP reference energies for 17 singlet states of six molecules. \cite{Kan14}
Three years later, the same authors reported 46 {\CCSDT}/{\AVTZ} transition energies in small compounds containing two or three non-hydrogen atoms (ethylene, acetylene, formaldehyde, formaldimine, and formamide). \cite{Kan17}
Following the same philosophy, two years ago, we reported a set of 106 transition energies for which it was technically possible to reach the full configuration interaction (FCI) limit by performing high-order CC (up to {\CCSDTQP}) and selected
CI (SCI) calculations on {\CCT}/{\AVTZ} GS structures. \cite{Loo18a} We exploited these TBE to benchmark many ES methods. \cite{Loo18a} Amongst our conclusions, we found that {\CCSDTQ} yields near-{\FCI} quality excitation energies, whereas we could not
detect any significant differences between {\CCT} and {\CCSDT} transition energies, both being very accurate with mean absolute errors (MAE) as small as $0.03$ eV compared to {\FCI}.
Although these conclusions agree well with earlier studies, \cite{Wat13,Kan14,Kan17} they obviously only hold for single excitations, \ie, transitions with $\Td$ in the range $80$--$100\%$. Therefore, we also recently proposed a set of 20 TBE for transitions exhibiting a significant double-excitation character (\ie, with $\Td$ typically below $80\%$). \cite{Loo19c}
Unsurprisingly, our results clearly evidenced that the error in CC methods is intimately related to the $\Td$ value.
CC methods including triples deliver rather accurate estimates (MAE of $0.11$ eV with {\CCT} and $0.06$ eV with {\CCSDT}), surprisingly outperforming second-order multi-reference schemes such as {\CASPT} or
the generally robust $n$-electron valence state perturbation theory ({\NEV}). In contrast, for ES with a dominant double excitation character, \eg, the low-lying $(n,n)\ra(\pis,\pis)$ excitation in nitrosomethane ($\Td=2\%$),
These were limited to $1$--$3$ non-hydrogen atoms, hence introducing a potential ``chemical'' bias. Therefore, we have decided, in the present contribution, to consider larger molecules with organic
compounds encompassing 4, 5, and 6 non-hydrogen atoms. For such systems, performing {\CCSDTQ} calculations with large one-electron basis sets is elusive. Moreover, the convergence of the {\SCI} energy with respect to the number of determinants is obviously slower for these larger compounds, hence
extrapolating to the {\FCI} limit with an error of $\sim0.01$ eV is rarely achievable in practice. Consequently, the ``brute-force'' determination of {\FCI}/CBS estimates, as in our earlier work, \cite{Loo18a} is definitely out of reach here.
Anticipating this problem, we have recently investigated bootstrap CBS extrapolation techniques. \cite{Loo18a,Loo19c}
In particular, we have demonstrated that, following an ONIOM-like scheme, \cite{Chu15} one can very accurately estimate such limit by correcting high-level values obtained in a small basis by the difference between {\CCT} results obtained in a larger basis and in the same small basis.\cite{Loo18a}
We globally follow such strategy here. In addition, we also perform {\NEV} calculations in an effort to check the consistency of our estimates. This is particularly critical for ES with intermediate $\Td$ values.
Using this protocol, we define a set of more than 200 \emph{aug}-cc-pVQZ reference transition energies, most being within $\pm0.03$ eV of the {\FCI} limit. These reference energies are obtained on {\CCT}/\emph{aug}-cc-pVTZ geometries and additional basis set
corrections (up to quadruple-$\zeta$ at least) are also provided for {\CCT}. Together with the results obtained in our two earlier works, \cite{Loo18a,Loo19c} the present TBE will hopefully contribute to climb a rung higher on the ES accuracy ladder.
Pople's {\Pop} and Dunning's \emph{aug}-cc-pVXZ (X $=$ D, T, Q, and 5) atomic basis sets are systematically employed in our excited-state calculations.
Various statistical quantities are reported in the remaining of this paper: the mean signed error (MSE), mean absolute error (MAE), root mean square error (RMSE), standard deviation of the errors (SDE), as well as the positive [\MaxP] and negative [\MaxN] maximum errors.
Here, we globally follow the same procedure as in Ref.~\citenum{Loo18a}, so that we only briefly outline the various theoretical methods that we have employed in the subsections below.
Consistently with our previous work, \cite{Loo18a} we systematically use {\CCT}/{\AVTZ} GS geometries obtained without applying the frozen-core approximation.
Several structures have been extracted from previous contributions, \cite{Bud17,Jac18a,Bre18a} whereas the missing structures were optimized using DALTON \cite{dalton} and/or CFOUR, \cite{cfour} applying default parameters in both cases.
All the SCI calculations have been performed in the frozen-core approximation with the latest version of QUANTUM PACKAGE \cite{Gar19} using the Configuration Interaction using a Perturbative Selection made Iteratively (CIPSI) algorithm to select the
most important determinants in the FCI space. Instead of generating all possible excited determinants like a conventional CI calculation, the iterative CIPSI algorithm performs a sparse exploration of the FCI space via a selection of the most relevant
determinants using a second-order perturbative criterion. At each iteration, the variational (or reference) space is enlarged with new determinants. CIPSI can be seen as a deterministic version of the FCIQMC algorithm developed by Alavi and
coworkers. \cite{Boo09} We refer the interested reader to Ref.~\citenum{Gar19} where our implementation of the CIPSI algorithm is detailed.
Excited-state calculations are performed within a state-averaged formalism which means that the CIPSI algorithm select determinants simultaneously for the GS and ES. Therefore, all electronic states share the same set of determinants with different CI coefficients.
Our implementation of the CIPSI algorithm for ES is detailed in Ref.~\citenum{Sce19}. For each system, a preliminary SCI calculation is performed using Hartree-Fock orbitals in order to generate SCI wavefunctions with at least 5,000,000 determinants.
State-averaged natural orbitals are then computed based on this wavefunction, and a new, larger SCI calculation is performed with this new set of orbitals. This has the advantage to produce a smoother and faster convergence of the SCI energy towards the FCI limit.
For the largest systems, an additional iteration is sometimes required in order to obtain better quality natural orbitals and hence well-converged calculations.
The total SCI energy is defined as the sum of the (zeroth-order) variational energy (computed via diagonalization of the CI matrix in the reference space) and a second-order perturbative correction which takes into account the external determinants, \ie,
the determinants which do not belong to the variational space but are linked to the reference space via a non-zero matrix element. The magnitude of this second-order correction, $E^{(2)}$, provides a qualitative idea of the ``distance" to the FCI limit.
For maximum efficiency, the total SCI energy is linearly extrapolated to $E^{(2)}=0$ (which effectively corresponds to the FCI limit) using the two largest SCI wavefunctions. These extrapolated total energies (simply labeled as FCI in the remaining of the paper)
are then used to computed vertical excitation energies. Although it is not possible to provide a theoretically-sound error bar, we estimate the extrapolation error by the difference in excitation energy between the largest SCI wavefunction and its corresponding
extrapolated value. We believe that it provides a very safe estimate of the extrapolation error. Additional information about the SCI wavefunctions and excitation energies as well as their extrapolated values can be found in the SI.
The {\NEV} calculations have been performed with MOLPRO \cite{molpro} within the partially-contracted scheme ({\PNEV}), which is theoretically superior to its strongly-contracted version due to the larger number of perturbers and greater flexibility. \cite{Ang01,Ang01b,Ang02}
These {\NEV} calculations are performed on top of a state-averaged complete-active-space self-consistent field (SA$n$-CASSCF) calculation (where $n$ is the number of states that has been averaged).
The definition of the active space considered for each system as well as the number of states in the state-averaged calculation is provided in the SI.
For the other levels of theory, we apply a variety of programs, namely, CFOUR,\cite{cfour} DALTON,\cite{dalton} GAUSSIAN,\cite{Gaussian16} ORCA,\cite{Nee12} MRCC,\cite{Rol13,mrcc} and Q-CHEM. \cite{Sha15} CFOUR is used for
{\CCT}, \cite{Chr95b,Koc97} CCSDT-3, \cite{Wat96,Pro10}{\CCSDT}\cite{Nog87} and {\CCSDTQ}\cite{Kuc91}; Dalton for {\CCD}, \cite{Chr95,Hat00}{\CCSD},\cite{Pur82} CCSDR(3), \cite{Chr96b} and {\CCT}\cite{Chr95b,Koc97}; Gaussian
for CIS(D); \cite{Hea94,Hea95} ORCA for the similarity-transformed EOM-CCSD ({\STEOM})\cite{Noo97,Dut18}; MRCC for {\CCSDT}\cite{Nog87} and {\CCSDTQ}; \cite{Kuc91} and Q-Chem for {\AD}. \cite{Dre15}
Default program settings were applied. We note that for {\STEOM} we only report states that are characterized by an active character percentage of $98\%$ or larger.
In the following, we present results obtained for molecules containing four, five, and six (non-hydrogen) atoms. In all cases, we test several atomic basis sets and push the CC excitation order as high as technically possible.
Given that the {\SCI} energy converges rather slowly for these larger systems, we provide an estimated error bar for these extrapolated {\FCI} values (\emph{vide supra}). In most cases, these extrapolated FCI reference data are used as a ``safety net'' to demonstrate the overall
consistency of the various approaches rather than as definitive reference values (see next Section). As a further consistency check, we also report {\NEV}/{\AVTZ} excitation energies for all states. We underline that, except
when specifically discussed, all ES present a dominant single-excitation character (see also next Section), so that we do not expect serious CC breakdowns. This is especially true for triplet ES that are known to be characterized by
very large $\Td$ values in the vast majority of the cases. \cite{Sch08} Consequently, we concentrate most of our computational effort on the obtention of accurate transition energies for singlet states. To assign the different ES, we use literature data, as well as
the usual criteria, \ie, relative energies, spatial and spin symmetries, compositions from the underlying molecular orbitals, and oscillator strengths. This allows clear-cut assignments for the vast majority of the cases. There are however some
state/method combinations for which strong mixing between ES of the same symmetry makes unambiguous assignments almost impossible.
\titou{Such cases are therefore discarded from our benchmark set.}
$^d${Experimental 0-0 energies from Refs.~\citenum{Cal63} ($^3\Sigma_u^+$), \citenum{Bel69} ($^1\Sigma_u^-$), and \citenum{Fis72} ($^1\Delta_u$), all analyzing vacuum electronic spectra;}
$^e${Experimental 0-0 energies from Ref.~\citenum{Hai79} (singlet ES, vacuum UV experiment) and Ref.~\citenum{All84} (triplet ES, EELS). In the latter contribution, the $2.7$ eV value for the $^3\Sigma_u^+$
state is the onset, whereas an estimate of the vertical energy ($4.2\pm0.2$ eV) is given for the $^3\Delta_u$ state.}
The ES of these three closely related linear molecules containing two triple bonds have been quite rarely theoretically investigated, \cite{Fis03,Pat06,Luo08,Loo18b,Loo19a} though (rather old) experimental measurements of their
0-0 energies are available for several ES. \cite{Cal63,Job66a,Job66b,Bel69,Fis72,Har77,Hai79,All84} Our main results are collected in Tables \ref{Table-1} and S1. We consider only low-lying
valence $\pi\ra\pis$ transitions, which are all characterized by a strongly dominant single excitation nature ($\Td > 90\%$, \emph{vide infra}). For cyanoacetylene, the {\FCI}/{\Pop} estimates come with small error bars, and one notices an excellent agreement between these values and their {\CCSDTQ} counterparts, a statement holding for the Dunning's double-$\zeta$ basis set results for which the {\FCI} uncertainties are however
larger. Using the {\CCSDTQ} values as references, it appears that the previously obtained {\CASPT} estimates\cite{Luo08} are, as expected, too low and that the {\CCT} transition energies are slightly more accurate than
their CCSDT counterparts, although all CC estimates of Table \ref{Table-1} come, for a given basis set, in a very tight energetic window. There is also a very neat agreement between the CC/{\AVTZ} and {\NEV}/{\AVTZ}. All these facts provide strong evidences that the CC estimates can be fully trusted for these three linear systems. The basis set effects are quite significant for the valence ES of cyanoacetylene with successive drops of the transition
energies by approximately $0.10$ eV, when going from {\Pop} to {\AVDZ}, and from {\AVDZ} to {\AVTZ}. The lowest triplet state appears less basis set sensitive, though. As expected,
extending further the basis set size (to quadruple- and quintuple-$\zeta$) leaves the results pretty much unchanged. The same observation holds when adding a second set of diffuse functions, or when correlating the core electrons (see the SI). Obviously,
both cyanogen and diacetylene yield very similar trends, with limited methodological effects and quite large basis set effects, except for the $^1\Sigma_g^+\ra{}^3\Sigma_u^+$ transitions. We note that all CC3 and CCSDT values are, at worst,
within $\pm0.02$ eV of the {\FCI} window, \ie, all methods presented in Table \ref{Table-1} provide very consistent estimates. For all the states reported in this Table, the average absolute deviation between {\NEV}/{\AVTZ} and {\CCT}/{\AVTZ}
({\CCSDT}/{\AVTZ}) is as small as $0.02$ ($0.03$) eV, the lowest absorption and emission energies of cyanogen being the only two cases showing significant deviations. As a final note, all our vertical absorption (emission) energies are significantly larger
(smaller) than the experimentally measured 0-0 energies, as it should. We refer the interested reader to previous works, \cite{Fis03,Loo19a} for comparisons between theoretical ({\CASPT} and {\CCT}) and experimental 0-0 energies for these three compounds.
These three related compounds present a three-member $sp^2$ carbon cycle conjugated to an external $\pi$ bond. While the ES of methylenecyclopropene have regularly been investigated with theoretical tools in the past,
\cite{Mer96,Roo96,Car10b,Lea12,Gua13,Dad14,Gua14,Sch17,Bud17} the only investigations of vertical transitions we could find for the two other derivatives are a detailed {\CASPT} study of Serrano-Andr\'es and
coworkers in 2002, \cite{Ser02} and a more recent work reporting the three lowest-lying singlet states of cyclopropenone at the {\CASPT}/6-31G level.\cite{Liu14b}
$^b${Electron impact experiment from Ref.~\citenum{Har74}. Note that the $5.5$ eV peak was assigned differently in the original paper, and we follow here the analysis of Serrano-Andr\'es, \cite{Ser02}
whereas the $6.1$ eV assignment was ``supposed'' in the original paper; experimental $\lambda_{\mathrm{max}}$ have been measured at $3.62$ eV and $6.52$ eV for the $^1B_1$ ($n \ra\pis$) and
$^e${$\lambda_{\mathrm{max}}$ in pentane at $-78^o$C from Ref.~\citenum{Sta84};}
$^f${Significant state mixing with the $^1A_1$($\pi\ra3p$) transition, yielding unambiguous attribution difficult;}
$^g${As can be seen in the SI, our {\FCI}/{\AVDZ} estimates are $3.45\pm0.04$ and $4.79\pm0.02$ eV for the two lowest triplet states of methylenecyclopropene hinting that the CC3 and CCSDT
Our results are listed in Tables \ref{Table-2} and S2. As above, considering the {\Pop} basis set, we notice very small differences between {\CCT}, {\CCSDT}, and {\CCSDTQ}, the latter method giving transition energies
systematically falling within the {\FCI} extrapolation incertitude, except in one case (the lowest totally symmetric state of methylenecyclopropene for which the {\CCSDTQ} value is ``off'' by $0.02$ eV only). Depending on the state, it is
either {\CCT} or {\CCSDT} that is closest to {\CCSDTQ}. In fact, considering the {\CCSDTQ}/{\Pop} data listed in Table \ref{Table-2} as reference, the MAE of {\CCT} and {\CCSDT} is $0.019$ and $0.016$ eV, respectively,
hinting that the improvement brought by the latter, more expensive method is limited for this set of compounds. For the lowest $B_2$ state of methylenecyclopropene, one of the most challenging cases ($\Td=85\%$),
it is clear from the {\FCI} value that only {\CCSDTQ} is energetically close, the {\CCT} and {\CCSDT} results being slightly too large by $\sim0.05$ eV. It seems reasonable to believe that the same observation can be made for the corresponding state of
cyclopropenethione, although in this case the FCI error bar is too large to prevent any definitive conclusion. Interestingly, at the {\CCT} level of theory, the rather small {\Pop} basis set provides data within $0.10$ eV of the CBS limit for $80\%$ of
the transitions. There are, of course, exceptions to this rule, \eg, the strongly dipole-allowed $^1A_1(\pi\ra\pis)$ ES of cyclopropenone and the $^1B_1(\pi\ra3s)$ ES of methylenecyclopropene which are significantly
over blueshifted with the Pople basis set (Table S2). For cyclopropenone, our {\CCSDT}/{\AVTZ} estimates do agree reasonably well with the {\CASPT} data of Serrano-Andr\'es, except for the $^1B_2(\pi\ra\pis)$ state
that we locate significantly higher in energy and the three Rydberg states that our CC calculations predict at significantly lower energies. The present {\NEV} results are globally in better agreement with the CC values,
though non-negligible deviations pertain. Even if comparisons with experiment should be made very cautiously, we note that, for the Rydberg states, the present CC data are clearly more coherent with the electron impact measurements\cite{Har74} than the original {\CASPT} values. For cyclopropenethione, we typically obtain transition energies in agreement or larger than those obtained with {\CASPT}, \cite{Ser02} though there is no obvious relationship between the
valence/Rydberg nature of the ES and the relative {\CASPT} error. The average absolute deviation between our {\NEV} and {\CCT} results is $0.08$ eV only. Finally, in the case of methylenecyclopropene, our values logically agree
very well with the recent estimates of Schwabe and Goerigk, \cite{Sch17} obtained at the {\CCT}/{\AVTZ} level of theory on a different geometry.
As anticipated, the available {\CASPT} values \cite{Mer96,Roo96} appear too low as compared to the
present {\NEV} and {\CCSDT} values. For this compound, the available experimental data are based on the wavelength of maximal absorption determined in condensed phase. \cite{Sta84} Hence, only a qualitative match is reached between theory and experiment.
Let us now turn our attention to the excited states of three pseudo-linear $\pi$-conjugated systems that have been the subject to several investigations in the past, namely, acrolein, \cite{Aqu03,Sah06,Car10b,Lea12,Gua13,Mai14,Aza17b,Sch17,Bat17}
butadiene, \cite{Dal04,Sah06,Sch08,Sil10c,Li11,Wat12,Dad12,Lea12,Ise12,Ise13,Sch17,Shu17,Sok17,Chi18,Cop18,Tra19,Loo19c} and glyoxal. \cite{Sta97b,Koh03,Hat05c,Sah06,Lea12,Poo14,Sch17,Aza17b,Loo18b}
Amongst these works, it is worth highlighting the detailed theoretical investigation of Saha, Ehara, and Nakatsuji, who reported a huge
number of ES for these three systems using a coherent theoretical protocol based on the symmetry-adapted-cluster configuration interaction (SAC-CI) method. \cite{Sah06}
In the following, these three molecules are considered in their most stable \emph{trans} conformation.
Our results are listed in Tables \ref{Table-3} and S3.
$^e${Electron impact experiment from Refs.~\citenum{Fli78} and \citenum{Doe81} for the singlet states and from Ref.~\citenum{Mos73} for the two lowest triplet transitions;
$^h${Electron impact experiment from Ref.~\citenum{Ver80} except for the second $^1B_g$ ES for which the value is from another work; \cite{Rob85b} note that
for the lowest $^1B_g$ ($^1B_u$) ES, a range of $4.2$--$4.5$ ($7.4$--$7.9$) eV is given in Ref.~\citenum{Ver80}. }
Acrolein, due to its lower symmetry and high density of ES with mixed characters, is challenging from a theoretical point of view, and {\CCSDTQ} calculations were technically impossible despite all our efforts. For the lowest $n \ra\pis$
transitions of both spin symmetry, the {\FCI} estimates come with a tiny error bar, and it is obvious that the CC excitation energies are slightly too low, especially with {\CCSDT}. Nevertheless, at the exception of the second singlet
and triplet $A''$ ES, the {\CCT} and {\CCSDT} transition energies are within $\pm0.03$ eV of each other. These $A''$ ES are also the only two transitions for which the discrepancies between {\CCT} and {\NEV} exceed $0.20$ eV.
\titou{This hints at a good accuracy for all other transitions.}
This statement is supported by the fact that the present CC values are nearly systematically bracketed by previous {\CASPT} (lower bound)\cite{Aqu03} and SAC-CI (upper bound)\cite{Sah06} results,
consistently with the typical error sign of these two models. For the two lowest triplet states, the present {\CCT}/{\AVTZ} values are also within $\pm0.05$ eV of recent MRCI estimates ($3.50$ and $3.89$ eV). \cite{Mai14} As
can be seen in Table S3, the {\AVTZ} basis set delivers excitation energies very close to the CBS limit: the largest variation when upgrading from {\AVTZ} to {\AVQZ} ($+0.04$ eV) is obtained for the second $^1A'$ Rydberg ES. As experimental data
are limited to measured UV spectra, \cite{Wal45,Bec70} one has to be ultra cautious in establishing TBE for acrolein (\emph{vide infra}).
The nature and relative energies of the lowest bright $B_u$ and dark $A_g$ ES of butadiene have bamboozled theoretical chemists for many years. It is beyond the scope of the present study to provide an exhaustive
list of previous calculations and experimental measurements for these two hallmark ES, and we refer the readers to Refs.~\citenum{Wat12} and \citenum{Shu17} for a general and broader overview and the corresponding references. For the $B_u$ transition,
we believe that the most solid TBE is the $6.21$ eV value obtained by Watson and Chan using a computational strategy similar to ours. \cite{Wat12} Our {\CCSDT}/{\AVTZ} value of $6.24$ eV is obviously compatible
with their reference value, and our TBE value is actually $6.21$ eV as well (\emph{vide infra}). For the $A_g$ state, we believe that our previous basis set corrected {\FCI} estimate of $6.50$ eV \cite{Loo19c} remains
the most accurate available to date. These two values are slightly lower than the semi-stochastic heath-bath CI data obtained by Chien \emph{et al.} with a double-$\zeta$ basis and a slightly different geometry: $6.45$ and $6.58$ eV for $B_u$ and $A_g$,
respectively. \cite{Chi18} For these two thoroughly studied ES, one can of course find many other estimates, \eg, at the SAC-CI, \cite{Sah06}{\CCT}, \cite{Sil10c,Sch17}{\CASPT}, \cite{Sil10c} and {\NEV}\cite{Sok17} levels.
Globally, for butadiene, we find an excellent coherence between the {\CCT}, {\CCSDT}, and {\CCSDTQ} estimates, that all fall in a $\pm0.02$ eV window. Unsurprisingly, this does not apply to the
already mentioned $^1A_g$ ES that is $0.2$ and $0.1$ eV too high with the two former CC methods, \titou{a direct consequence of the large electronic reorganization taking place during this transition.} For all the other butadiene ES listed in
Table \ref{Table-3}, both {\CCT} and {\CCSDT} can be trusted. We also note that the {\NEV} estimates are within $0.1$--$0.2$ eV of the CC values, except for the lowest $B_u$ ES for which the associated excitation energy is highly dependent on the selected
active space (see the SI). Finally, as can be seen in Table S3, {\AVTZ} produces near-CBS excitation energies for most ES.
However, a significant basis set effect exists for the Rydberg $^1B_u (\pi\ra3p)$ ES with an energy lowering as large as $-0.12$ eV
when going from {\AVTZ} to {\AVQZ}. For the records, we note that the available electron impact data \cite{Mos73,Fli78,Doe81} provide the very same ES ordering as our calculations.
Globally, the conclusions obtained for acrolein and butadiene pertain for glyoxal, \ie, highly consistent CC estimates, reasonable agreement between {\NEV} and {\CCT}, and limited basis set effects beyond {\AVTZ}, except for the $^1B_u (n \ra3p)$ Rydberg state (see Tables \ref{Table-3} and S3). This Rydberg state also exhibits an unexpectedly large deviation of $0.04$ eV between {\CCT} and {\CCSDTQ}. More interestingly,
glyoxal presents a genuine low-lying double ES of $^1A_g$ symmetry. The corresponding $(n,n)\ra(\pis,\pis)$ transition is totally unseen by approaches that cannot model double excitations, \eg, TD-DFT, {\CCSD},
or {\AD}. Compared to the {\FCI} values, the {\CCT} and {\CCSDT} estimates associated with this transition are too large by $\sim1.0$ and $\sim0.5$ eV, respectively, whereas both the {\CCSDTQ} and {\NEV} approaches are much closer, as already
mentioned in our previous work. \cite{Loo19c} For the other transitions, the present {\CCT} estimates are logically coherent with the values of Ref.~\citenum{Sch17} obtained with the same approach on a different
geometry, and remain slightly lower than the SAC-CI estimates of Ref.~\citenum{Sah06}. Once more, the experimental data \cite{Ver80,Rob85b} are unhelpful in view of the targeted accuracy.
Let us now turn towards six other four-atom compounds. There are several earlier studies reporting estimates of the vertical transition energies for both acetone \cite{Gwa95,Mer96b,Roo96,Wib98,Toz99b,Wib02,Sch08,Sil10c,Car10,Pas12,Ise12,Gua13,Sch17}
and isobutene. \cite{Wib02,Car10,Ise12} To the best of our knowledge, for the four other compounds, the previous computational efforts were mainly focussed on the 0-0 energies of the lowest-lying states. \cite{Koh03,Hat05c,Sen11b,Loo18b,Loo19a}
There are also very few experimental data available for these six derivatives. \cite{Bir73,Jud83,Bra74,Sta75,Joh79,Jud83,Jud84c,Rob85,Pal87,Kar91b,Xin93}
Our main results are reported in Tables \ref{Table-4} and S4.
$^c${Two lowest singlet states: various experiments summarized in Ref.~\citenum{Rob85}; three next singlet states: REMPI experiments from Ref.~\citenum{Xin93}; lowest triplet: trapped electron measurements from Ref.~\citenum{Sta75};}
$^d${0-0 energy reported in Ref.~\citenum{Kar91b};}
For acetone, one should clearly distinguish the valence ES, for which both methodological and basis set effects are small, and the Rydberg transitions that are both very
basis set sensitiv, and upshifted by ca.~$0.04$ eV with {\CCSDTQ} as compared to {\CCT} and {\CCSDT}. For this compound, the 1996 {\CASPT} transition energies of Merch\'an and coworkers listed on the
right panel of Table \ref{Table-4} are clearly too low, especially for the three valence ES. \cite{Mer96b} As expected, this error can be partially ascribed to the computational set-up, as the Urban group obtained {\CASPT}
excitation energies of $4.40$, $4.09$ and $6.22$ eV for the $^1A_2$, $^3A_2$, and $^3A_1$ ES, \cite{Pas12} in much better agreement with ours. Their estimates of the three $n \ra3p$ transitions, $7.52$, $7.57$, and $7.53$ eV
for the $^1A_2$, $^1A_1$, and $^1B_2$ ES, also systematically fall within $0.10$ eV of our current CC values, whereas for these three ES, the current {\NEV} values are clearly too large.
In contrast to acetone, both valence and Rydberg ES of thioacetone are rather insensitive to the excitation order of the CC expansion, as illustrated by the maximal discrepancies of $\pm0.02$ eV between the {\CCT}/{\Pop} and {\CCSDTQ}/{\Pop} results.
While the lowest $n \ra\pis$ transition of both spin symmetries are rather basis set insensitive, all the other states need quite large one-electron bases to be correctly described (Table S4).
As expected, our theoretical vertical transition energies show the same ranking but are systematically larger than the available experimental 0-0 energies.
For the isoelectronic isobutene molecule, we have considered two singlet Rydberg and one triplet valence ES. For these three cases, we note, for each basis, a very nice agreement between {\CCT} and {\CCSDT}, the
CC results being also very close to the {\FCI} estimates obtained with the Pople basis set. The similarity with the {\CCSD} results of Caricato and coworkers \cite{Car10} is also very satisfying.
For the three remaining compounds, namely, cyanoformaldehyde, propynal, and thiopropynal, we report low-lying valence transitions with a definite single excitation character. The basis set
effects are clearly under control (they are only significant for the second $^1A''$ ES of cyanoformaldehyde) and we could not detect any variation larger than $0.03$ eV between the {\CCT} and {\CCSDT} values for
As we have seen for the 15 four-atom molecules considered here, we found extremely consistent transition energies between CC and {\FCI} estimates in the vast majority of the cases.
Importantly, we confirm our previous conclusions obtained on smaller compounds: \cite{Loo18a} i) {\CCSDTQ} values systematically fall within (or are extremely close to) the {\FCI} error bar,
ii) both {\CCT} and {\CCSDT} are also highly trustable when the considered ES does not exhibit a strong double excitation character. Indeed, considering the 54 ``single'' excitations
for which {\CCSDTQ} estimates could be obtained (only excluding the lowest $^1A_g$ ES of butadiene and glyoxal), we determined negligible MSE of $0.00$ eV, tiny
MAE ($0.01$ and $0.02$ eV), and small maximal deviations ($0.05$ and $0.04$ eV) for {\CCT} and {\CCSDT}, respectively. This clearly indicates that these two approaches provide chemically-accurate
estimates (errors below $1$ kcal.mol$^{-1}$ or $0.043$ eV) for most electronic transitions. Interestingly, some of us have shown that {\CCT} also provides chemically-accurate 0-0 energies as compared
to experimental values for most valence transitions. \cite{Loo18b,Loo19a,Sue19} When comparing the {\NEV} and {\CCT} ({\CCSDT}) results obtained with {\AVTZ} for the 91 (65) ES for which comparisons are possible (again excluding only the lowest $^1A_g$ states of butadiene and glyoxal),
This seems to indicate that {\NEV}, as applied here, has a slight tendency to overestimate the transition energies. This contrasts with {\CASPT} that is known to generally underestimate transition energies, as further illustrated and discussed above and below.
We now consider five-membered rings, and, in particular, five common derivatives that have been considered in several previous theoretical studies (\emph{vide infra}): cyclopentadiene, furan, imidazole, pyrrole, and thiophene.
As the most advanced levels of theory employed in the previous Section, namely {\CCSDTQ} and {\FCI}, become beyond reach for these compounds (except in very rare occasions), one has to rely on the nature of the ES and the consistency between results to deduce TBE.
For furan, \textit{ab initio} calculations have been performed with almost every single wavefunction methods.\cite{Ser93b,Nak96,Tro97b,Chr98b,Chr98c,Wan00,Gro03,Pas06b,Sch08,She09b,Li10c,Sau11,Sil10b,Sil10c,Hol15,Sch17}
However, the present work is, to the best of our knowledge, the first to disclose {\CCSDT} values as well as {\CCT} energies obtained with a quadruple-$\zeta$ basis set. Our results for ten low-lying ES states are listed in Tables \ref{Table-5} and S5.
All singlet (triplet) transitions are characterized by $\Td$ values in the $92$--$94\%$ ($97$--$99\%$) range. Consistently, the maximal discrepancy between {\CCT} and {\CCSDT} is small ($0.04$ eV). In addition, there is a
be significantly higher than in most previous works, even after accounting for the quite large basis set effects ($-0.10$ eV between the {\AVTZ} and {\AVQZ} estimates, see Table S5). We believe that our estimate is the most accurate to date for this particularly tricky ES.
Interestingly, the recent {\AT} values of Ref.~\citenum{Hol15} are consistently smaller by ca.~$-0.2$ eV as compared to {\CCSDT} (see Table \ref{Table-6}), consistently with the error sign we observed in smaller compounds for {\AT}. \cite{Loo18a}
Again, we note that the experimental data \cite{Vee76b,Fli76,Rob85b} provide the same state ordering as our calculations.
$^g${Electron impact from Ref.~\citenum{Vee76b}: for the $^1A_1$ state two values ($6.44$ and $6.61$ eV) are reported, whereas for the two lowest triplet states, Two values ($3.99$ eV and $5.22$ eV) can be found in Ref.~\citenum{Fli76};}
Like furan, pyrrole has been extensively investigated in the literature using a large panel of theoretical methods. \cite{Ser93b,Nak96,Tro97,Pal98,Chr99,Wan00,Roo02,Pal03b,Pas06c,Sch08,She09b,Li10c,Sau11,Sil10b,Sil10c,Nev14,Sch17,Hei19}
Here, we report six low-lying singlet and four triplet ES in Tables \ref{Table-5} and S5. All these transitions have very large $\Td$ values except for the totally symmetric $\pi\ra\pis$ excitation ($\Td=86\%$). For all the states, we found
highly consistent {\CCT} and {\CCSDT} results, often significantly larger than older multi-reference estimates, \cite{Ser93b,Roo02,Li10c} but in nice agreement with the very recent XMS-{\CASPT} results of the Gonzalez
group, \cite{Hei19} at the exception of the $^1A_2(\pi\ra3p)$ transition. The match obtained with the twenty years old extrapolated CC values of Christiansen and coworkers \cite{Chr99} is quite remarkable. The only exceptions are the two
$B_2$ transitions that were reported as significantly mixed in this venerable work. For the lowest singlet ES, the {\FCI}/{\Pop} value is $5.24\pm0.02$ eV confirming the performances of both {\CCT} and {\CCSDT} for this transition.
As can be seen in Table S5, {\AVTZ} yields basis-set converged transition energies, except, like in furan, for the Rydberg $^1B_2(\pi\ra3p)$ transition that is significantly redshifted ($-0.09$ eV) when pushing to the
quadruple-$\zeta$ basis set. As mentioned in Thiel's work, \cite{Sch08} the experimental spectra of pyrrole are quite ``disparate'', and the rare available experiments \cite{Hor67,Bav76,Fli76b,Vee76b,Pal98,Pal03b} can only be considered as general guidelines.
$^g${Energy loss from Ref.~\citenum{McD85} for the two valence states; two-photon resonant experiment from Ref.~\citenum{Sab92} for the $^1A_2$ Rydberg ES;}
Although a diverse array of wavefunction studies has been performed on cyclopentadiene (including {\CASPT}, \cite{Ser93b,Sch08,Sil10c} CC, \cite{Sch08,Sil10b,Sch17} SAC-CI \cite{Wan00b} and various multi-reference approaches \cite{Nak96,She09b}), this compound has received less attention that other members of the five-membered ring family, namely furan and pyrrole (\emph{vide infra}). This is probably due to the presence of the
methylene group that renders computations significantly more expensive. Most transitions listed in Tables \ref{Table-6} and S6 are characterized by $\Td$ exceeding $93\%$, the only exception being the $^1A_1(\pi\ra\pis)$
excitation that has a similar nature as the lowest $A_g$ state of butadiene ($\Td=79\%$). Consistently, the {\CCT} and {\CCSDT} results are nearly identical for all ES except for the $^1A_1$ ES. By comparing the results
obtained for this $A_1(\pi\ra\pis)$ transition to its butadiene counterpart, one can infer that the {\CCSDT} estimate is probably too large by roughly $0.04$--$0.08$ eV, and that the {\NEV} value is unlikely to be accurate enough. This statement is also in line with the results of Ref.~\citenum{Loo19c}. For the two $B_2(\pi\ra\pis)$ transitions, we could obtain a {\FCI}/{\Pop} estimates of $5.78\pm0.02$ eV (singlet)
and $3.33\pm0.05$ eV (triplet), the {\CCT} and {\CCSDT} transition energies falling inside these energetic windows in both cases. As one can see in Tables \ref{Table-6} and S6, the basis set effects are rather moderate
for the electronic transitions of cyclopentadiene, with no variation larger than $0.10$ eV ($0.02$ eV) between {\AVDZ} and {\AVTZ} ({\AVTZ} and {\AVQZ}). When comparing to literature data, our values are unsurprisingly consistent with the {\CCT} values of
Schwabe and Goerigk, \cite{Sch17} and tend to be significantly larger than earlier {\CASPT}\cite{Ser93b,Sil10c} and MR-MP \cite{Nak96} estimates. As expected, a few gas-phase experiments are available as well for this
Due to its lower symmetry, imidazole has been less investigated, the most advanced studies available probably remain the {\CASPT} work of Serrano-Andr\`es and coworkers from 1996, \cite{Ser96b} and the
basis-set extrapolated {\CCT} results of Silva-Junior \emph{et al.} for the valence transitions. \cite{Sil10c} The experimental data in gas-phase are also limited.\cite{Dev06}
Our results are displayed in Tables \ref{Table-6} and S6. The {\CCT} and {\CCSDT} values are quite consistent despite the fact that the $\Td$ values of the two singlet $A'$ states
are slightly smaller than $90\%$. For the eight transitions considered here, the basis set effects are moderate and {\AVTZ} yield results within $0.03$ eV of their {\AVQZ} counterparts (Table S6 in the SI). \hl{NEV ?}
Finally, the ES of thiophene, which is one of the most important building block in organic electronic devices, were the subject of a previous theoretical investigations, \cite{Ser93c,Pal99,Wan01,Kle02,Pas07,Hol14} that unveiled a series of
transitions that were not yet characterized in the available measurements. \cite{Dil72,Fli76,Fli76b,Var82,Hab03,Pal99,Hol14} To the best of our knowledge, the present work is the first to report CC calculations obtained with (iterative)
triples and therefore constitutes the most accurate estimates to date. Indeed, all the transitions listed in Tables \ref{Table-6} and S6 are characterized by a largely dominant single excitation character, with $\Td$ above
$90\%$ except for the two $^1A_1$ transitions for which $\Td=88\%$ and $87\%$, respectively. The agreement between {\CCT} and {\CCSDT} remains nevertheless excellent for the lowest totally symmetric transition. Thiophene is
also one of these compounds for which the unambiguous characterization of the nature of the ES is difficult, with, \eg, a strong mixing between the second and third singlet ES of $B_2$ symmetry. This makes the assignment of the valence
($\pi\ra\pis$) or Rydberg ($\pi\ra3p$) character of this transition particularly tricky at the {\CCT} level. We note that contradictory assignments can be found in the literature. \cite{Ser93c,Wan01,Pas07} As for the
previously discussed isostructural systems, the only ES that undergoes significant basis set effects beyond {\AVTZ} is the Rydberg $^1B_2(\pi\ra3p)$ ($-0.09$ eV when upgrading to {\AVQZ}, see Table S6).
The data of Table \ref{Table-6} are globally in good agreement with the previously reported values with discrepancies that are significant only for the three highest-lying singlet states.
Let us now turn towards seven six-membered rings which plays a key role in chemistry: benzene, pyrazine, pyridazine, pyridine, pyrimidine, tetrazine, and triazine. To the best of our knowledge, the present work
is the first to propose {\CCSDT} reference energies as well as {\CCT} values obtained with {\AVQZ} for all these compounds. Of course, these systems have been investigated before, and beyond Thiel's
benchmarks, \cite{Sch08,Sil10b,Sil10c} it is worth pointing out the investigations of Del Bene and coworkers, \cite{Del97b} performed with a CC approach including perturbative corrections for the triples.
Following a theoretically consistent protocol, Nooijen \cite{Noo99} also performed {\STEOM} calculations to study the ES of all these derivatives. However, these two works only considered singlet ES.
These three highly-symmetric systems allow to directly perform {\CCSDT}/{\AVTZ} calculations for singlet states without the need of basis set extrapolations. Benzene was studied many times
before, \cite{Sob93,Lor95b,Chr96c,Pac96,Del97b,Noo99,Hal02,Li07b,Sch08,Dev08,Sil10b,Sil10c,Li11,Lea12,Kan14,Sch17,Dut18,Sha19,Loo19c} and we report in Tables \ref{Table-7} and S7 estimates obtained for
five singlet and three triplet ES, all characterized by $\Td$ exceeding $90\%$ except for the lowest singlet ($86\%$). As one can see, the two CC approaches are again yielding very consistent transitions energies
and {\AVTZ} is essentially providing basis set converged transition energies. The present estimates are also very consistent with earlier {\CCT}\cite{Chr96c} and very recent RASPT2 \cite{Sha19} values.
For both spin symmetry, our values are slightly larger than the available electron impact/multi-photon measurements. \cite{Doe69,Nak80,Joh76,Joh83,Hir91}
$^f${Jet-cooled experiment from Ref.~\citenum{Hir91} for the two lowest states, multi-photon experiments from Refs.~\citenum{Joh76} and \citenum{Joh83} for the Rydberg states.}
$^l${From Ref.~\citenum{Pal97}, the singlets are from EEL, except for the $4.97$ and $5.92$ eV values that are from VUV; the triplets are from EEL, and other triplet peaks are mentioned at $4.21$, $4.6$, and $5.2$ eV but not identified;}
Numerous previous theoretical estimates are available for both pyrazine, \cite{Ful92,Del97b,Web99,Noo99,Li07b,Sch08,Sau09,Sil10c,Woy10,Car10,Lea12,Kan14,Sch17,Dut18} and tetrazine, \cite{Sta96,Del97b,Rub99,Noo99,Ada00,Noo00,Ang09,Sch08,Sau09,Sil10b,Sil10c,Car10,Lea12,Kan14,Sch17,Dut18,Pas18b}
for which the $D_{2h}$ symmetry helps distinguishing the different ES. Our results are collected in Tables \ref{Table-8} and S8. In pyrazine, all transitions are characterized by $\Td > 85\%$ (except for the $^1B_{1g}(n \ra\pis)$ transition).
The excitation energies are basically unchanged going from {\CCT} to {\CCSDT} with one exception (the highest-lying singlet state). Going from triple- to quadruple-$\zeta$ basis, the variations do not exceed $0.04$ eV, even for the four Rydberg states, indicating that one can probably be highly confident in the present estimates.
Again, the previous {\CASPT} estimates \cite{Ful92,Web99,Sch08} appear to be globally too low, while the (unconventional) CASPT3 results \cite{Woy10} seem too high.
The same holds for the SAC-CI results. \cite{Li07b} In fact, our best match is obtained with Nooijen's {\STEOM} values (except for the highest ES), \cite{Noo99} and recent {\CCT} estimates. \cite{Sch17}
In all other cases, there is a global consistency between our CC values. Moreover, the basis set effects are very small beyond {\AVTZ} with maximal variations of $0.02$ eV only (Table S8). The present values are
almost systematically larger than previous {\CASPT}, \cite{Rub99}{\STEOM}, \cite{Noo00} and GVVPT2 \cite{Dev08} estimates. They are also globally in agreement with Thiel's {\CCT}/{\AVTZ} values, \cite{Sil10c} although we note variations
of approximately $0.20$ eV for specific excitations like the $B_{2g}$ transitions.
This feature is likely due to the use of distinct geometries in the two studies. The experimental EEL values from Palmer's work, \cite{Pal97} show a reasonable agreement with our estimates.
Those four azabenzenes with $C_{2v}$ and $D_{3h}$ spatial symmetry are also popular molecules in terms of ES calculations. \cite{Pal91,Ful92,Wal92,Lor95,Del97b,Noo97,Noo99,Fis00,Cai00b,Wan01b,Sch08,Sil10b,Sil10c,Car10,Lea12,Kan14,Sch17,Dut18}
Our results for pyridazine and pyridine are gathered in Tables \ref{Table-9} and S9. For the former compound, the available wavefunction results \cite{Pal91,Ful92,Del97b,Noo99,Fis00,Sch08,Sil10b,Sil10c,Kan14,Sch17,Dut18}
only considered singlet transitions, at the exception of a rather old MRCI, \cite{Pal91} and {\CASPT} investigations. \cite{Fis00} Again, the $\Td$ values are larger than $85\%$ ($95\%$) for the singlet (triplet) transitions,
and the only state for which there is a variation larger than $0.03$ eV between the {\CCT}/{\AVDZ} and {\CCSDT}/{\AVDZ} energies is the $^3B_2(\pi\ra\pis)$ transition. \hl{BASIS SETS}
For the singlet valence ES, we find again a rather good agreement with previous {\STEOM}\cite{Noo99} and CC \cite{Del97b,Sil10c} estimates, which are again significantly higher than the {\CASPT} estimates reported in Refs.~\citenum{Ful92} and \citenum{Sil10c}. For the triplets, the
present data represents the best results published to date. Interestingly, beyond the usually cited experiments, \cite{Inn88,Pal91} there is a very recent experimental EEL analysis for pyridazine, \cite{Lin19} that locates
Pyridine has been more scrutinized with various wavefunction approaches. \cite{Ful92,Lor95,Del97b,Noo97,Noo99,Cai00b,Wan01b,Sch08,Sil10b,Sil10c,Car10,Kan14,Sch17,Dut18} Beside, we have found two detailed EEL experiments. \cite{Wal90,Lin16}
For pyridine, the general trends described above for pyridazine pertain: i) large $\Td$ values and consistency between our various CC estimates for all transitions listed in Table \ref{Table-9}; ii) \hl{basis}; iii) qualitative agreement with past
CC results; and iv) same ES ranking as in the most recent measurements. \cite{Lin16} Beyond these aspects, it is worth mentioning that the second $^1B_2(\pi\ra\pis)$ ES is strongly mixed with a nearby
Rydberg transition that is separated by only $0.03$ eV at the {\CCT}/{AVTZ} level. This obviously makes the analysis particularly challenging for that specific transition. \hl{Keep or not A1 transitiion}
The results obtained for both pyrimidine and triazine are listed in Tables \ref{Table-10} and S10. For the former derivative, previous theoretical \cite{Ful92,Del97b,Ser97b,Noo99,Ohr01,Fis03b,Li07b,Sch08,Sil10b,Sil10c,Car10,Kan14,Sch17,Dut18}
and experimental \cite{Bol84,Pal90,Lin15} studies are rather extensive, as it can be viewed as the smallest model of DNA bases. For triazine, which does not have an abelian point group, theoretical studies are scarcer,
\cite{Ful92,Wal92,Pal95,Del97b,Noo99,Oli05,Sch08,Sil10b,Sil10c,Kan14,Sch17,Dut18} especially for the triplets. \cite{Wal92,Pal95,Oli05} Experimental data are also limited. \cite{Bol84,Wal92}
As in pyridazine and pyridine, all the ES listed in Table \ref{Table-10} show $\Td$ values larger than $85\%$ for singlets and $95\%$ for triplets, so that {\CCT} and {\CCSDT} are highly coherent, except maybe for the $^3A_1(\pi\ra\pis)$ transitions in pyrimidine.
The basis set effects are also small, with no variation larger than $0.10$ ($0.03$) eV between double- and triple-$\zeta$ (triple- and quadruple-$\zeta$) and only
slightly larger variations for the two Rydberg transitions. For both compounds, the current values are almost systematically larger than previously published data. For the triplets of triazine, the three lowest ES previously estimated by {\CASPT} are too low by roughly half an eV.
Table \ref{Table-tbe} reports our two sets of TBE: a set obtained with the {\AVTZ} basis set and one set with an additional correction for the one-electron basis set incompleteness error. The details of our protocol employed to generate these TBE are also provided in Table \ref{Table-tbe}.
For all states with a dominant single-excitation character (that is when $\Td > 80\%$), we rely on CC results using an incremental strategy to generate these TBE. For ES with $\Td$ between $70\%$ and $80\%$,
our previous works indicated that {\CCSDT} tends to overshoot the transition energies by roughly $0.05$--$0.10$ eV, but that {\NEV} errors tend to be, on average, slightly larger. \cite{Loo19c} Therefore, if {\CCSDTQ} or {\FCI}
results are not available, it is extremely difficult to make the final call. For the other transitions, we relied either on the current or previous FCI data or the {\NEV} values as reference. The italicized transition energies in Table \ref{Table-tbe} are believed to be (relatively) less accurate. This is the case when: i) {\NEV} results have to be selected; ii) the CC calculations yield quite large changes in excitation energies while incrementing the excitation order by one unit despite large $\Td$;
and iii) there is a very strong ES mixing making hard to follow a specific transition from one method (or one basis) to another.
To determine the basis set corrections beyond augmented triple-$\zeta$,
we use the {\CCT}/{\AVQZ} or {\CCT}/{\AVPZ} results. For several compounds, we also provide in the SI, {\CCT}/d-{\AVQZ} transition energies (\ie, with an additional set of diffuse functions). However, we do not consider such values as reference because
the addition of a second set of diffuse functions only significantly modifies the transition energies for strongly-mixing ES.
We also stick to the frozen-core approximation for two reasons: i) the effect of correlating the core electrons is generally negligible (typically $\pm0.02$ eV) for the compounds under study (see the SI for examples); and ii) it would be,
in principle, necessary to add core polarization functions in such cases.
Table \ref{Table-tbe} encompasses 238 ES, each of them obtained, at least, at the {\CCSDT} level. This set can be decomposed as follows: 144 singlet and 94 triplet transitions, or 174 valence (99 $\pi\ra\pis$, 71
$n \ra\pis$ and 4 double excitations) and 64 Rydberg transitions. Amongst these transition energies, fifteen can be considered as ``unsafe'' and are reported in italics accordingly. This definitely corresponds to a very significant extension of our previous
ES data sets (see Introduction).
Taken all together, they offer a consistent, diverse and accurate ensemble of transition energies for approximately 350 electronic transitions of various natures in small and medium-sized organic molecules.
Table \ref{Table-tbe} also reports 90 oscillator strengths, $f$, which makes it, to the best of our knowledge, the largest set of {\CCT}/{\AVTZ} oscillator strengths reported to date, the previous effort being mostly performed
at the {\CCT}/TZVP level for Thiel's set. \cite{Kan14} It should also be pointed out that all these data are obtained on {\CCT}/{\AVTZ} geometries, consistently with our previous works. \cite{Loo18a,Loo19c}
\caption{\small TBE values (in eV) for all considered states alongside their corresponding oscillator strength, $f$, and percentage of single excitations, $\Td$, obtained at the \CCT/{\AVTZ} level.
The composite protocol to generate these TBE is also reported (see footnotes). In the right-most column, we list the TBE values obtained by including an additional correction (obtained at the {\CCT} level) for basis set incompleteness error.
Values displayed in italics are likely to be relatively less accurate. All values are obtained in the FC approximation.}\label{Table-tbe}\\
Having at hand such a large set of accurate transition energies, it seems natural to pursue previous benchmarking efforts. More specifically, we assess here the performances of eight popular wavefunction approaches, namely, CIS(D), {\AD},
{\CCD}, {\STEOM}, {\CCSD}, CCSDR(3), CCSDT-3 and {\CCT}. The complete list of results can be found in Table \hl{SXXX} of the SI. Because all these approaches are single-reference methods, we have removed from the
reference set the ``unsafe'' transition energies (in italics in Table \ref{Table-tbe}), as well as the four transitions with a dominant double excitation character (with $\Td < 50\%$ as listed in Table \ref{Table-tbe}).
A comprehensive list of results are collected in Table \ref{Table-bench} which, more specifically, gathers the MSE, MAE, RMSE, SDE, \MaxP, and \MaxN.
Figure \ref{Fig-1} shows histograms of the error distributions for these eight methods. Before discussing these, let us stress two obvious biases of this benchmark set: i) it encompasses only conjugated organic molecules
containing 4 to 6 non-hydrogen atoms; and ii) we mainly used {\CCSDTQ} (4 atoms) or {\CCSDT} (5--6 atoms) reference values. As discussed in Section \ref{sec-ic} and in our previous work, \cite{Loo18a} the MAE obtained
with these two methods are of the order of $0.01$ and $0.03$ eV, respectively. This means that any statistical quantity smaller than $\sim0.02$--$0.03$ eV is very likely to be irrelevant.
\caption{Mean signed error (MSE), mean absolute error (MAE), root-mean square error (RMSE), standard deviation of the errors (SDE), as well as the positive [\MaxP] and negative [\MaxN] maximal errors with respect to the TBE.
All these statistical quantities are reported in eV and have been obtained with the {\AVTZ} basis set.
Let us analyze the global performances of all methods, starting with the most accurate and computationally demanding models. The relative accuracies of {\CCT} and {\CCSDT}-3 as compared to {\CCSDT} remains an open question in the literature. \cite{Wat13,Dem13}
Indeed, to the best of our knowledge, the only two previous studies discussing this specific aspect are limited to very small compounds. \cite{Kan17,Loo18a} According to the results gathered in Table \ref{Table-bench}, it appears that {\CCT} has a slight edge over {\CCSDT}-3, although
{\CCSDT}-3 is closer to {\CCSDT} in \titou{formal} terms. Indeed, {\CCSDT}-3 seems to provide slightly too large transition energies (MSE of $+0.05$ eV). These conclusions are qualitatively consistent with the analyses performed on smaller derivatives,
\cite{Kan17,Loo18a} but the amplitude of the {\CCSDT}-3 errors is larger in the present set. Although the performances of {\CCT} might be unduly inflated by the use of {\CCSDT} and {\CCSDTQ} reference values, it is also clear that this
method very rarely fails (Figure \ref{Fig-1}). Consequently, {\CCT} transition energies can be viewed as extremely solid references for any transition with a dominant single-excitation character. This conclusion is again consistent with previous
analyses performed on smaller compounds, \cite{Kan17,Loo18a} as well as with recent comparisons between theoretical and experimental 0-0 energies performed by some of us on medium-sized molecules. \cite{Loo18b,Loo19a,Sue19}
To state it more boldly: it appears likely that {\CCT} is even more accurate than previously thought. In addition, thanks to the exhaustive and detailed comparisons made in the present work, we can safely conclude that {\CCT} regularly outperforms {\CASPT} and {\NEV}, even when
these methods are combined with large active spaces. This statement seems to hold as long as the considered ES does not show a strong multiple excitation character, that is, when $\Td < 70\%$.
The perturbative inclusion of triples as in CCSDR(3) yields a very small MAE ($0.05$ eV) for a much lighter computational cost as compared to {\CCSDT}. Nevertheless, as with {\CCSDT}-3, the CCSDR(3) transition energies have a clear tendency
of being too large, an error sign likely inherited from the parent {\CCSD} model. This $0.05$ eV MAE for CCSDR(3) is rather similar to the one obtained for smaller compounds when comparing to {\FCI} ($0.04$ eV), \cite{Loo18a} and is also inline with the
2009 benchmark study of Sauer et al. \cite{Sau09}
{\CCSD} provides an interesting case. The calculated MSE ($+0.11$ eV), indicating an overestimation of the transition energies, fits well with several previous recent reports.
It is, nonetheless, larger than the one determined for smaller molecules ($+0.05$ eV), \cite{Loo18a} hinting that the performances of {\CCSD} deteriorates for larger compounds.
Moreover, {\CCSD} MAE of $0.13$ eV is much smaller than the one reported by Thiel in its original work ($0.49$ eV) \cite{Sch08} but of the same order of magnitude as in the more recent study of K\'ann\'ar and Szalay performed
on Thiel's set ($0.18$ eV for transitions with $\Td > 90\%$). \cite{Kan14} Retrospectively, it is pretty obvious that Thiel's much larger value is very likely due to the {\CASPT} reference values. \cite{Sch08}
Indeed, as we have shown several times in the present study, {\CASPT} transitions energies tend to be significantly too low, therefore exacerbating the usual {\CCSD} overestimation.
With a single detailed benchmark study to date, \cite{Dut18} the {\STEOM} approach has received relatively little attention and its overall accuracy still needs to be corroborated.
It is noteworthy that {\STEOM} provides a smaller MSE than {\CCSD} and comparable MAE and RMSE. The spread of the error is however slightly larger as evidenced by Figure \ref{Fig-1} and the SDE values
reported in Table \ref{Table-bench}. These trends are the same as for smaller compounds. \cite{Loo18a} For Thiel's set, Dutta and coworkers also reported rather good performance
for {\STEOM} with respect to the {\CCT}/TZVP reference data, though a slightly negative MSE is obtained in their case. \cite{Dut18} This could well be due to the different basis set considered by these two studies. It should be nevertheless stressed that we, here, only consider only ``clean'' {\STEOM} results
(see Computational Details), therefore removing several difficult cases that are included in the {\CCSD} statistics, \eg, the $A_g$ excitation in butadiene, which can slightly bias the relative performance of {\STEOM} and {\CCSD}.
Finally, for the three second-order methods, namely CIS(D), {\AD}, and {\CCD}, that are often used as reference to benchmark TD-DFT for ``real-life'' applications, the performance of the former method clearly deteriorates compared to the two latter which exhibit very
similar statistical behaviors. These trends were also reported in previous works. \cite{Hat05c,Jac18a,Sch08,Sil10c,Win13,Har14,Jac15b,Kan17,Loo18a} Interestingly, the {\CCD} MAE obtained here ($0.15$ eV) is significantly
smaller than the one we found for smaller compounds ($0.22$ eV). \cite{Loo18a} Therefore, in contrast to {\CCSD}, {\CCD} performance seems to improve with molecular size. As above, Thiel's original MAE for {\CCD} ($0.29$ eV) was likely too large due
to the selection of {\CASPT} reference values. \cite{Sch08} As already noticed by Szalay's group, \cite{Kan14,Kan17} although the MSE of {\CCD} is smaller than the one of {\CCSD}, the standard deviation is significantly larger
In Table \ref{Table-bench2}, we report a MAE decomposition for different subsets of ES. Note that, due to implementational limitations, only singlet ES could be computed with CCSDR(3) and CCSDT-3 which explains the lack of data for triplet ES.
A few interesting conclusions emerge from these results. First, the errors for singlet and triplet transitions are rather similar with all models, except for {\CCSD} that
is significantly more effective for triplets. Dutta and coworkers observed the same trend for Thiel's set with MAE of $0.20$ eV and $0.11$ eV for the singlet and triplet ES, respectively. \cite{Dut18}
Turning to the comparison between valence and Rydberg states, we find that {\CCD} provide a better description of the former, whereas {\CCSD} (and higher-order methods) yields the opposite trend.
In fact, {\CCD} has the clear tendency to overestimate valence ES energies (MSE of $+0.10$ eV), and to underestimate Rydberg ES energies (MSE of $-0.17$ eV).
{\CCSD} is found to be much more consistent with MSE of $0.12$ and $0.09$ eV, respectively (see the SI). This relatively poorer performance of {\CCD} as compared to {\CCSD} for Rydberg ES is again perfectly coherent with other benchmarks, \cite{Kan17,Dut18} although the MAE
for {\CCD} ($0.18$ eV) reported in Table \ref{Table-bench2} remains relatively small as compared to the one given in Ref.~\citenum{Kan17}. We believe that it is likely due to the distinct types of Rydberg states considered in these two studies. Indeed, we consider here (relatively) low-lying
Rydberg transitions in medium-sized molecules, while K\'ann\'ar and Szalay (mostly) investigated higher-lying Rydberg states in smaller compounds. Finally, CIS(D), {\AD}, {\CCD}, and {\STEOM}
better describe $n\ra\pis$ transitions, whereas {\CCSD} seems more suited for $\pi\ra\pis$ transitions.
The variations between the two subsets are probably not significant for the
higher-order approaches. The former observation agrees well with previous results on smaller compounds \cite{Loo18a} and the Thiel set, \cite{Sch08,Kan14} whereas \titou{the latter, less expected observation is likely dependent on the selected ES subset.}\cite{Sch08,Kan17}
We have computed highly-accurate vertical transition energies for a set of 27 medium-sized organic molecules containing from 4 to 6 (non-hydrogen) atoms. To this end, we employed several state-of-the-art theoretical models with increasingly large diffuse basis sets.
listed excited states, the present contribution is the very first to disclose (sometimes basis-set extrapolated) {\CCSDT}/{\AVTZ} and (true) {\CCT}/{\AVQZ} transition energies as well as {\CCT}/{\AVTZ} oscillator strengths
for each dipole-allowed transition. Our set contains a total of 238 transition energies and 90 oscillator strengths, with a reasonably good balance between singlet, triplet, valence,
and Rydberg states. Amongst these 238 transitions, we believe that 224 are ``solid'' TBE, \ie, they are chemically accurate (MAE below $0.043$ eV or $1$ kcal.mol$^{-1}$) for the considered geometry.
It allows us to establish a reasonable error bar for several popular ES models with lower computational cost: CIS(D), {\AD}, {\CCD}, {\STEOM}, {\CCSD},
CCSDR(3), CCSDT-3, and {\CCT}. It turns out that the latter approach is extremely accurate, and, very likely should be considered as more robust and trustworthy than {\CASPT} or {\NEV}, except for ES with a predominant double
excitation character. Other methods including corrections for the triples yield a mean absolute deviation around $0.05$ eV, whereas none of the second-order approach has been found to be chemically accurate with MAE
Paraphrasing Thiel and coworkers, \cite{Sch08} we hope that this new set of vertical transition energies, combined or not with the ones described in our previous works, \cite{Loo18a,Loo19c} will be useful for the community,
will stimulate further developments and analyses in the field, and will provide new grounds for appraising the \emph{pros} and \emph{cons} of ES models already available or currently under development. We can
crystal-ball that the emergence of new {\SCI} algorithms optimized for modern supercomputer architectures will likely lead to the revision of some the present TBE, allowing to climb even higher on the accuracy ladder.
P.F.L.~would like to thank \textit{Centre National de la Recherche Scientifique} for funding.
D.J.~acknowledges the \emph{R\'egion des Pays de la Loire} for financial support. This research used resources of i) the GENCI-TGCC (Grant No.~2018-A0040801738); ii) CCIPL (\emph{Centre de Calcul Intensif des Pays de Loire});
iii) the Troy cluster installed in Nantes; and iv) CALMIP under allocations 2019-18005 (Toulouse).
This work has been supported through the EUR grant NanoX ANR-17-EURE-0009 in the framework of the \textit{``Programme des Investissements d'Avenir''}.