% Include section in journal if known, otherwise delete
\paperfield{Journal Section}
\title{QUESTDB: a database of highly-accurate excitation energies for the electronic structure community}
% List abbreviations here, if any. Please note that it is preferred that abbreviations be defined at the first instance they appear in the text, rather than creating an abbreviations list.
\abbrevs{ABC, a black cat; DEF, doesn't ever fret; GHI, goes home immediately.}
% Include full author names and degrees, when required by the journal.
% Use the \authfn to add symbols for additional footnotes and present addresses, if any. Usually start with 1 for notes about author contributions; then continuing with 2 etc if any author has a different present address.
We describe our efforts of the past few years to create a mega set of more than \alert{470} highly-accurate vertical excitation energies of various natures ($\pi\to\pis$, $n \to\pis$, double excitation, Rydberg, singlet, doublet, triplet, etc) for small- and medium-sized molecules.
These values have been obtained using a combination of high-order coupled cluster and selected configuration interaction calculations using increasingly large diffuse basis sets.
One of the key aspect of the so-called QUEST dataset of vertical excitations is that it does not rely on any experimental values, avoiding potential biases inherently linked to experiments and facilitating in the process theoretical cross comparisons.
Following this composite protocol, we have been able to produce theoretical best estimate (TBEs) with the aug-cc-pVTZ basis set, as well as basis set corrected TBEs (i.e., near the complete basis set limit) for each of these transitions.
These TBEs have been employed to benchmark a large number of (lower-order) wave function methods such as CIS(D), ADC(2), STEOM-CCSD, EOM-CCSD, CCSDR(3), CCSDT-3, ADC(3), CC3, CASPT2, NEVPT2, and others.
In order to gather the huge number of data produced during the QUEST project, we have created a website where one can easily test and compare the accuracy of a given method with respect to various variables such as the molecule size or its family, the nature of the excited states, the size of the basis set, and many others.
We hope that the present review will provide a useful summary of our work so far and foster new developments around excited-state methods.
Nowadays, there exists a very large number of electronic structure computational approaches, more or less expensive depending on their overall accuracy, able to quantitatively predict the absolute and/or relative energies of electronic states in molecular systems \cite{JensenBook}.
One important aspect of some of these theoretical methods is their ability to access the energies of electronic excited states, i.e., states that have higher total energies than the so-called ground state (that is, the lowest-energy state).
The faithful description of excited states is particularly challenging from a theoretical point of view \cite{Gonzales_2012,Ghosh_2018,Loos_2020a} and is key to a deeper understanding of photochemical and photophysical processes like absorption, fluorescence, or even chemoluminescence \cite{Bernardi_1996,Olivucci_2010,Robb_2007,Navizet_2011}.
The reasons behind this are (at least) twofold: i) one might lack a proper variational principle for excited-state energies, and ii) excited states are often very close in energy from each other but they can have very different natures ($\pi\to\pis$, $n \to\pis$, charge transfer, double excitation, valence, Rydberg, singlet, doublet, triplet, etc).
Designing excited-state methods able to tackle on the same footing all these types of excited states at an affordable cost remain an open challenge in theoretical computational chemistry \cite{Gonzales_2012,Ghosh_2018,Loos_2020a}.
When one designs a new theoretical model, the first feature that one might want to test is its overall accuracy, i.e., its ability to reproduce reference (or benchmark) values for a given system with well-defined setup (same geometry, basis set, etc).
These values can be absolute or relative energies, geometrical parameters, physical or chemical properties, extracted from experiments, high-level theoretical calculations, or a combination of both.
To do so, the electronic structure community has designed along the years benchmark sets, i.e., sets of molecules for which one could (very) accurately compute theoretical estimates and/or access solid experimental data for given properties.
Regarding ground-states properties, two of the oldest and most employed sets are probably the Gaussian-1 and Gaussian-2 benchmark sets \cite{Pople_1989,Curtiss_1991,Curtiss_1997} developed by the group of Pople in the 1990's.
For example, the Gaussian-2 set gathers atomization energies, ionization energies, electron affinities, proton affinities, bond dissociation energies, and reaction barriers.
This set was subsequently extended and refined \cite{Curtiss_1998,Curtiss_2007}.
Another very useful set for the design of methods able to catch dispersion effects is the S22 benchmark set \cite{Jureka_2006} (and its extended S66 version \cite{Rezac_2011}) of Hobza and collaborators which provides benchmark interaction energies for weakly-interacting (non covalent) systems.
One could also mentioned the $GW$100 set \cite{vanSetten_2015,Krause_2015,Maggio_2016} (and its $GW$5000 extension \cite{Stuke_2020}) of ionization energies which has helped enormously the community to settle on the implementation of $GW$-type methods for molecular systems \cite{vanSetten_2013,Bruneval_2016,Caruso_2016,Govoni_2018}.
The extrapolated ab initio thermochemistry (HEAT) set designed to achieve high accuracy for enthalpies of formation of atoms and small molecules (without experimental data) is yet another successful example of benchmark set \cite{Tajti_2004,Bomble_2006,Harding_2008}.
More recently, the benchmark datasets provided by the \textit{Simons Collaboration on the Many-Electron Problem} have been extremely valuable to the community by providing, for example, highly-accurate ground state energies for hydrogen chains \cite{Motta_2017} and transition metal atoms and their ions and monoxides \cite{Williams_2020}.
Let us also mention the set of Zhao and Truhlar for small transition metal complexes employed to compare the accuracy density-functional methods for $3d$ transition-metal chemistry \cite{Zhao_2006}.
The examples presented above are all designed for ground-state properties, and there exists now specific protocols designed to accurately model excited-state energies and properties.
Benchmark datasets of excited-state energies and/or properties are less numerous than their ground-state counterparts but their number have been growing at a consistent pace in the past few years.
Below, we provide a short description for some of them.
One of the most characteristic example is the benchmark set of vertical excitations proposed by Thiel and coworkers \cite{Schreiber_2008,Silva-Junior_2008,Silva-Junior_2010,Silva-Junior_2010b,Silva-Junior_2010c}.
The so-called Thiel (or M\"ulheim) set of excitation energies gathers a large number of excitation energies determined in 28 medium-size organic molecules with a total of 223 valence excited states (152 singlet and 71 triplet states) for which theoretical best estimates (TBEs) were defined.
In their first study, Thiel and collaborators performed CC2 \cite{Christiansen_1995a,Hattig_2000}, CCSD \cite{Purvis_1982}, CC3 \cite{Christiansen_1995b,Koch_1997}, and CASPT2 \cite{Andersson_1990,Andersson_1992,Roos,Roos_1996} calculations (with the TZVP basis) on MP2/6-31G(d) geometries in order to provide (based on additional high-quality literature data) TBEs for these transitions \cite{Silva-Junior_2010b}.
These TBEs were quickly refined with the larger aug-cc-pVTZ basis set, highlighting the importance of diffuse functions in the faithful description of excited states (especially for Rydberg states).
In the same spirit, it is also worth mentioning Gordon's set of vertical transitions (based on experimental values) used to benchmark the performance of time-dependent density-functional theory (TD-DFT) \cite{Leang_2012}, as well as its extended version by Goerigk and coworkers who decided to replace the experimental reference values by CC3 excitation energies instead \cite{Schwabe_2017,Casanova-Paez_2019,Casanova_Paes_2020}.
Let us also mention the new benchmark set of charge-transfer excited states recently introduced by Szalay and coworkers [based on coupled cluster (CC) methods] \cite{Kozma_2020} as well as the Gagliardi-Truhlar set employed to compare the accuracy of multiconfiguration pair-density functional theory \cite{Ghosh_2018} against the well-established CASPT2 method \cite{Hoyer_2016}.
Following a similar philosophy and striving for chemical accuracy, we have recently reported in several studies highly-accurate vertical excitations for small- and medium-sized molecules \cite{Loos_2020a,Loos_2018a,Loos_2019,Loos_2020b,Loos_2020c}.
The so-called QUEST dataset of vertical excitations which we will describe in details in the present review article is composed by 5 subsets (see Fig.~\ref{fig:scheme}): i) a subset of excitations in small molecules containing from 1 to 3 non-hydrogen atoms known as QUEST\#1, ii) a subset of double excitations for molecules of small and medium sizes known as QUEST\#2, iii) a subset of excitation energies for medium-sized molecules containing from 4 to 6 non-hydrogen atoms known as QUEST\#3, iv) a subset composed by more ``exotic'' molecules and radicals labeled as QUEST\#4, and v) a subset known as QUEST\#5, specifically designed for the present article, gathering excitation energies in larger molecules as well as additional smaller molecules.
One of the key aspect of the QUEST dataset is that it does not rely on any experimental values, avoiding potential biases inherently linked to experiments and facilitating in the process theoretical comparisons.
Moreover, our protocol has been designed to be as uniform as possible, which means that we have designed a very systematic procedure for all excited states in order to make cross-comparison as straightforward as possible.
Importantly, it allowed us to benchmark, in a very systematic and fair way, a series of popular excited-state wave function methods partially or fully accounting for double and triple excitations as well as multiconfigurational methods (see below).
In the same vein, we have also produced chemically-accurate theoretical 0-0 energies \cite{Loos_2018,Loos_2019a,Loos_2019b} which can be more straightforwardly compare to experimental data \cite{Kohn_2003,Dierksen_2004,Goerigk_2010a,Send_2011a,Winter_2013,Fang_2014}.
We refer the interested reader to Ref.~\cite{Loos_2019b} where we review the generic benchmark studies devoted to adiabatic and 0-0 energies performed in the past two decades.
The QUEST dataset has the particularity to be based in a large proportion on selected configuration interaction (SCI) reference excitation energies as well as high-order CC methods such as CCSDT and CCSDTQ \cite{Oliphant_1991,Kucharski_1992}.
Recently, SCI methods have been a force to reckon with for the computation of highly-accurate energies in small- and medium-sized molecules as they yield near full configuration interaction (FCI) quality energies for only a fraction of the computational cost of a genuine FCI calculation \cite{Holmes_2017,Chien_2018,Loos_2018a,Li_2018,Loos_2019,Loos_2020b,Loos_2020c,Loos_2020a,Li_2020,Eriksen_2020,Loos_2020e,Yao_2020}.
Due to the fairly natural idea underlying these methods, the SCI family is composed by numerous members \cite{Bender_1969,Whitten_1969,Huron_1973,Abrams_2005,Bunge_2006,Bytautas_2009,Giner_2013,Caffarel_2014,Giner_2015,Garniron_2017b,Caffarel_2016a,Caffarel_2016b,Holmes_2016,Sharma_2017,Holmes_2017,Chien_2018,Scemama_2018,Scemama_2018b,Garniron_2018,Evangelista_2014,Schriber_2016,Schriber_2017,Liu_2016,Per_2017,Ohtsuka_2017,Zimmerman_2017,Li_2018,Ohtsuka_2017,Coe_2018,Loos_2019}.
Their fundamental philosophy consists, roughly speaking, in retaining only the most energetically relevant determinants of the FCI space following a given criterion to avoid the exponential increase of the size of the CI expansion.
Originally developed in the late 1960's by Bender and Davidson \cite{Bender_1969} as well as Whitten and Hackmeyer \cite{Whitten_1969}, new efficient SCI algorithms have resurfaced recently.
Four examples are adaptive sampling CI (ASCI) \cite{Tubman_2016,Tubman_2018,Tubman_2020}, iCI \cite{Liu_2016}, semistochastic heat-bath CI (SHCI) \cite{Holmes_2016,Holmes_2017,Sharma_2017,Li_2018}), and \textit{Configuration Interaction using a Perturbative Selection made Iteratively} (CIPSI) \cite{Huron_1973,Giner_2013,Giner_2015,Garniron_2019}.
These four flavors of SCI include a second-order perturbative (PT2) correction which is key to estimate the ``distance'' to the FCI solution (see below).
The QUEST set of excitation energies relies on the CIPSI algorithm, which is, from a historical point of view, one of the oldest SCI algorithm.
It was developed in 1973 by Huron, Rancurel, and Malrieu \cite{Huron_1973} (see also Refs.~\cite{Evangelisti_1983,Cimiraglia_1985,Cimiraglia_1987,Illas_1988,Povill_1992}).
Recently, the determinant-driven CIPSI algorithm has been efficiently implemented \cite{Garniron_2019} in the open-source programming environment QUANTUM PACKAGE by our group enabling to perform massively parallel computations \cite{Garniron_2017,Garniron_2018,Garniron_2019,Loos_2020e}.
CIPSI is also frequently employed to provide accurate trial wave functions for quantum Monte Carlo calculations in molecules \cite{Caffarel_2014,Caffarel_2016a,Caffarel_2016b,Giner_2013,Giner_2015,Scemama_2015,Scemama_2016,Scemama_2018,Scemama_2018b,Scemama_2019,Dash_2018,Dash_2019,Scemama_2020} and more recently for periodic solids \cite{Benali_2020}.
We refer the interested reader to Ref.~\cite{Garniron_2019} where one can find additional details regarding the implementation of the CIPSI algorithm.
In Sec.~\ref{sec:tools}, we detail the specificities of our protocol by providing computational details regarding geometries, basis sets, (reference and benchmarked) computational methods, and a new way of estimating rigorously the extrapolation error in SCI calculations.
We then describe in Sec.~\ref{sec:QUEST} the content of our five QUEST subsets providing for each of them the number of reference excitation energies, the nature and size of the molecules, the list of benchmarked methods, as well as other specificities.
A special emphasis is placed on our latest add-on, QUEST\#5, specifically designed for the present manuscript where we have considered, in particular but not only, larger molecules as well as additional FCI values for five- and six-membered rings.
Section \ref{sec:TBE} discusses the generation of the TBEs, while Sec.~\ref{sec:bench} proposes a comprehensive benchmark of various methods on the entire QUEST set which is composed by more than \alert{470} excitations with, in addition, a specific analysis for each type of excited states.
Section \ref{sec:website} describes the feature of the website that we have specifically designed to gather the entire data generated during these last few years.
Thanks to this website, one can easily test and compare the accuracy of a given method with respect to various variables such as the molecule size or its family, the nature of the excited states, the size of the basis set, etc.
Finally, we draw our conclusions in Sec.~\ref{sec:ccl} where we discuss, in particular, future projects aiming at expanding and improving the usability and accuracy of the QUEST database.
For the open-shell derivatives \cite{Loos_2020c}, the geometries are optimized at the UCCSD(T)/aug-cc-pVTZ level using the GAUSSIAN16 program \cite{Gaussian16} and applying the ``tight'' convergence threshold.
For the entire set, we rely on the 6-31+G(d) Pople basis set, the augmented family of Dunning basis sets aug-cc-pVXZ (where X $=$ D, T, Q, and 5), and sometimes its doubly- and triply-augmented variants, d-aug-cc-pVXZ and t-aug-cc-pVXZ respectively.
Doubly- and triply-augmented basis sets are usually employed for Rydberg states where it is not uncommon to observe a strong basis set dependence due to the very diffuse nature of these excited states.
In order to compute reference vertical energies, we have designed different strategies depending on the actual nature of the transition and the size of the system.
For small molecules (typically 1--3 non-hydrogen atoms), we resort to SCI methods which can provide near-FCI excitation energies for compact basis sets.
Obviously, the smaller the molecule, the larger the basis we can afford.
For larger systems (\ie, 4--6 non-hydrogen atom), one cannot afford SCI calculations anymore expect in a few exceptions, and we then rely on CC theory (CCSDT and CCSDTQ typically) to obtain accurate transition energies.
Note that all our excited-state CC calculations are performed within the equation-of-motion (EOM) or linear-response (LR) formalism that yield the same excited-state energies.
For open-shell molecules, the CCSDT, CCSDTQ, and CCSDTQP calculations performed with MRCC \cite{Rolik_2013,mrcc} do consider an unrestricted Hartree-Fock wave function as reference.
All excited-state calculations are performed, except when explicitly mentioned, in the frozen-core (FC) approximation using large cores for the third-row atoms.
All the SCI calculations are performed within the FC approximation using QUANTUM PACKAGE \cite{Garniron_2019} where the CIPSI algorithm \cite{Huron_1973} is implemented. Details regarding this specific CIPSI implementation can be found in Refs.~\cite{Garniron_2019} and \cite{Scemama_2019}.
A state-averaged formalism is employed, i.e., the ground and excited states are described with the same set of determinants, but different CI coefficients.
Our usual protocol \cite{Scemama_2018,Scemama_2018b,Scemama_2019,Loos_2018a,Loos_2019,Loos_2020a,Loos_2020b,Loos_2020c} consists of performing a preliminary CIPSI calculation using Hartree-Fock orbitals in order to generate a CIPSI wave function with at least $10^7$ determinants.
Natural orbitals are then computed based on this wave function, and a new, larger CIPSI calculation is performed with this new set of orbitals.
The CIPSI energy $E_\text{CIPSI}$ is defined as the sum of the variational energy $E_\text{var}$ (computed via diagonalization of the CI matrix in the reference space) and a PT2 correction $E_\text{PT2}$ which estimates the contribution of the determinants not included in the CI space \cite{Garniron_2017b}.
By linearly extrapolating this second-order correction to zero, one can efficiently estimate the FCI limit for the total energies.
These extrapolated total energies (simply labeled as $E_\text{FCI}$ in the remainder of the paper) are then used to compute vertical excitation energies.
Depending on the set, we estimated the extrapolation error via different techniques.
For example, in Ref.~\cite{Loos_2020b}, we estimated the extrapolation error by the difference between the transition energies obtained with the largest SCI wave function and the FCI extrapolated value.
This definitely cannot be viewed as a true error bar, but it provides a rough idea of the quality of the FCI extrapolation and estimate.
Below, we provide a much cleaner way of estimating the extrapolation error in SCI methods, and we adopt this scheme for the five- and six-membered rings.
The particularity of the current implementation is that the selection step and the PT2 correction are computed \textit{simultaneously} via a hybrid semistochastic algorithm \cite{Garniron_2017,Garniron_2019}.
Moreover, a renormalized version of the PT2 correction (dubbed rPT2) has been recently implemented for a more efficient extrapolation to the FCI limit \cite{Garniron_2019}.
Note that, all our SCI wave functions are eigenfunctions of the $\Hat{S}^2$ spin operator which is, unlike ground-state calculations, paramount in the case of excited states \cite{Applencourt_2018}.
Using a large variety of codes, our benchmark effort consists in evaluating the accuracy of vertical transition energies obtained at lower levels of theory.
For example, we rely on GAUSSIAN \cite{Gaussian16} and TURBOMOLE 7.3 \cite{Turbomole} for CIS(D) \cite{Head-Gordon_1994,Head-Gordon_1995};
Q-CHEM 5.2 \cite{Krylov_2013} for EOM-MP2 [CCSD(2)] \cite{Stanton_1995c} and ADC(3) \cite{Trofimov_2002,Harbach_2014,Dreuw_2015};
Q-CHEM \cite{Krylov_2013} and TURBOMOLE \cite{Turbomole} for ADC(2) \cite{Trofimov_1997,Dreuw_2015};
DALTON \cite{dalton} and TURBOMOLE \cite{Turbomole} for CC2 \cite{Christiansen_1995a,Hattig_2000};
DALTON \cite{dalton} and GAUSSIAN \cite{Gaussian16} for CCSD \cite{Purvis_1982};
DALTON \cite{dalton} for CCSDR(3) \cite{Christiansen_1996b};
CFOUR \cite{cfour} for CCSDT-3 \cite{Watts_1996b,Prochnow_2010};
and ORCA \cite{Neese_2012} for similarity-transformed EOM-CCSD (STEOM-CCSD) \cite{Nooijen_1997,Dutta_2018}.
In addition, we evaluate the spin-opposite scaling (SOS) variants of ADC(2), SOS-ADC(2), as implemented in both Q-CHEM \cite{Krauter_2013} and TURBOMOLE \cite{Hellweg_2008}.
Note that these two codes have distinct SOS implementations, as explained in Ref.~\cite{Krauter_2013}.
We also test the SOS and spin-component scaled (SCS) versions of CC2, as implemented in TURBOMOLE \cite{Hellweg_2008,Turbomole}.
Discussion of various spin-scaling schemes can be found elsewhere \cite{Goerigk_2010a}.
When available, we take advantage of the resolution-of-the-identity (RI) approximation in TURBOMOLE and Q-CHEM.
For the STEOM-CCSD calculations, it was checked that the active character percentage was, at least, $98\%$.
When comparisons between various codes/implementations were possible, we could not detect variations in the transition energies larger than $0.01$ eV.
For radicals, we applied both the U (unrestricted) and RO (restricted open-shell) versions of CCSD and CC3 as implemented in the PSI4 code \cite{Psi4} to perform our benchmarks.
Finally, the composite approach, ADC(2.5), which follows the spirit of Grimme's and Hobza's MP2.5 approach \cite{Pitonak_2009} by averaging the ADC(2) and ADC(3) excitation energies, is also tested in the following \cite{Loos_2020d}.
State-averaged (SA) CASSCF and CASPT2 \cite{Roos,Andersson_1990} have been performed with MOLPRO (RS2 contraction level) \cite{molpro}.
Concerning the NEVPT2 calculations, the partially-contracted (PC) and strongly-contracted (SC) variants have been tested \cite{Angeli_2001a,Angeli_2001b,Angeli_2002}.
From a strict theoretical point of view, we point out that PC-NEVPT2 is supposed to be more accurate than SC-NEVPT2 given that it has a larger number of perturbers and greater flexibility.
In the case of double excitations \cite{Loos_2019}, we have also performed calculations with multi-state (MS) CASPT2 (MS-MR formalism), \cite{Finley_1998} and its extended variant (XMS-CASPT2) \cite{Shiozaki_2011} when there is a strong mixing between states with same spin and spatial symmetries.
The CASPT2 calculations have been performed with level shift and IPEA parameters set to the standard values of $0.3$ and $0.25$ a.u., respectively, unless otherwise stated.
Large active spaces carefully chosen and tailored for the desired transitions have been selected.
The definition of the active space considered for each system as well as the number of states in the state-averaged calculation is provided in their corresponding publication.
For the $m$th excited state (where $m =0$ corresponds to the ground state), we usually estimate its FCI energy $E_{\text{FCI}}^{(m)}$ by performing a linear extrapolation of its variational energy $E_\text{var}^{(m)}$ as a function of its rPT2 correction $E_{\text{rPT2}}^{(m)}$ as follows
$E_\text{var}^{(m)}$ varies almost linearly as a function of $E_{\text{rPT2}}^{(m)}$, but with a coefficient $\alpha^{(m)}$ which deviates slightly from unity in well-behaved cases.
For the large systems considered here, $\abs{E_{\text{rPT2}}} > 2$ eV.
Therefore, the accuracy of the excitation energy estimates will strongly depend on our ability to compensate the errors in the calculations.
Because our selection procedure ensures that the rPT2 values of both states match as well as possible (a trick known as PT2 matching \cite{Dash_2018,Dash_2019}), i.e., $E_{\text{rPT2}}= E_{\text{rPT2}}^{(0)}\approx E_{\text{rPT2}}^{(m)}$, the extrapolated excitation energy associated with the $m$th excited state can be estimated as
which evidences that the error in $\Delta E_{\text{FCI}}^{(m)}$ can be expressed as $\qty(\alpha^{(m)}-\alpha^{(0)}) E_{\text{rPT2}}+\order{E_{\text{rPT2}}^2}$.
Moreover, using a common set of state-averaged natural orbitals for the ground and excited states tends to make the values of $\alpha^{(0)}$ and $\alpha^{(m)}$ very close to each other, such that the error on the energy difference is practically of the order of $E_{\text{rPT2}}^2$.
At the $n$th CIPSI iteration, we have access to the variational energies of both states, $E_\text{var}^{(0)}(n)$ and $E_\text{var}^{(m)}(n)$, as well as their rPT2 corrections, $E_{\text{rPT2}}^{(0)}(n)$ and $E_{\text{rPT2}}^{(m)}(n)$.
The $m$th excitation energy at iteration $n$ is then assumed to be a Gaussian random variable with mean and variance
We then search for a confidence interval $\mathcal{I}$ such that the true value of the excitation energy $\Delta E_{\text{FCI}}^{(m)}$ lies within one standard deviation of $\Delta E_\text{CIPSI}^{(m)}$, i.e., $P(\Delta E_{\text{FCI}}^{(m)}\in[\Delta E_\text{CIPSI}^{(m)}\pm\sigma]\; | \;\mathcal{G})=0.6827$.
The inverse of the cumulative distribution function of the $t$-distribution, $t_{\text{CDF}}^{-1}$, allows us to find how to scale the interval by a parameter
The QUEST database gathers more than \alert{470} highly-accurate excitation energies of various natures (valence, Rydberg, $n \ra\pis$, $\pi\ra\pis$, singlet, doublet, triplet, and double excitations) for molecules ranging from diatomics to molecules as large as naphthalene (see Fig.~\ref{fig:molecules}).
Throughout the present article, we report several statistical indicators: the mean signed error (MSE), mean absolute error (MAE), root-mean square error (RMSE), and standard deviation of the errors (SDE).
The QUEST\#1 benchmark set \cite{Loos_2018a} consists of 110 vertical excitation energies (as well as oscillator strengths) from 18 molecules with sizes ranging from one to three non-hydrogen atoms (water, hydrogen sulfide, ammonia, hydrogen chloride, dinitrogen, carbon monoxide, acetylene, ethylene, formaldehyde, methanimine, thioformaldehyde, acetaldehyde, cyclopropene, diazomethane, formamide, ketene, nitrosomethane, and the smallest
streptocyanine). For this set, we provided two sets of TBEs: i) one obtained within the frozen-core approximation and the aug-cc-pVTZ basis set, and ii) another one including further corrections for basis set incompleteness and ``all electron'' effects.
For the former set, we systematically selected FCI/aug-cc-pVTZ values to define our TBEs except in very few cases.
For the latter set, both the ``all electron'' correlation and the basis set corrections were systematically obtained at the CC3 level of theory and with the d-aug-cc-pV5Z basis for the nine smallest molecules, and slightly more compact basis sets for the larger compounds.
Our TBE/aug-cc-pVTZ reference excitation energies were employed to benchmark a series of popular excited-state wave function methods partially or fully accounting for double and triple excitations, namely CIS(D), CC2, CCSD, STEOM-CCSD, CCSDR(3), CCSDT-3, CC3, ADC(2), and ADC(3).
Our main conclusions were that i) ADC(2) and CC2 show strong similarities in terms of accuracy, ii) STEOM-CCSD is, on average, as accurate as CCSD, the latter overestimating transition energies, iii) CC3 is extremely accurate (with a mean absolute error of only $\sim0.03$ eV) and that although slightly less accurate than CC3, CCSDT-3 could be used as a reliable reference for benchmark studies, and iv) ADC(3) was found to be significantly less accurate than CC3 by overcorrecting ADC(2) excitation energies.
The QUEST\#2 benchmark set \cite{Loos_2019} reports reference energies for double excitations.
This set gathers 20 vertical transitions from 14 small- and medium-size molecules (acrolein, benzene, beryllium atom, butadiene, carbon dimer and trimer, ethylene, formaldehyde, glyoxal, hexatriene, nitrosomethane, nitroxyl, pyrazine, and tetrazine).
The TBEs of the QUEST\#2 set are obtained with SCI and/or multiconfigurational [CASSCF, CASPT2, (X)MS-CASPT2, and NEVPT2] calculations depending on the size of the molecules and the level of theory that we could afford.
An important addition to this second study was the inclusion of various flavors of multiconfigurational methods (CASSCF, CASPT2, and NEVPT2) in addition to high-order CC methods including, at least, perturbative triples (CC3, CCSDT, CCSDTQ, etc).
Our results demonstrated that the error of CC methods is intimately linked to the amount of double-excitation character in the vertical transition.
For ``pure'' double excitations (i.e., for transitions which do not mix with single excitations), the error in CC3 and CCSDT can easily reach $1$ and $0.5$ eV, respectively, while it goes down to a few tenths of an eV for more common transitions (such as in butadiene and benzene) involving a significant amount of single excitations.
The quality of the excitation energies obtained with CASPT2 and NEVPT2 was harder to predict as the overall accuracy of these methods is highly dependent on both the system and the selected active space.
Nevertheless, these two methods were found to be more accurate for transition with a small percentage of single excitations (error usually below $0.1$ eV) than for excitations dominated by single excitations where the error is closer from $0.1$--$0.2$ eV
The QUEST\#3 benchmark set \cite{Loos_2020b} is, by far, our largest set, and consists of highly accurate vertical transition energies and oscillator strengths obtained for 27 molecules encompassing 4, 5, and 6 non-hydrogen atoms (acetone, acrolein, benzene, butadiene, cyanoacetylene, cyanoformaldehyde, cyanogen, cyclopentadiene, cyclopropenone, cyclopropenethione, diacetylene, furan, glyoxal, imidazole, isobutene, methylenecyclopropene, propynal, pyrazine, pyridazine, pyridine, pyrimidine, pyrrole, tetrazine, thioacetone, thiophene, thiopropynal, and triazine) for a total of 238 vertical transition energies and 90 oscillator strengths with a reasonably good balance between singlet, triplet, valence, and Rydberg excited states.
For these 238 transitions, we have estimated that 224 are chemically accurate for the considered geometry.
To define the TBEs of the QUEST\#3 set, we employed CC methods up to the highest technically possible order (CC3, CCSDT, and CCSDTQ), and, when affordable SCI calculations with very large reference spaces (up to hundred million determinants in certain cases), as well as the most reliable multiconfigurational method, NEVPT2, for double excitations.
Most of our TBEs are based on CCSDTQ (4 non-hydrogen atoms) or CCSDT (5 and 6 non-hydrogen atoms) excitation energies.
For all the transitions of the QUEST\#3 set, we reported at least CCSDT/aug-cc-pVTZ (sometimes with basis set extrapolation) and CC3/aug-cc-pVQZ transition energies as well as CC3/aug-cc-pVTZ oscillator strengths for each dipole-allowed transition.
Pursuing our previous benchmarking efforts, we confirmed that CC3 almost systematically delivers transition energies in agreement with higher-level theoretical models ($\pm0.04$ eV) except for transitions presenting a dominant double-excitation character where multiconfigurational methods like NEVPT2 have clearly the edge.
This settles down, at least for now, the debate by demonstrating the superiority of CC3 (in terms of accuracy) compared to methods like CCSDT-3 or ADC(3).
This was further demonstrated in a recent study by two of the present authors \cite{Loos_2020d}.
An ``exotic'' subset of 30 excited states for closed-shell molecules containing F, Cl, P, and Si atoms (carbonyl fluoride, \ce{CCl2}, \ce{CClF}, \ce{CF2}, difluorodiazirine, formyl fluoride, \ce{HCCl}, \ce{HCF}, \ce{HCP}, \ce{HPO}, \ce{HPS}, \ce{HSiF}, \ce{SiCl2}, and silylidene) and a ``radical'' subset of 51 doublet-doublet transitions in small radicals (allyl, \ce{BeF}, \ce{BeH}, \ce{BH2}, \ce{CH}, \ce{CH3}, \ce{CN}, \ce{CNO}, \ce{CON}, \ce{CO+}, \ce{F2BO}, \ce{F2BS}, \ce{H2BO}, \ce{HCO}, \ce{HOC}, \ce{H2PO}, \ce{H2PS}, \ce{NCO}, \ce{NH2}, nitromethyl, \ce{NO}, \ce{OH}, \ce{PH2}, and vinyl) characterized by open-shell electronic configurations and an unpaired electron.
This represents a total of 81 high-quality TBEs, the vast majority being obtained at the FCI level with at least the aug-cc-pVTZ basis set.
We further performed high-order CC calculations to ascertain these estimates.
For the exotic set, these TBEs have been used to assess the performances of 15 ``lower-order'' wave function approaches, including several CC and ADC variants.
Consistent with our previous works, we found that CC3 is very accurate, whereas the trends for the other methods are similar to that obtained on more standard organic compounds.
In contrast, for the radical set, even the refined ROCC3 method yields a MAE of $0.05$ eV.
Likewise, the excitation energies obtained with CCSD are much less satisfying for open-shell derivatives (MAE of $0.20$ eV with UCCSD and $0.15$ eV with ROCCSD) than for the closed-shell systems (MAE of $0.07$ eV).
The QUEST\#5 subset is composed by additional accurate excitation energies that we have produced for the present article.
This new set gathers small molecules as well as larger molecules (aza-naphthalene, benzoquinone, cyclopentadienone, cyclopentadienethione, hexatriene, maleimide, naphthalene, nitroxyl, streptocyanine-C3, streptocyanine-C5, and thioacrolein).
QUEST\#5 does also provide additional FCI/6-31+G* estimates of the lowest singlet and triplet transitions for the five- and six-membered rings considered in QUEST\#3.
The extrapolation errors for these quite challenging calculations are computed with the scheme presented in Sec.~\ref{sec:error}.
The singlet and triplet excitation energies obtained at the FCI/6-31+G(d) level are reported in Table \ref{tab:cycles} alongside the CC3 and CCSDT values in the same basis from Ref.~\cite{Loos_2020b}.
\caption{Singlet and triplet excitation energies obtained at the CC3, CCSDT, and FCI levels of theory with the 6-31+G* basis set for various five- and six-membered rings.}
In this section, we report a comprehensive benchmark of various lower-order methods on the entire QUEST set which is composed by more than \alert{470} excitations.
Additionally, we also provide a specific analysis for each type of excited states.
Here we describe the feature of the website that we have specifically designed to gather the entire data generated during these last few years.
Thanks to this website, one can easily test and compare the accuracy of a given method with respect to various variables such as the molecule size or its family, the nature of the excited states, the size of the basis set, etc.
The previous QUEST publications \cite{Loos_2018a,Loos_2019,Loos_2020b,Loos_2020c,Loos_2020d} expose vertical excitation data, some statistics were provided considering the most relevant parameters.
Because computing 450 excitation energies can be a costly exercise, we are planning on developing a ``diet set'' following the philosophy of the ``diet GMTKN55'' set \cite{Goerigk_2017} proposed recently by Gould \cite{Gould_2018b}.
Although our present goal is to produce chemically accurate vertical excitation energies, we are currently devoting great efforts to obtain of highly-accurate excited-state properties as such dipoles and oscillator strengths for molecules of small and medium sizes \cite{Chrayteh_2020}.
This work was performed using HPC resources from GENCI-TGCC (Grand Challenge 2019-gch0418) and from CALMIP (Toulouse) under allocation 2020-18005.
AS, MC, and PFL thank the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (Grant agreement No.~863481) for financial support.
He received his B.Sc.~in Molecular Chemistry from the Universit\'e Paul Sabatier (Toulouse, France) in 2015 and his M.Sc.~in Computational and Theoretical Chemistry and Modeling from the same university in 2018.
Since 2018, he is a Ph.D.~student in the group of Dr.~Pierre-Fran\c{c}ois Loos at the Laboratoire de Chimie et Physique Quantiques in Toulouse.
He is currently developing QUANTUM PACKAGE and the web application linked to the QUEST project.
received his Ph.D.~in Computational and Theoretical Chemistry from the Universit\'e Pierre et Marie Curie (Paris, France) in 2004.
He then moved to the Netherlands for a one-year postdoctoral stay in the group of Claudia Filippi, and came back in France for another year in the group of Eric Canc\`es.
In 2006, he obtained a Research Engineer position from the \textit{``Centre National de la Recherche Scientifique (CNRS)} at the \textit{Laboratoire de Chimie et Physique Quantiques} in Toulouse (France) to work on computational methods and high-performance computing for quantum chemistry. He was awarded the Crystal medal of the CNRS in 2019.
Please check with the journal's author guidelines whether author biographies are required. They are usually only included for review-type articles, and typically require photos and brief biographies (up to 75 words) for each author.
Please check with the journal's author guidelines whether author biographies are required. They are usually only included for review-type articles, and typically require photos and brief biographies (up to 75 words) for each author.
Please check with the journal's author guidelines whether author biographies are required. They are usually only included for review-type articles, and typically require photos and brief biographies (up to 75 words) for each author.
From 2009 to 2013, He was undertaking postdoctoral research with Peter M.W.~Gill at the Australian National University (ANU).
From 2013 to 2017, he was a \textit{``Discovery Early Career Researcher Award''} recipient and, then, a senior lecturer at the ANU.
Since 2017, he holds a researcher position from the \textit{``Centre National de la Recherche Scientifique (CNRS)} at the \textit{Laboratoire de Chimie et Physique Quantiques} in Toulouse (France), and was awarded, in 2019, an ERC consolidator grant for the development of new excited-state methodologies.
received his PhD in Chemistry from the University of Namur in 1998, before moving to the University of Florida for his postdoctoral stay. He is currently full Professor at the University of Nantes (France).
His research is focused on modeling electronically excited-state processes in organic and inorganic dyes as well as photochromes using a large panel of \emph{ab initio} approaches. His group collaborates with many experimental and theoretical groups.
He is the author of more than 500 scientific papers. He has been ERC grantee (2011--2016), member of Institut Universitaire de France (2012--2017) and received the WATOC's Dirac Medal (2014).
