The Band-Gap Problem and Why DFT Needs Help

Learning Objectives

By the end of this section you should be able to:

1. State, in one sentence, why the band gap of a standard PBE calculation is roughly half of the experimental value — and why this is not a numerical bug but a structural feature of any local exchange-correlation approximation.
2. Identify self-interaction error as the underlying disease, and recognise its two clinical symptoms: under-binding of localised states and over-delocalisation of charge.
3. Read a hybrid functional like $E_\text{xc}^\text{PBE0} = \tfrac{1}{4}\,E_\text{x}^\text{HF} + \tfrac{3}{4}\,E_\text{x}^\text{PBE} + E_\text{c}^\text{PBE}$ and translate every term back to the physical principle it enforces.
4. Place PBE, PBE0, HSE06, and the exact Kohn–Sham potential on Jacob's ladder — and predict, before running a single calculation, which one is appropriate for a given material.
5. Explain why the choice of algorithm (Davidson, RMM-DIIS, Pulay mixing) is just as load-bearing for a hybrid run as the choice of functional, and why a good algorithm can turn a 30× slowdown into a 3× one.

One-line preview: Pure DFT is wrong about band gaps in a very specific, very fixable way. The fix — mixing in a fraction of Hartree–Fock exchange — is conceptually one line and computationally a full chapter, because the integrals involved are nonlocal and the iterative algorithms that worked for PBE no longer converge on autopilot.

The Failure That Launched a Thousand Functionals

At the end of Chapter 5 we built the band structure of CdSe in PBE and read off a fundamental gap of $\sim 0.7\,\text{eV}$. The experimental gap of bulk zincblende CdSe is $1.74\,\text{eV}$ at room temperature. PBE is not slightly off — it is off by more than a factor of two. This is not a CdSe problem; it is a generic feature of every standard exchange-correlation functional in the LDA/GGA family. The pattern is so robust that it has its own name: the DFT band-gap problem.

Three lessons from that table. First, the error is systematic: pure DFT gaps are almost always too small, never too large. Second, the error is material-dependent: covalent semiconductors lose a factor of two; transition-metal oxides like NiO are not just quantitatively but qualitatively wrong, predicted as metals when they are insulators. Third, the error scales with the localisation of the relevant orbitals — Ni 3d in NiO is far more localised than Si 3p in silicon, and the gap collapses accordingly. We will see why in a moment.

Where the Error Comes From

The Kohn–Sham equations replace the interacting many-electron problem with a fictitious system of non-interacting electrons moving in an effective potential

The first two pieces — the external potential and the Hartree (classical Coulomb) potential — are exact and local. Everything quantum and interacting has been pushed into the third term, the exchange-correlation potential $v_\text{xc}$. The exact $v_\text{xc}$ is unknown; LDA approximates it as a function of the local density $\rho(\mathbf{r})$, GGA also as a function of the gradient $\nabla \rho$. Both are local — they depend only on the density at point $\mathbf{r}$.

The exact exchange interaction, by contrast, is irreducibly nonlocal:

That double integral over occupied orbitals $\psi_i, \psi_j$ contains the Pauli exclusion principle in its most physical form: an electron at $\mathbf{r}$ sees a hole at $\mathbf{r}'$ with weight set by the orbital overlap. Replace it with a function of the local density and you have thrown away the long-range tail of that hole. The price you pay is a residual self-interaction: in pure DFT, every electron partly repels itself.

Two Symptoms of the Same Disease

Self-interaction error manifests in two ways that look unrelated until you stare at them long enough to see they are the same equation in different clothes.

Symptom 1 — Underestimated band gaps

The fundamental gap of a solid is the difference between the ionisation potential and the electron affinity:

The exact Kohn–Sham gap differs from this fundamental gap by a discontinuity $\Delta_\text{xc}$ in the exchange-correlation potential when an electron is added — a well-known formal result of Perdew & Levy and Sham & Schlüter (1983). LDA and GGA functionals are continuous functionals of the density, so by construction their $\Delta_\text{xc} = 0$. The DFT gap is missing a contribution that is, in many materials, comparable in size to the gap itself. That is the gap problem in one equation.

Symptom 2 — Over-delocalised charge

Localise an electron on a defect site, and self-interaction pushes its density outward to lower the spurious self-repulsion. The result is a defect orbital that looks too diffuse, polarons that prefer to delocalise across many sites instead of self-trapping, and band edges whose orbital character is smeared across too many atoms. In NiO, the Ni 3d electrons want to be tightly localised on each Ni site — Hund's rules and crystal-field splitting demand it. PBE refuses to let them, and the system collapses to a metallic ground state.

The Fix in One Sentence: Mix in Some Exact Exchange

Hartree–Fock has zero self-interaction (it cancels exactly between the exchange and Hartree terms) but no correlation, so HF gaps come out far too large: 7+ eV for silicon, 13 eV for diamond. PBE has decent correlation but residual self-interaction, so its gaps come out too small. The simplest possible thing to do is mix them:

The fraction $a$ is the exact-exchange mixing parameter. The two most influential choices in the literature both fall out of physical arguments rather than fitting:

For solids, plain PBE0 is usually too expensive and slightly over-corrects the gap; HSE06 is the default workhorse because the screening of the long-range Coulomb interaction in a metal or small-gap semiconductor justifies dropping the long-range exchange tail. We dedicate Section 6.5 to HSE06 in detail.

Jacob's Ladder of Functionals

John Perdew organised the zoo of exchange-correlation functionals into what he called Jacob's ladder: each rung adds a new ingredient and, in principle, a new piece of physics.

Two things to take away. First, every rung adds non-local information about the electronic structure — first the gradient, then the kinetic energy density, then the occupied orbitals (for HF exchange), then the virtual orbitals. Second, the cost grows roughly an order of magnitude per rung. Hybrid functionals are the first rung where the gap problem is meaningfully fixed, but they are also the first rung where the algorithmic cost forces you to think hard about the underlying SCF machinery.

Why Algorithms Matter as Much as Functionals

Switching from PBE to HSE06 is a one-line change in your INCAR:

1# INCAR — switch from PBE to HSE06
2LHFCALC = .TRUE.        # turn on Hartree-Fock exchange
3HFSCREEN = 0.2          # range-separation parameter (Å^-1) — defines HSE06
4AEXX    = 0.25          # 25% short-range HF exchange
5ALGO    = ALL           # all-bands algorithm; needed for nonlocal exchange

The functional has changed by three lines. The cost of the calculation has changed by a factor of 30 to 100. Why? Because every iteration of the SCF cycle now has to evaluate the nonlocal Fock operator — that double integral over all pairs of occupied orbitals — at every k-point. The naive scaling is $\mathcal{O}(N_k^2 \, N_\text{bands}^2 \, N_\text{plane-waves}\,\log N_\text{plane-waves})$ per SCF step, and the SCF cycle itself is harder to converge because the Fock operator is non-diagonal in the plane-wave basis. The naive Davidson algorithm that worked beautifully for PBE struggles. Pulay mixing parameters that were robust for GGAs throw the iteration into oscillation. The functional gives you the right answer; the algorithm decides whether you ever reach it.

What This Looks Like in VASP

Before we dive into the formal pieces, here is the full INCAR you will be running by the end of this chapter on Mn:CdSe — annotated so each tag is recognisable now even if its meaning will be unpacked section by section:

1# INCAR — HSE06 hybrid run for Mn:CdSe (preview)
2
3# ---- Functional ----
4LHFCALC  = .TRUE.       # turn on HF exchange (§6.4, §6.5)
5HFSCREEN = 0.2          # screening (Å^-1); 0.2 = HSE06 (§6.5)
6AEXX     = 0.25         # fraction of HF exchange (§6.4)
7
8# ---- Iterative diagonalization ----
9ALGO     = ALL          # all-bands minimisation; required for nonlocal X (§6.6)
10NELM     = 100          # max SCF steps
11NELMIN   = 6            # min SCF steps
12EDIFF    = 1E-6         # SCF tolerance
13TIME     = 0.4          # trial step for direct minimisation (§6.8)
14
15# ---- Density mixing ----
16AMIX     = 0.2          # linear mixing weight (§6.7)
17BMIX     = 0.0001       # Kerker damping (§6.7)
18AMIX_MAG = 0.8          # mixing for magnetisation (Mn doping)
19BMIX_MAG = 0.0001
20
21# ---- Cost knobs ----
22PRECFOCK = Fast         # coarse FFT grid for the Fock operator (§6.9)
23NKRED    = 2            # downsample k-grid for HF only (§6.9)
24LMAXFOCK = 4            # max ang. mom. for one-centre HF terms (§6.9)
25KPAR     = 4            # k-point parallelism (§6.10)

Twelve tags — each one is a chapter section in disguise. $\texttt{LHFCALC}$, $\texttt{AEXX}$, and $\texttt{HFSCREEN}$ select the functional; $\texttt{ALGO}$, $\texttt{NELMIN}$, and $\texttt{TIME}$ control the iterative diagonalisation; $\texttt{AMIX}$, $\texttt{BMIX}$ set the density mixer; $\texttt{PRECFOCK}$, $\texttt{NKRED}$, and $\texttt{KPAR}$ are the cost-acceleration knobs. By the end of Section 6.10 you will be able to write, annotate, and tune that file from memory.

The Map of Chapter 6

Summary

• Standard LDA/GGA functionals systematically underestimate band gaps by ~40–60 % for typical semiconductors and qualitatively miss the insulating ground state of strongly correlated oxides like NiO.
• The root cause is self-interaction error: local exchange-correlation approximations let an electron partially repel its own density, which (a) shifts band-edge energies and (b) over-delocalises charge.
• The cure is to mix a fraction of exact (Hartree–Fock) exchange into the functional. PBE0 mixes 25 % everywhere; HSE06 mixes 25 % only at short range. Both fix the gap of typical semiconductors to within ~10 % of experiment.
• Jacob's ladder organises functionals by the ingredients they use; hybrids sit on rung 4 and are the first rung that consistently gets band gaps right.
• Hybrids are 10–100× more expensive than GGA per SCF step, and the iteration is harder to converge. Choosing the right algorithm (Davidson / RMM-DIIS / direct minimisation) and density mixer (Pulay, Broyden, Kerker-damped) is just as important as choosing the functional.
• In VASP, switching to HSE06 is a three-tag change to the INCAR ($\texttt{LHFCALC}, \texttt{HFSCREEN}, \texttt{AEXX}$), but a robust hybrid run typically tunes a dozen tags in concert.

Coming next: Section 6.2 — Hartree–Fock Exchange Recap — where we re-derive the Fock operator from the antisymmetrised product wavefunction and look closely at the nonlocal integral that hybrids will eventually mix into the Kohn–Sham potential.