The term for the instability of an atomic nucleus leading to radiation emission is radioactivity. Certain isotopes are unstable and will decay, releasing energy in the form of radiation. This process can produce alpha particles, beta particles, or gamma rays and is a natural mechanism through which elements transform into different elements or isotopes.