HETDEX Opens Massive Cosmic Dataset to Scientists, Novices, and AI

Today, the Hobby-Eberly Telescope Dark Energy Experiment (HETDEX) - which recently completed the largest survey ever taken of the early universe – has released all of its immense, information-rich database to the public. Built from more than half a petabyte of raw and processed data, it will allow astronomers to study how the first galaxies formed and evolved, measure how gas and stars were distributed within these galaxies, map the large-scale structure of the cosmos, and investigate rare and unexpected objects not easily found in traditional surveys.

HETDEX observations make use of a technique called spectroscopy. With it, light is broken apart into its various wavelengths: a spectrum. Astronomers examine spectra for peaks and valleys, which tell them about an object’s chemistry, movement through space, and distance from Earth. The HETDEX database, contains a whopping 600 million spectra for a period of history known as Cosmic Noon, 10 billion to 12 billion years ago.

From 2017 to 2024, the Hobby-Eberly Telescope at McDonald Observatory surveyed a region of night sky equivalent to 2,000 full Moons, creating a map of the distant universe. HETDEX is using that map to solve the riddle of dark energy, the unknown substance causing our universe to expand more and more quickly over time. To do this, it is charting the location of over a million early galaxies. However, it has also gathered data on all of the space in between.

“HETDEX uniquely uses so-called Lyman Alpha Emitting Galaxies in the very distant universe to map out the mass distribution at early times.” says Max Fabricius, scientist at the Max Planck Institute for Extraterrestrial Physics (MPE), and co-author on the paper. “The detection of these early galaxies, presents a unique set of challenges in observation technique and data analysis but holds the key to studying the expansion of the universe during its infancy.”

In addition to raw data, the release also contains a catalog of every object HETDEX has found so far: over one million distant galaxies, half a million nearby star-forming galaxies, 18,000 supermassive black holes, and over 150,000 stars. Scientists, students, and citizen researchers can download customized subsets of data based on sky location.

While the release is based on half a petabyte of data, the team was able to process it down to a more manageable 10 terabytes. It also developed extensive tutorials and tools to help users – both human and AI – to make the most of this massive, complex dataset.

Today’s release marks the first time the full HETDEX dataset and survey catalog have been made available together. MPE was involved in the survey conceptual design and the development of the data analysis pipelines and hosts a copy of the database. While the core survey is now complete, observations are ongoing, calibrations continue to improve, and supplementary releases are expected for the future.

Here you can have access to the data: https://hetdex.org/data-results/