What do you do when you can’t make a telescope big enough? You make three thousand smaller telescopes and band them together. Naturally. Scientists are creating what will become the most sensitive radio telescope in human history. The aptly-named Square Kilometer Array (SKA) has a collecting area of 1,000,000 m2, with its closest competition having only a 13,000 m2 collecting area. Oh, and it will collect over one exabyte, one billion gigabytes of data, every day—more than twice as much data as is transferred on the internet in the same amount of time.
Read more about SKA at this Ars Technica article. We’re going to talk about the data.
This is a map of SKA, spanning about 3,000 km. An instrument that big generates big data; how would you manage it?
To deal with the massive amounts of data, computer scientists are working to design a computer that can perform at 100 petaflops in order to deal with the massive amount of data it takes in. That’s roughly 50 times as powerful as the fastest supercomputer out there right now.
These computer scientists will also have to devise a way to process what data needs to be kept and what can be thrown out. At the end of the year they plan to have as much as 1,500 petabytes of good data left over, something like 0.4% of the data they initially took in. They’ll be doing things like throwing out faulty data (likely to happen with such a sensitive device) and processing to find what is interesting enough to keep.
But it gets better. They’ll be doing Fourier transformations to convert the radio frequency data they do keep into images like we’re used to seeing.
And they’re doing a good many other processes too.
This kind of computer science is sometimes called informatics, the science of dealing with ridiculously huge amounts of data. And with instruments like SKA becoming more common, there’s a growing need for computer scientists who are comfortable in dealing with big data. That means you.
GetReal 