AMIGA for GTC, ALMA and SKA precursors (AMIGA4GAS)

The efficient exploitation of the Exabyte–a-day data rates, such as those expected to be generated by the SKA, represents the biggest challenge for ICTs (Information and Communication Technologies) in the world to date. SKA pathfinders will reach the Terabyte-a-day scale, remarkably increasing the leap in the volume and complexity of present astronomical archives. These volumes preclude standard storage and analysis approaches, and need analysis and data mining tasks to live within data repositories, transforming data providers also into services providers, and hence supplying all the individual components of internet-based workflows. This transformation needs the joint contribution of different computing paradigms (HPC clusters, grid computing, or cloud computing), and to investigate how standard communication procedures could integrate the existing heterogeneous network of resources into federated distributed infrastructures for computing and storage, capable of addressing user needs and technical issues.

The AMIGA group not only has the need, but also the technical expertise in Grid and software development to approach this challenge, so that in collaboration with specialized groups providing the complementary skills can lead to a relevant Spanish contribution to the international state of the art. These technical research require the combined expertise of the AMIGA group in 3D data analysis and VO with groups skilled in the federation of HPC and Grid architectures.  In this context we are working with the Fundación Centro de Supercomputación Castilla y León (FCSCL), with a proven expertise in the domain of virtualization techniques and cloud computing solutions, as demonstrated e.g. by their contribution to NUBA project. They are leading this ICT research exploring possible issues in network, storage and processing in order to deploy AMIGA4GAS scientific workflows on a heterogeneous distributed infrastructure. This infrastrucrure will integrate Spanish High Performance Computing centers and the IberGrid initiative in a federated infrastructure of heterogeneous resources. The Barcelona Supercomputing Center (BSC) and the Instituto de Telecomunicaçoes in Portugal collaborate in these tasks both with knowledge and infrastructure. The AMIGA group contributes providing migrated workflows to the Grid and technical requirements.