5.1. Distributed computing: basic concepts
Page contributors
Michel Villanueva
Learning Objectives
Understand why FCC simulation and analysis need computing resources distributed over many sites
Know the difference between a local batch system and the grid
Learn the vocabulary used in the rest of this chapter: Virtual Organization, certificate, proxy, job, Storage Element, logical file name, replica, …
This page is written for readers who have never used grid computing in High Energy Physics (HEP). Its purpose is to explain the concepts that the following pages take for granted. Experienced users of DIRAC, Rucio or familiar with the WLCG can skip straight to Getting started with FCC distributed computing.
5.1.1. Why distributed computing?
A full detector simulation of a single FCC-ee event takes seconds of CPU time. Physics studies need billions of events for every process and every detector concept, and the resulting files add up to tens of petabytes. As FCC consolidates its physics case and detector designs, the computing needs will grow even further. No single computing center is expected to provide that alone.
The High Energy Physics community solved this problem with the Worldwide LHC Computing Grid (WLCG): a federation of computing centers, from CERN itself to national laboratories and university clusters, which agree to run each other’s jobs and store each other’s data using common interfaces and a common mechanism to identify users. “The grid” is the common name for this distributed computing system.
At this moment, FCC does not have its own grid. It uses a share of the existing WLCG.
The grid is different from your local resources in different ways:
Nothing is local. Your program runs on a machine you have never logged into. Everything it needs (software, input data, configuration) has to be brought there, and everything it produces has to be stored somewhere.
Authentication is centrally handled. A remote site has no account for you. Instead, sites trust the FCC Virtual Organization (VO). To use the grid, you need to become a member of the VO. All the certificate and proxy machinery described below exists to make this trust chain work.
5.1.1.1. The grid and local batch systems
A batch system is a scheduler that queues jobs and runs them on a farm of machines inside one computing center.
For example, CERN runs HTCondor on its farm, reachable from lxplus. Many universities run HTCondor or Slurm. In batch system
you already have an account, the shared filesystem (AFS, EOS) is mounted on the worker nodes, and you can ssh in to look at things.
The grid glues many batch systems together behind a single interface. In exchange for scale, there is no shared filesystem across sites, you cannot log in, and you need a grid identity rather than a local account.
5.1.1.2. DIRAC
DIRAC is the Interware that provides a single interface to the grid. It is a software framework that sits between users and the grid, hiding the complexity of the underlying batch and storage systems.
It provides a uniform way to submit jobs, manage data, and monitor the status of tasks across multiple sites. Read more about DIRAC at diracgrid.org.
Next pages will show how to use DIRAC to submit jobs and manage data for FCC simulation and analysis.
5.1.2. Design principles
5.1.2.1. Compute vs Storage
A core design principle for distributed computing in HEP is the separation of compute vs storage.
Jobs are transient entities that simply consume and produce data. They run on worker nodes, and once done they disappear. No information is preserved in the long term.
In contrast, data has persistence, replication policies, and provenance — it “lives” long after a job finishes.
Following this logic, elements on the grid are divided into two categories:
Building block |
What it is |
|---|---|
Computing Element (CE) |
The site’s front door for jobs. DIRAC talks to the CE; the CE hands work to the site’s local batch system. |
Storage Element (SE) |
A disk or tape service where files live. Each SE has a name inside DIRAC and speaks one or more transfer protocols. |
5.1.2.2. Networking
Another core component of the grid is the network. Without networking, there is no distributed computing.
Sites are connected to each other through a combination of public internet and dedicated regional research networks (e.g. GÉANT, ESnet, Internet2).
CERN (as the Tier-0 center) is connected to the Tier-1 data centers around the world on a dedicated, private network called the LHC Optical Private Network (LHCOPN). It relies on dedicated long-distance optical-fiber links (10 to 100 gigabits per second), spanning oceans and continents.
Multiple federations across the Europe, Asia Pacific and the Americas are interconnected with LHC Open Network Environment (LHCONE). Unlike LHCOPN, LHCONE is not a set of dedicated physical links, but a virtual network that provides high-bandwidth, low-latency connectivity between sites, while also ensuring security and reliability without requiring new dedicated infrastructure.
Without going deep into the details, the idea is to illustrate the importance of networking in distributed computing.
Note
At the time of writing, FCC does not have its own dedicated network. It uses the WLCG infrastructure, sharing it the with the LHC and other HEP experiments.
5.1.2.3. Software distribution
The CERN VM File System (CVMFS) is a read-only filesystem that distributes software to worker nodes. It is mounted on every WLCG site, and it is a very efficient way to make software available to jobs that run on remote sites.
CVMFS is what makes it possible for a job to source /cvmfs/sw.hsf.org/key4hep/setup.sh on any worker node in the world and find exactly
the same software you tested on a local resource like lxplus.
Read more about CVMFS in the documentation.
5.1.4. Other names you will hear
Rucio is a scientific data management system, developed by ATLAS and now used by CMS, Belle II, DUNE and many others. It plays the role of Data Management System and File Catalog, with a rich rule-based model: you declare how many replicas a dataset should have and where, and Rucio takes care. FCC is considering the adoption of Rucio for its data management, but currently uses DIRAC’s own File Catalog.
FTS (File Transfer Service) is the service that performs bulk third-party transfers between Storage Elements. DIRAC and Rucio both hand large replication requests to FTS.
Key Points
The grid uses batch systems from many sites behind one interface; the price is that nothing is local or interactive.
Trust flows through a chain: a CA certifies you, a VO vouches for you, a proxy carries that identity to wherever your job runs. Proxies expire after about a day.
A job ships small files in sandboxes and stores large results on Storage Elements.
Files have one logical name (LFN) and possibly many physical replicas; the File Catalog connects them.