5. Distributed Computing

Page contributors

Gerardo Ganis, Juraj Smiesko, Michel Villanueva

This chapter covers the tools used to run FCC simulation and analysis at scale: EventProducer and DIRAC.

The first section is a primer on the basic concepts of distributed computing, written for readers who have never used grid computing in HEP. It explains the vocabulary (Virtual Organization, job, sandbox, storage element logical file name, etc.) that the following pages use. If you already familiar with the grid, you can skip it.

In addition, this chapter documents workflows based on the DIRAC Interware system through iLCDirac, the extension developed by the Linear Collider community and also used by CALICE. These tutorials provide concrete examples of running distributed workflows on grid resources, including job submission, production management, and output retrieval.

The setup procedure needed to use the FCC resources through DIRAC is described first. Additional information about the use of iLCDirac can be found in the CLIC and ILC dedicated Wiki pages.

Unless specified otherwise, in the remainder of this section the word DIRAC refers to the iLCDirac extension mentioned above.

The chapter closes with the EventProducer framework. It provides a lightweight and reproducible interface to define, submit, monitor, and manage centrally produced event samples. It integrates the different stages of the production chain from generator-level event creation to detector simulation and output handling.

Contents:

• 5.1. Distributed computing: basic concepts
  • 5.1.1. Why distributed computing?
  • 5.1.2. Design principles
  • 5.1.3. Identity and authorization
  • 5.1.4. Other names you will hear
• 5.2. Getting started with FCC distributed computing
  • 5.2.1. Registering to the FCC VO
  • 5.2.2. Enabling DIRAC
  • 5.2.3. Copying, browsing, accessing files
  • 5.2.4. The web portal
• 5.3. FCC DIRAC Example Workflows
  • 5.3.1. Overview of the job submission script
  • 5.3.2. Workflow 1: Ditau with KKMCee + Delphes
• 5.4. Structure for /eos/experiment/fcc/prod
  • 5.4.1. Overview
• 5.5. EventProducer
  • 5.5.1. Clone and initialisation
  • 5.5.2. Generate LHE files from gripacks
  • 5.5.3. Generate LHE files directly from MG5
  • 5.5.4. Generate STDHEP files directly from Whizard + Pythia6
  • 5.5.5. Generate EDM4hep files from the LHE and decay with Pythia8
  • 5.5.6. Generate EDM4hep files from STDHEP
  • 5.5.7. Generate EDM4hep files from Pythia8
  • 5.5.8. Expert mode
  • 5.5.9. Produce samples with EventProducer outside of official campaign