The Infrastructure and Services of the Tier2, 2019

2019 .

The Infrastructure of    JINR-LCG2  site  comprises :

Computing farm.

CICC data-processing farm is composed of  248 64-bit  models SuperMicro Blade, SuperMicroTwin2, Dell FX.  Since   multi-core processors are independent processors on a single chip, we have in total: 

 for  248   WNs:
 4128 cores
 55488.92 HEP-SPEC06
 13872.23 HEP-kSI2k

 

The Infrastructure of    JINR-LCG2  site  comprises :

Computing farm.

CICC data-processing farm is composed of  248 64-bit  models SuperMicro Blade, SuperMicroTwin2, Dell FX.  Since   multi-core processors are independent processors on a single chip, we have in total: 

 for  248   WNs:
 4128 cores
 55488.92 HEP-SPEC06
 13872.23 HEP-kSI2k

Interactive cluster.

Five 64-bit PCs with an interactive user acces are provided for own software development and for other tasks.

 CVMFS (/cvmfs)  2 machines: 2x70TB h/w RAID1.

The CVMFS  is used to deploy large software packages of collaborations working in the WLCG.  At present, we already have versions of the NICA, BM@N, MPD software (/ cvmfs / nica, / cvmfs / ...) and occupy  140TB (before may 2019 : 9.5GB)

OSG HT-CONDOR

The computing element OSG HT-CONDOR has been integrated in the Tier-2 centre infrastructure This provides a way for VO STAR to process data using our Tier-2 with an over 90% first pass efficiency.

WLCG grid–environment for the VOs.

To service the WLCG site at JINR (the site is a separate cluster in the distributed environment of the WLCG) and other international collaborations, 22 servers with the gLite system (WLCG intermediate level software) are installed. In addition to the support functions of the JINR-LCG2 site itself, some of the servers implement important services and support functions for the Russian segment of the WLCG project.

Virtual organizations have been added to the JINR Tier2 :

ILC (WLCG http://www.linearcollider.org/),

MPD (JINR NICA),

BM @ N (JINR NICA),

COMPASS (WLCG CERN).

For VO JUNO, several services will be installed and configured at Tier2:

  • CE JUNO – will be allowed to run tasks at  the JINR  Tier2 farm
  • VOMS server, a mirror of the main VOMS in China;
  • CVMSF stratum-1 server, to support access to the JUNO software repositories in China.

In recent years, a number of works have been carried out at JINR LIT to apply grid technologies for processing of non-LHC experiments data: COMPASS, NICA MPD.
A data processing management system for the COMPASS experiment on SPS has been developed and is in operation. It provides automatic data processing and management on different types of computing resources . Currently, the computing resources available to the collaboration are

the CERN Batch Service,

the JINR computing centre and

the Frontera supercomputer of the Texas supercomputer centre.

The DIRAC Interware was used to enable the integration of distributed heterogeneous computing and storage resources for NICA MPD experiment simulation. More than 50,000 of Monte-Carlo simulation jobs of the NICA MPD experiment were performed at JINR Tier1 and Tier2 using DIRAC.

The standard WLCG program stack such as

• 2 x CREAM,
• 4 x ARGUS,
• BDII top, BDII site,
• APEL parsers, APEL publisher,
• EMI-UI, 220 x EMI-WN + gLExec-wn,
• 4 x FTS3,
• LFC,
• WMS,
• L & B,
• glite-proxyrenewal

is used for data processing.

Software :

  • OS: Scientific Linux release 6 x86_64.
  • BATCH: Torque 4.2.10 (home made)
  • Maui 3.3.2 (home made)
  • CMS Phedex
  • ALICE VObox
  • dCache-3.2
  • EOS aquamarine
  • cvmfs
  • UMI-4

  • WLCG grid–environment for the VOs.

Storage Systems.

  • dCache

dCache is the main software and harware system used for big data storage in JINR CICC.  We support the following dCache settings:

1st disk for 2 virtual organizations LHC CMS and ATLAS (Typically Supermicro and DELL): 2PB (2789TB)

58  disk servers  ( 2 x CPU 24GB RAM, 24 SATA h / w RAID6, 43-63TB2 x CPU (Xeon E5-2660 v3 @ 2.60GHz); 128GB of RAM; 76TB ZFS (16x6000GB NL SAS); 2x10G )

2nd disk  for EGI Vos & local users:   147TB
  • EOS
 EOS is intended for storing and accessing large amounts of information, including for distributed collective data generation, storing raw data of installations, data conversion and analysis. Currently our EOS has a capacity of 4PB. It is common for Tier1 and Tier2. The BM@n experiment uses our EOS for storing raw data ~ 81TB, and the MPD experiment uses ~ 84GB  for simulation data, for  ALICE  712TB.
  • XRootD (40GB).
XRootD software framework is a fully generic suite for fast, low latency and scalable data access, which can serve natively any kind of data, organized as a hierarchical filesystem-like namespace, based on the concept of directory. As a general rule, particular emphasis has been put in the quality of the core software parts. At the CICC, the XrootD is intended for the VO PANDA.
  • AFS (Andrew File System).
The AFS (Andrew File System) Service provides networked file storage for  users, in particular home directories and project spaces The CICC has 7 AFS servers installed. The total space of the AFS is ~12.5 TB.
  • NFS
 CICC has 5 NFS servers installed. Total NFS space is ~ 11 TB

Storage Software:

  • dCache-5.2
  • Enstore 4.2.2 for tape robot
  • CVMFS
  • EOS aquamarine
  • UMI-4
  • XROOTD 3 —>UMI-4 (for ALICE)
  • WLCG —> EOS(for ALICE)
  • ALICE VObox
  • CMS Phedex

Monitoring.

To ensure a correct and reliable operation of the computer complex equipment and provide relevant timely information about its work, a special monitoring system has been designed . This system allows one to keep in check climate control and power supply systems, the local network equipment, telecommunication links and computing nodes, running jobs, disk and tape storage systems. An operation centre has been launched to allow real-time monitoring of the whole JINR grid infrastructure. The number of nodes included in monitoring is more than 1,200. To ensure such extensive monitoring, a cluster monitoring system based on the Icinga2 software is used. Visualization is done with Grafana and NagVis.

CICC compises several servers for JINR users and services: 
batch, 

WWW, 

mySQL DB and Oracle; 

e-mail; D

NS , 

Nagios monitoring and other.
 These servers operate mainly with 64-bit Xeon and Opteron on board.

 

Network and telecommunication channels.

One of the most important components of JINR and MICC providing access to resources and the possibility to work with the big data is a network infrastructure.

Local Area Nertwork (LAN)  10Gbps, planned upgrade to 100Gbps
Wide  Area Network  (WAN)  100Gbps, 2x10Gbps,
                            upgrade WAN to 2x100Gbps planned

nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn

2019 .

The Infrastructure of    JINR-LCG2  site  comprises :

Computing farm.

CICC data-processing farm is composed of  248 64-bit  models SuperMicro Blade, SuperMicroTwin2, Dell FX.  Since   multi-core processors are independent processors on a single chip, we have in total:

 

CE:
 for  248   WNs:
 4128 cores
 55488.92 HEP-SPEC06
 13872.23 HEP-kSI2k 

Storage Systems:
dCache 
 1st disk for 2 virtual organizations LHC CMS and ATLAS (Typically Supermicro and DELL): 2PB (2789TB)
  58  disk servers  ( 2 x CPU 24GB RAM, 24 SATA h / w RAID6, 43-63TB2 x CPU (Xeon E5-2660 v3 @ 2.60GHz); 128GB of RAM; 76TB ZFS (16x6000GB NL SAS); 2x10G )
  2nd disk  for EGI Vos & local users:   147TB

EOS 4PB (NICA), EOS ALICE 712 TB

AFS  12.5 TB

NFS  11 TB

XRootD (40GB) PANDA

Interactive cluster.

Five 64-bit PCs with an interactive user acces are provided for own software development and for other tasks.

 CVMFS (/cvmfs)  2 machines: 2x70TB h/w RAID1.

The CVMFS  is used to deploy large software packages of collaborations working in the WLCG.  At present, we already have versions of the NICA, BM@N, MPD software (/ cvmfs / nica, / cvmfs / ...) and occupy  140TB (before may 2019 : 9.5GB)

OSG HT-CONDOR

The computing element OSG HT-CONDOR has been integrated in the Tier-2 centre infrastructure This provides a way for VO STAR to process data using our Tier-2 with an over 90% first pass efficiency.

WLCG grid–environment for the VOs.

To service the WLCG site at JINR (the site is a separate cluster in the distributed environment of the WLCG) and other international collaborations, 22 servers with the gLite system (WLCG intermediate level software) are installed. In addition to the support functions of the JINR-LCG2 site itself, some of the servers implement important services and support functions for the Russian segment of the WLCG project.

Virtual organizations have been added to the JINR Tier2 :

ILC (WLCG http://www.linearcollider.org/),

MPD (JINR NICA),

BM @ N (JINR NICA),

COMPASS (WLCG CERN).

For VO JUNO, several services will be installed and configured at Tier2:

  • CE JUNO – will be allowed to run tasks at  the JINR  Tier2 farm
  • VOMS server, a mirror of the main VOMS in China;
  • CVMSF stratum-1 server, to support access to the JUNO software repositories in China.

In recent years, a number of works have been carried out at JINR LIT to apply grid technologies for processing of non-LHC experiments data: COMPASS, NICA MPD.
A data processing management system for the COMPASS experiment on SPS has been developed and is in operation. It provides automatic data processing and management on different types of computing resources . Currently, the computing resources available to the collaboration are

the CERN Batch Service,

the JINR computing centre and

the Frontera supercomputer of the Texas supercomputer centre.

The DIRAC Interware was used to enable the integration of distributed heterogeneous computing and storage resources for NICA MPD experiment simulation. More than 50,000 of Monte-Carlo simulation jobs of the NICA MPD experiment were performed at JINR Tier1 and Tier2 using DIRAC.

The standard WLCG program stack such as

• 2 x CREAM,
• 4 x ARGUS,
• BDII top, BDII site,
• APEL parsers, APEL publisher,
• EMI-UI, 220 x EMI-WN + gLExec-wn,
• 4 x FTS3,
• LFC,
• WMS,
• L & B,
• glite-proxyrenewal

is used for data processing.

Software :

  • OS: Scientific Linux release 6 x86_64.
  • BATCH: Torque 4.2.10 (home made)
  • Maui 3.3.2 (home made)
  • CMS Phedex
  • ALICE VObox
  • dCache-3.2
  • EOS aquamarine
  • cvmfs
  • UMI-4

  • WLCG grid–environment for the VOs.

Storage Systems.

  • dCache

dCache is the main software and harware system used for big data storage in JINR CICC.  We support the following dCache settings:

1st disk for 2 virtual organizations LHC CMS and ATLAS (Typically Supermicro and DELL): 2PB (2789TB)

58  disk servers  ( 2 x CPU 24GB RAM, 24 SATA h / w RAID6, 43-63TB2 x CPU (Xeon E5-2660 v3 @ 2.60GHz); 128GB of RAM; 76TB ZFS (16x6000GB NL SAS); 2x10G )

2nd disk  for EGI Vos & local users:   147TB
  • EOS
 EOS is intended for storing and accessing large amounts of information, including for distributed collective data generation, storing raw data of installations, data conversion and analysis. Currently our EOS has a capacity of 4PB. It is common for Tier1 and Tier2. The BM@n experiment uses our EOS for storing raw data ~ 81TB, and the MPD experiment uses ~ 84GB  for simulation data, for  ALICE  712TB.
  • XRootD (40GB).
XRootD software framework is a fully generic suite for fast, low latency and scalable data access, which can serve natively any kind of data, organized as a hierarchical filesystem-like namespace, based on the concept of directory. As a general rule, particular emphasis has been put in the quality of the core software parts. At the CICC, the XrootD is intended for the VO PANDA.
  • AFS (Andrew File System).
The AFS (Andrew File System) Service provides networked file storage for  users, in particular home directories and project spaces The CICC has 7 AFS servers installed. The total space of the AFS is ~12.5 TB.
  • NFS
 CICC has 5 NFS servers installed. Total NFS space is ~ 11 TB

Storage Software:

  • dCache-5.2
  • Enstore 4.2.2 for tape robot
  • CVMFS
  • EOS aquamarine
  • UMI-4
  • XROOTD 3 —>UMI-4 (for ALICE)
  • WLCG —> EOS(for ALICE)
  • ALICE VObox
  • CMS Phedex

Monitoring.

To ensure a correct and reliable operation of the computer complex equipment and provide relevant timely information about its work, a special monitoring system has been designed . This system allows one to keep in check climate control and power supply systems, the local network equipment, telecommunication links and computing nodes, running jobs, disk and tape storage systems. An operation centre has been launched to allow real-time monitoring of the whole JINR grid infrastructure. The number of nodes included in monitoring is more than 1,200. To ensure such extensive monitoring, a cluster monitoring system based on the Icinga2 software is used. Visualization is done with Grafana and NagVis.

CICC compises several servers for JINR users and services: 
batch, 

WWW, 

mySQL DB and Oracle; 

e-mail; D

NS , 

Nagios monitoring and other.
 These servers operate mainly with 64-bit Xeon and Opteron on board.

 

Network and telecommunication channels.

One of the most important components of JINR and MICC providing access to resources and the possibility to work with the big data is a network infrastructure.

Local Area Nertwork (LAN)  10Gbps, planned upgrade to 100Gbps
Wide  Area Network  (WAN)  100Gbps, 2x10Gbps,
                            upgrade WAN to 2x100Gbps planned

 

 The Infrastructure and Services of the Tier2, 2018

 The Infrastructure and Services of the Tier2, 2017