Experience with Infiniband for CMS Event Building

The CMS data acquisition system is incorporating Infiniband (IB) technology to boost event-building performance for the LHC run 2 (2015-2018). IB should provide substantial increase in data transmission speeds compared to the Gigabit Ethernet network used during the LHC run 1.

Several options exist to developers when choosing a foundation for their software, including the uDAPL (DAT Collaborative) and IB verbs libraries (OFED). The CMS DAQ group has been able to evaluate both the ibverbs and the uDAPL technologies.

The choice over which direction to go when integrating new technologies into a system is based upon the system’s requirements with respect to common software aspects, such as performance, usability, scalability and reliability. The CMS DAQ group has done a feasibility study with regard to how each approach meets these qualities.

Due to advances in technology, the CMS data acquisition system will be able to achieve the required throughput of 100 kHz with increased event sizes while downsizing the number of nodes by using a combination of 10GE, 40GE and 56 Gb Infiniband FDR.

The Infiniband technologies were integrated into the CMS Online Software XDAQ framework, permitting us to demonstrate the effective enhancement in performance for the existing CMS event builder. XDAQ is a software platform designed specifically for the development of distributed data acquisition systems. The development is carried out at CERN, the European Organization for Nuclear Research. The XDAQ framework is used by the CMS and CALICE experiments for their data acquisition and control systems.