Stanford ATLAS Fast TracKer, Higgs Physics and More

Dispatches from ATLAS@Stanford

Welcome to our blog! Group members occasionally post content here about what we are working on.

Zihao Awarded PD Soros Fellowship for New Americans

By Lauren Tompkins

Group member Zihao Jiang has been named a member of the 2016 PD Soros Fellows! Text of the announcement is below. Congratulations Zihao!!

Thirty Outstanding Immigrants & Children of Immigrants Each Awarded $90,000 For Graduate School By The Paul & Daisy Soros Fellowships for New Americans

The Paul & Daisy Soros Fellowships for New Americans program announces 2016 Fellows; Fellows reflect accomplishments and diversity of immigrants and refugees in the United States; Launch 2017 Application

April, 12, 2016 (New York) - Today, The Paul & Daisy Soros Fellowships for New Americans, the premier graduate school fellowship for immigrants and children of immigrants, announced their 2016 recipients. The thirty recipients, called “Fellows”, were selected for their potential to make significant contributions to US society, culture, or their academic field, and were selected from a pool of 1,443 applicants. With a two percent acceptance rate, it was the most competitive year in the Fellowship’s history.

Daisy M. Soros and Paul Soros (1926-2013) founded the Fellowship program in 1997, which has awarded more than 550 Fellowships over its 18 year history. The couple, both Hungarian immigrants, has contributed $75 million to the organization’s charitable trust.

In addition to receiving up to $90,000 in funding for the graduate program of their choice, each new Fellow joins the prestigious community of recipients from past years, which includes US Surgeon General Vivek Murthy, leading Ebola researcher Pardis Sabeti, Aspiration founder Andrei Cherny, Oscar health insurance co-founder Kevin Nazemi and over 535 other New American leaders.

“The Fellows are from all different countries and socio-economic and religious backgrounds, and they have come to the United States in a myriad of ways – but they all bring excellence to the table,” said Craig Harwood, who directs the Fellowship program. “They demonstrate that immigrants, regardless of their background, continue to be a critical part of our nation.”

The 2016 Fellows, who are 30 or younger, come from a range of socio-economic backgrounds, and are all naturalized citizens, green card holders, DACA recipients, or the children of immigrants. Their backgrounds reflect the diversity of recent immigrants and refugees in the United States. Those who were born abroad hail from Bangladesh, Burma, Canada, China, the Dominican Republic, Egypt, Germany, India, Iran, Israel, Mexico, Nigeria, Pakistan, Saudi Arabia, the United Arab Emirates, and the United Kingdom.

Here is a sampling of their stories: Veronica Manzo, who was born in California to Mexican parents in a family of farmworkers, has performed brain cancer research that was published in Nature. Goran Micevic, who grew up in war-torn Yugoslavia, was wrongly arrested and held by secret police at the age of 12, and moved to the United States to attend Iowa State University. He is now a National Cancer Institute Fellow and MD/PhD student at Yale University. Jenna Nicholas, whose parents are of Middle Eastern and European descent, is studying business at Stanford and is the founder and CEO of an impact investment firm.

Denisse Rojas Marquez, the first undocumented student to attend Icahn School of Medicine at Mount Sinai, was born in Mexico City and co-founded a Pre-Health Dreamers organization for undocumented students. Zihao Jiang’s father, who lived in poverty while working as a school teacher during the Cultural Revolution in China, could have only dreamed that his son would reach the highest levels of the United States university system. Zihao is now performing research at the Large Hadron collider at CERN as a PhD student at Stanford. Shadi Gaheri, who was one of the first women to direct campus theater at her university in Tehran, came to the United States at the age of 23 to pursue a career as a theater director; she is now at the Yale School of Theater. Suan Lian Tuang, who was born in Burma and immigrated to the US at the age 16, is pursuing an MD/PhD in chemistry at Harvard and MIT.

Data Formatter Configurations Database

 By Victor Ruelas

The FTK system is a highly parallelized processor. To be able to process the tremendous data rate in the ATLAS detector, about 1 billion proton-proton collisions per second, the trigger system filters out most uninteresting physics. From proton bunches crossing at 40 MHz, the level-1 trigger system reduces this rate to 100 kHz. FTK reconstructs tracks from data in the inner Pixel, IBL, and SCT detectors after every level-1 accepted event, and feeds the tracking information to the High Level Trigger (HLT). The ATLAS inner detector geometry is mapped to 64 spatial regions using a hardware layer of the FTK system called the Data Formatter (DF). After the mapping step, each region is processed independently. In the Data Formatter, 128 Input Mezzanine (IM) cards or lanes perform clustering. Clustering is the process of figuring out where a particle interacted with the detector. Data sharing for the overlap of neighboring regions to avoid inefficiencies due to various track curvatures and the finite size of the luminous regions of the beam in the z coordinate is also performed by the DF boards. A full mesh Advanced Telecom Computing Architecture (ATCA) backplane interconnect is implemented for the communication of DF boards within a create. There are 8 DF boards in one create. Four crates communicate using fiber links. The data sharing of all the DF boards creates a network.

Each time a DF board is inserted in a create, it needs to be configured with information such as its physical location in the crate and the DF board network, what SCT and Pixel modules are connected to it, and what information should be processed by that particular DF board or get sent to a different board for processing. There are different processes which produce the final configuration of a DF board, the whole process is depicted in the diagram shown below.

Plan for the Data Formatter (DF) Database Project. Now: Information is extracted for the Input Mezzanine (IM) cards and Data Formatter from SCT and Pixel detectors. Information for the IM consists of a map of Read-Out Buffers (ROBs) to Pixel and SCT modules. DF information is a map of SCT and Pixel modules to FTK regions. This is then converted to a text file made up of module lists which is used by two pieces of software to create DF configuration txt files and Read-Out Driver (ROD) Look Up Tables (LUT). These output files are the input for the FtkDFApi software which controls the DF. New: Maps of ROB to Pixel and SCT modules and map of modules to FTK regions is migrated to a centralized Data Formatter Database. Separate programs are rewritten to a new integrated software which accesses the database. This approach allows for better integration and a more natural design in the DF software.

The project I started, has the aim to create a Data Formatter configurations Database (DFDB). The DFDB will act as a centralized bank of configurations rather than having a myriad of individual configuration txt files. The first step in the project is to migrate the mapping information from SCT and Pixel cabling services, Root LUT in the diagram above, to the DFDB. Over the summer, while at CERN I researched the best implementation of the database and created the first prototype. Back at California State University, Fresno, I created the first working prototype of the database using the COOL API (CERN’s own relational database infrastructure) and C++.

Since no one in the group had experience in building such a database, I had to find people at CERN with experience and expertise in database frameworks. I coordinated with people in the FTK group to figure out the specific content and use cases for the database. I carried out my research by writing emails, face-to-face and remote meetings. Once I gathered the specifications and had knowledge of database frameworks, I started designing the first prototype. As I designed and created the first working prototype, I learned about FTK and the ATLAS Experiment more in depth. Through my interactions with scientists from different backgrounds, I started to build a network and realized the great deal of skills involved in undertaking such a massive scientific experiment, the ATLAS Experiment. Just in the FTK group alone, organization and distribution of work is very well managed. There are subgroups which work on a specific part of the system and have their own scheduled deliverables. Then the whole group meets constantly to discuss integration. It is all very beautiful in the way it’s orchestrated with the same goal in mind, to build a device that could potentially lead to the next big discovery in physics. I too had a role, and it was such an amazing experience to be part of this great scientific endeavor.

Data Formatter Board Production Testing

By Rex Brown

We recently finished testing 37 FPGA-based boards that will be used to augment the triggering system of the ATLAS experiment at LHC. What do these boards allow us to do, you ask? They allow us to organize a high-speed stream of data, 800 Gbps, into an easily readable and accessible format for downstream usage. ATLAS collects raw data at a rate of 1 petabyte, or 1000 terabytes, per second! It is impossible to write all of this data to disk due to this astronomical rate, so triggering (rapidly deciding which events in a particle detector should be kept) becomes very important. We want to save the most interesting events, so this need be done carefully. Currently, the job that these boards will do is done with software. This is great, except for the fact that the software can’t run quite fast enough to keep up with the flow of data. Therefore, implementing our software algorithms into hardware allows us to be able to keep up with a data rate that increases as as the LHC increases the collision rate and the experiment makes modifications. This forces us to be on the cutting-edge of technology, since the boards will need to remain current for many years.

The Data Formatter (DF) board and attached Rear Transition Module (RTM). Note the four uninhabited areas of the board where the Input Mezzanines (IMs) usually live. The large black area near the center of the board is the heat sink for the Virtex-7 FPGA.

Testing such high-performance hardware takes great care. A visual inspection upon receiving the boards rules out obvious issues such as incorrect capacitor polarity or warped circuit boards. After passing this test, the boards are labeled to prevent any confusion that arises from having 37 identical pieces of hardware. Next, we attempt to program the Field-Programmable Gate Array (FPGA) after inserting the board into our crate. FPGAs offer unparalleled flexibility in processing data exactly the way that we want to. Different firmware fundamentally alters how the FPGA operates, making it maximally customizable. At this point, we start more intense testing of the individual transceivers of the FPGA and their connections. Data Formatter (DF) boards are connected to each other within a single crate by what’s called a backplane, that allows communication speeds of up to 40 Gbps per transceiver quad, of which there are 20 total on the FPGA. Connections between the total of four crates is accomplished through QSFP cables that are made of optical fibers. This allows for reliable, high-speed communication over longer distances. One final test is a check of the communication between the DF and the Input Mezzanines (IMs) that the data comes in through. The network topology used in the Data Formatter system is somewhat beautiful. It consists of 32 boards, spread across four crates of 8 boards each, and is illustrated below in Figure 1.

The fabric interface connections in 14-slot full mesh ATCA backplane. Each line represents a multi-lane bidirectional channel rated for up to 40 Gbps. A graphical depiction of the 32 boards (in green) and high speed interconnect lines in four crate system. Blue lines represent backplane data paths. Orange lines represent inter-crate fiber links.

As you might imagine, challenges are encountered along the way. Mechanical issues and improper soldering of various components caused some Rear Transition Modules (RTMs) to be unusable. The RTM connects to the back of the DF, and provides the means to connect to either downstream, for further processing, or other crates of boards. New RTMs were being tested in parallel with new DFs, so test results often required in-depth interpretation to find what the issue was. In the end, the vast majority of the boards and RTMs were tested to be fully-functioning, and were shipped off to CERN!

The board which the DF system uses are Pulsar IIb boards, which were designed at Fermilab. More on the Pulsar IIbs can be found at ATCA@FNAL.

Summer Project Summary

by Mingyu Kang

This is the last week of my research. Here is the list of things I have done in this lab.

  • I built this website!

  • I tested the function of the first three Fast TracKer DF boards. A test consists of four procedures: 1. Visual Inspection 2. LED Blink Test 3. RTM Test 4. Backplane Test. Particularly the RTM (Rear Transition Module) test and the backplane test is done by IBERT (Integrated Bit Error Ratio Test) and Eye Scan test. All of the boards fully passed the tests, and two of them were sent to CERN.

  • I am working for opening access to a temperature sensor in the IPMC card. This is very crucial to the Fast TracKer because if a board gets overheated it can blow up the whole system. Currently, ICARE, the program we should use, does not support our sensor, so we are updating the information about the sensor to the program.

  • I am working on backplane characterization. Although the backplane in our lab functions perfectly in the rate of 8Gbps, it seems unstable in 10Gbps, the rate we should achieve. I am doing the IBERT and Eye Scan test for each link in the backplane in order to figure out which links should be fixed.