Skip to content

From Power On of DAQ Server

Jump to bottom
Tom Eichlersmith edited this page Dec 14, 2025 · 14 revisions

Running notes on what needs to happen to go from power-on to collecting data.

Establish Slow Control

Update Firmware?

conda activate ldmx-env
sudo $(which python) /u1/eichl008/ldmx-firmware/firmware/submodules/axi-pcie-core/scripts/updatePcieFpga --dev /dev/datadev_1 --path /u1/eichl008/ecal-hcal-firmware/

Changing firmware puts you here as if the DAQ Server just powered on. Current firmware (12/07/2025):

eichl008@rdsrv436:/u1/eichl008/mpod_control$ cat /proc/datadev_2
PCIe[BUS:NUM:SLOT.FUNC] : 0000:46:00.0
---------- Firmware Axi Version -----------
     Firmware Version : 0xffffffff
           ScratchPad : 0x0
        Up Time Count : 82579
             Git Hash : a81c1dd073104e5bffc79b02178ab7b8019f5f80
            DNA Value : 0x00000000400200000170e6432cf0c105
         Build String : Esa25EcalBittware: Vivado v2024.1, ldmxlab-Precision-3660 (Ubuntu 22.04.5 LTS), Built Fri Dec  5 11:36:01 AM CST 2025 by jmmans

(same output for datadev_1)

If `cat /proc/datadev_*` fails...

Then the driver is not loaded.

cd /u1/aes-stream-driver/datadev/driver
sudo ./load

Make Sure Clocks are Correct

The Ecal/Hcal Firmware for the Bittware needs to be connected to the Fast Control block so that its BX_CLK doesn't walk around. The OPTO.FULLSTATUS output (in pftool or pflpgbt) should show

  BX_CLK               : 37.143 MHz (0x9117)

This number is very stable and so if its different in any of the leading three digits something is wrong. You can connect the Ecal/Hcal Firmware with the FC block by running gui-less-pedestals.py from ldmx-firmware (currently on umn_dev branch in software/scripts).

Warning

We've seen ICEC failures even when BX_CLK is correct that are only resolved by updating the FcMini configuration like how the gui-less-pedestals.py script does it, so it may be worth it to just run it after a new firmware no matter what.

Details on What gui-less-pedestals.py Does

This file just uses the rogue device tree to enable the correct parameters and then attempts to collect a single event of pedestals. This output event at this stage will be junk, but the parameters being set will be stored on the Bittware and we are all good.

Maybe in the future we can move this setting of the parameters into pflib/pftool somehow.

# subsystem is either root.EcalSubsystem or root.HcalSubsystem
subsystem.Bittware.FcReceiver.FcMini.enable.set(True)
subsystem.Bittware.FcReceiver.FcMini.FcMiniEnable.set(True)
subsystem.Bittware.FcReceiver.FcMini.FcTxLogic.FcState.setDisp('IDLE')

Once the clocks are all set, you can move on to optical link setup.

Optical Link Set Up

See the full wiki page for the details, but in short:

  1. Power off front-end (or disconnect fiber)
  2. Run OPTO.RESET (on all links -- Hcal only has one link, but Ecal has three)
  3. Check OPTO.FULLSTATUS (on all links -- Hcal only has one link, but Ecal has three)
    • should see BUFFBYPASS_DONE is 1
    • clocks look good except LINK_ERROR and LINK_WORD are both zero
  4. Re-power front-end (or re-connect fiber)
  5. Check OPTO.FULLSTATUS (on all links -- Hcal only has one link, but Ecal has three)
    • should see READY is 1
    • LINK_WORD matching BX_CLK
    • LINK_ERROR is still zero

A Good OPTO.FULLSTATUS

Polarity -- TX: 0  RX: 0
Optical status:
  BUFFBYPASS_DONE      : 0x0001 # should be 1, if not need a RESET with frontend off
  BUFFBYPASS_ERROR     : 0x0000 # should be 0, if not need a RESET with frontend off
  CDR_STABLE           : 0x0001
  READY                : 0x0001 # should be 1, if not frontend needs to be powered on
  RX_RESETDONE         : 0x0001
  TX_RESETDONE         : 0x0001
Optical rates:
  BX_CLK               : 37.143 MHz (0x9117)   # also very stable, go back to "FcMini" settings if bad
  LCLS2_CLK            : 185.718 MHz (0x2d576) # very stable, should be this number
  LINK_ERROR           : 0.000 MHz (0x0000)    # should be zero, if not maybe misconnected
  LINK_WORD            : 37.143 MHz (0x9117)   # should be BX_CLK
  REF_CLK              : 297.149 MHz (0x488bd)
  RX-LINK              : 297.149 MHz (0x488bd)
  S_AXI_ACLK           : 125.000 MHz (0x1e848)
  TX_CLK               : 297.149 MHz (0x488bd)

Configure lpGBTs

This is done in pftool when the HcalBackplane or EcalSMM target is being constructed; however, as previously observe, it may fail. If you see the "failure to apply configuration" warning when opening pftool, go into OPTO and double check that FULLSTATUS looks good. If it does, just close and re-open pftool once or twice more.

Test Talk to Boards

In pftool, ECON.STATUS does many I2C transactions showing that slow control with the ECON is functional. You can do ROC.PAGE to look at parameters of the ROC and ROC.RUNMODE to test if you can set parameters.

Align Different Chips

  • make a hard reset on the ECON
  • put ECON and ROC that you want to align into run mode
  • load an econd_init file onto the ECON
    • Note: The init file you LOAD needs to have the channels that correspond to the ROC that is plugged in enabled in two places ERX.N_ENABLE and ROCDAQCTRL.GLOBAL_ACTIVE_ERXS (bit mask of enabled channels) as well as some other parameters that are in the config/econ examples.
  • TASKS.PHASE_WORD_ALIGN with chosen ROC, ECON
    • This aligns the ROC and the ECON
    • should see Successful header match in Snapshot:
  • TASKS.ALIGN_ECON_LPGBT with chosen ECON
    • This aligns the ECON and the lpGBT and assumes the init file you loaded earlier sets FORMATTERBUFFER.GLOBAL_IDLE_PATTERN = 0x1277cc
    • Should do bit alignment (see LOCKED) and then see Found alignment at N

Notes with Example Printouts: https://github.com/LDMX-Software/pflib/wiki/Bittware%E2%80%90lpGBT-Optical-Connection-Setup#configuring-the-frontend

Configuring to Prep for DAQ

DAQ preparation requires slow control access to both the ECON and ROC, and the different chips on the boards to be well aligned so we can capture the data be read out.

  • DAQ.SETUP.STATUS
    • should see ECON0.QUICKSPY is equal to the idle word shifted by a byte
  • DAQ.SETUP.FORMAT
    • make sure that pftool uses the ECOND format
  • DAQ.SETUP.CONFIG
    • set number of samples per ROR
    • Note: capturing data without AXIS seems to be limited to one sample per ROR. This is fine since real DAQ is going to use AXIS and non-AXIS readout is for pftool-tasks which can (for now) be single-sample

Internal ROR and pftool pedestals

  • DAQ.SETUP.ENABLE
    • if no prompts, then it was just disabling the DAQ, rerun
    • Enable external/central L1A? No -- we are going to do internal L1A for pftool pedestals
    • Enable readout to AXIS? No -- we are going to do internal readout
  • DAQ.PEDESTAL
    • Run number? not persisted so doesn't matter
    • How many events? choose number of RoRs to send
    • Readout rate? (Hz) 100 seems like its working
    • Filename choose output file
    • Should we decode? No -- live decoding is not operational as of 12/07/2025 https://github.com/LDMX-Software/pflib/pull/306

External ROR and Rogue GUI Pedestals

  • DAQ.SETUP.ENABLE
    • if no prompts, then it was just disabling the DAQ, rerun
    • Enable external/central L1A? Yes -- we are going to do receive L1As from Run Control
    • Enable readout to AXIS? Yes -- we are going to do send data to Rogue via AXIS

Outside GUI

You still need to "take" the system (I think) in order to avoid conflicting with others using the Fast Control hub, but the gui-less-pedestals.py script in slaclab/ldmx-firmware is an example of how to go through the run control motions without having to wait for the X server transactions.

cd ldmx-firmware/software/scripts
python gui-less-pedestals.py hcal
  • You can also use the --solo argument if you are at a test stand.
  • Open to updates to this script for more pedestals, but its main purpose to to confirm that DAQ is succeeding via AXIS

In GUI

# re-connect to SLAC DAQ server with X11 forwarding (-XY option)
# needs to be run from this directory
cd ldmx-firmware/software/scripts
# `--solo` is to specify to use /dev/datadev_0 like at UMN/UVA test stands
# here at SLAC DAQ server, I do not use `--solo`
python3 Esa25Gui.py --no-tracker --no-ts --no-tdaq --no-hcal --no-generic

Under HcalSubsystem->Bittware

  • FcReceiver "Read" (far right)
    • updates FcReceiver->LinkUp and FcReceiver->LinkNum (should be True and 80) by reading them from the Bittware
  • FcReceiver->FcMini->enable True
  • FcReceiver->FcMini->FcMiniEnable True
  • FcReceiver->FcMini->SyntheticTrigger->enable True
    • allow for us to send synthetic triggers
  • FcReceiver->FcMini->SyntheticTrigger->EnableRoR False
    • This seems to send triggers at a set frequency following the other SyntheticTrigger paramters. I have no idea how this works, so I'm just using the UsrRoR.

Under RunControl

  • "Exec" next to "NextRcState" seven times
    • cycles through Connect -> Configure -> TriggerAlign -> RorLatency -> PreStart -> OpenFile -> Go
    • all of these should succeed
    • this auto-names the output file to /u1/ldmx/data/esa25/Run_NNN_date_time.dat which I don't know how to change but we should probably change when doing these Ecal-only or Hcal-only runs
  • FcReceiver->FcMini->Synthetictrigger->UsrRoR Exec
    • should send exactly one ROR
    • as of 0900 CDT 12/07/2025, not seeing data output, expecting misconfiguration since I could see a EXT_ROR_UNMASKED increment when sending the UsrRoR

With Other Subsystems

  • FcReceiver->FcMini->FcMiniEnable False

Clone this wiki locally