justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 223125.186@dunegpschedd02.fnal.gov

Jobsub ID223125.186@dunegpschedd02.fnal.gov
Workflow ID8481
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2025-10-03 05:34:57
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce513
Last heartbeat2025-10-03 10:45:37
From worker nodeHostnameb9p10p3098.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-10-03 05:45:39
Input filesvd-protodune:np02vd_raw_run039812_0313_df-s04-d0_dw_0_20251003T011226.hdf5
JobscriptExit code0
Real time4h (16768s)
CPU time4h (16296s = 97%)
Max RSS bytes5872357376 (5600 MiB)
Outputting started2025-10-03 10:25:08
Output filesvd-protodune-det-reco:np02vd_raw_run039812_0313_df-s04-d0_dw_0_20251003T011226_reco_stage1_20251003T102455_keepup.root
Finished2025-10-03 10:45:37
Saved logsjustin-logs:223125.186-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

[ timer  ] Timer: WireCell::Aux::Resampler : 0.29 sec
[12:19:59.675] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.26 sec
[12:19:59.675] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.24 sec
[12:19:59.675] I [ timer  ] Timer: WireCell::Aux::Resampler : 0.21 sec
[12:19:59.675] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.03 sec
[12:19:59.675] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.02 sec
[12:19:59.675] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[12:19:59.675] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[12:19:59.675] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[12:19:59.675] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[12:19:59.675] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[12:19:59.675] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:19:59.675] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:19:59.675] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[12:19:59.675] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:19:59.675] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[12:19:59.675] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[12:19:59.675] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[12:19:59.675] I [ timer  ] Timer: Total node execution : 87.45000225678086 sec
wclsFrameSaver saving cooked to 10000 ticks
wclsFrameSaver: saving 74327 traces tagged "gauss"
FrameSaver: q=1.94743e+07 n=1872471 tag=gauss
wclsFrameSaver: saving 98939 traces tagged "wiener"
FrameSaver: q=2.13094e+07 n=1770015 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 8816 XUs and 5201 XVs -> 2612 XUVs
C:0 T:1 24729 XUs and 18864 XVs -> 3508 XUVs
C:0 T:2 7759 XUs and 4559 XVs -> 1390 XUVs
C:0 T:3 2784 XUs and 2404 XVs -> 382 XUVs
C:0 T:4 3810 XUs and 2391 XVs -> 806 XUVs
C:0 T:5 383 XUs and 320 XVs -> 19 XUVs
C:0 T:6 423 XUs and 264 XVs -> 46 XUVs
C:0 T:7 634 XUs and 910 XVs -> 49 XUVs
C:0 T:8 105243 XUs and 277054 XVs -> 39094 XUVs
C:0 T:9 21757 XUs and 41750 XVs -> 4792 XUVs
C:0 T:10 18584 XUs and 46791 XVs -> 6137 XUVs
C:0 T:11 1668 XUs and 4744 XVs -> 470 XUVs
C:0 T:12 10427 XUs and 17362 XVs -> 2320 XUVs
C:0 T:13 1065 XUs and 1074 XVs -> 123 XUVs
C:0 T:14 3137 XUs and 2986 XVs -> 112 XUVs
C:0 T:15 8332 XUs and 10029 XVs -> 1893 XUVs
63753 XUVs total
6391 collection wire objects
63753 potential space points
Neighbour search...
10137327 tests to find 4067786 neighbours
Iterating with no regularization...
Begin: 5.79933e+09
0 5.37922e+09
1 5.34795e+09
2 5.34412e+09
Now with regularization...
Begin: 5.28999e+09
0 5.28787e+09
this->SetupTestBeamPfo(pNuPfo, pTestBeamPfo, testBeamStartVertex) return STATUS_CODE_NOT_FOUND
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_16_00-buildFW/src/larpandoracontent/LArThreeDReco/LArEventBuilding/TestBeamParticleCreationAlgorithm.cc line#: 43
iter->second->Run() throw STATUS_CODE_NOT_FOUND
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0079, LArTestBeamParticleCreation, STATUS_CODE_NOT_FOUND
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
03-Oct-2025 12:24:55 CEST  Closed output file "np02vd_raw_run039812_0313_df-s04-d0_dw_0_20251003T011226_reco_stage1_20251003T102455_keepup.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        108.864       795.863       8905.67       216.106       1849.37        21     
----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                      6.7521e-05    0.000116468   0.000206402   0.000117771   2.86815e-05      21     
produce:tpcrawdecoder:PDVDTPCReader               5.56684       7.71785       21.4006       6.34348       3.54212        21     
produce:triggerrawdecoder:PDVDTriggerReader4    0.00670768     0.106886       1.96234      0.0107091     0.414988        21     
produce:pdvddaphne:DAPHNEReaderPDVD               2.22549       2.85688       4.08406       2.84972       0.38664        21     
produce:ophit:OpHitFinder                        0.0351021     0.0489234     0.0602745     0.0506955    0.00628332       21     
produce:opflash:OpFlashFinderVerticalDrift      0.00712603     0.0134305     0.0201685     0.013439     0.00321766       21     
produce:wclsdatavd:WireCellToolkit                56.8952       85.9616       144.431       85.092        25.0285        21     
produce:gaushit:GausHitFinder                    0.980874       2.0711        3.70955       1.87049       0.77836        21     
produce:nhitsfilter:NumberOfHitsFilter          0.000244682   0.000448843   0.000924958   0.000388033   0.000169209      21     
produce:reco3d:SpacePointSolver                   7.63375       28.4779       67.4016       17.5787       19.5813        21     
produce:hitpdune:DisambigFromSpacePoints         0.138539      0.606983       1.78321      0.309632      0.492621        21     
produce:pandora:StandardPandora                   24.7393       659.401       8742.97       103.703       1837.11        21     
produce:pandoraTrack:LArPandoraTrackCreation     0.580009       1.94525       5.64565       1.50221       1.29409        21     
produce:pandoraGnocalo:GnocchiCalorimetry        0.013559      0.0440989     0.0882147     0.0390825     0.0213887       21     
[art]:TriggerResults:TriggerResultInserter       1.787e-05    3.16778e-05   9.0921e-05     2.866e-05    1.48011e-05      21     
end_path:out1:RootOutput                         7.57e-06     1.10324e-05    3.733e-05     9.53e-06     6.00842e-06      21     
end_path:out1:RootOutput(write)                   4.60382       6.54971       8.2151        6.51502      0.842757        21     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7825.22 MB
  Peak resident set size usage (VmHWM): 5872.36 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np02vd_raw_run039812_0313_df-s04-d0_dw_0_20251003T011226_reco_stage1_20251003T102455_keepup.root
\tHists: np02vd_raw_run039812_0313_df-s04-d0_dw_0_20251003T011226_reco_stage1_20251003T102455_keepup_hists.root
Forming reco metadata
Successfully opened file np02vd_raw_run039812_0313_df-s04-d0_dw_0_20251003T011226_reco_stage1_20251003T102455_keepup.root
Ran successfully
{
  "name": "np02vd_raw_run039812_0313_df-s04-d0_dw_0_20251003T011226_reco_stage1_20251003T102455_keepup.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "artroot",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "full-reconstructed",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1759487096.0,
    "core.end_time": 1759487096.0,
    "core.events": [
      53196,
      53204,
      53212,
      53220,
      53228,
      53236,
      53244,
      53252,
      53260,
      53268,
      53276,
      53284,
      53292,
      53300,
      53308,
      53316,
      53324,
      53332,
      53340,
      53348,
      53356
    ],
    "core.event_count": 21,
    "core.first_event_number": 53196,
    "core.last_event_number": 53356,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39812
    ],
    "core.runs_subruns": [
      3981200001
    ],
    "dune.daq_test": true,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039812_0313_df-s04-d0_dw_0_20251003T011226.hdf5"
    }
  ]
}Forming hist metadata
formed
{
  "name": "np02vd_raw_run039812_0313_df-s04-d0_dw_0_20251003T011226_reco_stage1_20251003T102455_keepup_hists.root",
  "namespace": "vd-protodune-det-reco",
  "metadata": {
    "core.file_format": "root",
    "core.application.name": "reco",
    "core.application.family": "dunesw",
    "core.application.version": "v10_10_00d00",
    "core.data_tier": "root-tuple-virtual",
    "dune.config_file": "standard_reco_stage1_protodunevd_keepup_all.fcl",
    "dune.campaign": "vd-protodune-reco-keepup-v0",
    "core.start_time": 1759487096.0,
    "core.end_time": 1759487096.0,
    "core.data_stream": "cosmics",
    "core.file_content_status": "good",
    "core.file_type": "detector",
    "core.run_type": "vd-protodune",
    "core.runs": [
      39812
    ],
    "core.runs_subruns": [
      3981200001
    ],
    "dune.daq_test": true,
    "retention.status": "active",
    "retention.class": "physics"
  },
  "parents": [
    {
      "did": "vd-protodune:np02vd_raw_run039812_0313_df-s04-d0_dw_0_20251003T011226.hdf5"
    }
  ]
}
justIN time: 2025-11-04 07:52:28 UTC       justIN version: 01.05.01