justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 253101.131@dunegpschedd02.fnal.gov

Jobsub ID253101.131@dunegpschedd02.fnal.gov
Workflow ID10796
Stage ID1
User namelwhite86@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit3600 (1 hours)
Submitted time2025-11-26 14:22:36
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-11-26 17:45:03
From worker nodeHostnamedunegli-7735362-0-fnpc18003.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-11-26 14:40:01
Input filesfardet-hd:nu_dune10kt_1x2x6_1109_736_20230827T003349Z_gen_g4_detsim_hitreco__20240229T163938Z_reco2.root
JobscriptExit code0
Real time4m (284s)
CPU time4m (269s = 94%)
Max RSS bytes1293168640 (1233 MiB)
Outputting started2025-11-26 14:44:45
Output files
Finished2025-11-26 17:45:03
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

ssing the 34th record. run: 1109 subRun: 1 event: 73634 at 26-Nov-2025 14:41:57 UTC
Begin processing the 35th record. run: 1109 subRun: 1 event: 73635 at 26-Nov-2025 14:41:59 UTC
Begin processing the 36th record. run: 1109 subRun: 1 event: 73636 at 26-Nov-2025 14:42:01 UTC
Begin processing the 37th record. run: 1109 subRun: 1 event: 73637 at 26-Nov-2025 14:42:03 UTC
Begin processing the 38th record. run: 1109 subRun: 1 event: 73638 at 26-Nov-2025 14:42:07 UTC
Begin processing the 39th record. run: 1109 subRun: 1 event: 73639 at 26-Nov-2025 14:42:09 UTC
Begin processing the 40th record. run: 1109 subRun: 1 event: 73640 at 26-Nov-2025 14:42:10 UTC
Begin processing the 41st record. run: 1109 subRun: 1 event: 73641 at 26-Nov-2025 14:42:12 UTC
Begin processing the 42nd record. run: 1109 subRun: 1 event: 73642 at 26-Nov-2025 14:42:15 UTC
PfoList empty... moving on.
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0097, LArDLTrackCharacterisation, STATUS_CODE_FAILURE
Begin processing the 43rd record. run: 1109 subRun: 1 event: 73643 at 26-Nov-2025 14:42:16 UTC
Begin processing the 44th record. run: 1109 subRun: 1 event: 73644 at 26-Nov-2025 14:42:19 UTC
Begin processing the 45th record. run: 1109 subRun: 1 event: 73645 at 26-Nov-2025 14:42:22 UTC
Begin processing the 46th record. run: 1109 subRun: 1 event: 73646 at 26-Nov-2025 14:42:24 UTC
Begin processing the 47th record. run: 1109 subRun: 1 event: 73647 at 26-Nov-2025 14:42:26 UTC
Begin processing the 48th record. run: 1109 subRun: 1 event: 73648 at 26-Nov-2025 14:42:28 UTC
Begin processing the 49th record. run: 1109 subRun: 1 event: 73649 at 26-Nov-2025 14:42:30 UTC
Begin processing the 50th record. run: 1109 subRun: 1 event: 73650 at 26-Nov-2025 14:42:32 UTC
Begin processing the 51st record. run: 1109 subRun: 1 event: 73651 at 26-Nov-2025 14:42:34 UTC
Begin processing the 52nd record. run: 1109 subRun: 1 event: 73652 at 26-Nov-2025 14:42:36 UTC
Begin processing the 53rd record. run: 1109 subRun: 1 event: 73653 at 26-Nov-2025 14:43:02 UTC
Begin processing the 54th record. run: 1109 subRun: 1 event: 73654 at 26-Nov-2025 14:43:04 UTC
Begin processing the 55th record. run: 1109 subRun: 1 event: 73655 at 26-Nov-2025 14:43:06 UTC
Begin processing the 56th record. run: 1109 subRun: 1 event: 73656 at 26-Nov-2025 14:43:08 UTC
Begin processing the 57th record. run: 1109 subRun: 1 event: 73657 at 26-Nov-2025 14:43:10 UTC
Begin processing the 58th record. run: 1109 subRun: 1 event: 73658 at 26-Nov-2025 14:43:12 UTC
Begin processing the 59th record. run: 1109 subRun: 1 event: 73659 at 26-Nov-2025 14:43:14 UTC
Begin processing the 60th record. run: 1109 subRun: 1 event: 73660 at 26-Nov-2025 14:43:16 UTC
Begin processing the 61st record. run: 1109 subRun: 1 event: 73661 at 26-Nov-2025 14:43:17 UTC
Begin processing the 62nd record. run: 1109 subRun: 1 event: 73662 at 26-Nov-2025 14:43:19 UTC
Begin processing the 63rd record. run: 1109 subRun: 1 event: 73663 at 26-Nov-2025 14:43:21 UTC
PandoraContentApi::GetList(*this, m_trackPfoListName, pTrackPfoList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetAllTrackFeatures
    in file:     /exp/dune/app/users/lwhite86/DUNE-FD/pandoraPID/srcs/larpandoracontent/larpandoradlcontent/LArTrackShowerId/DlTrackCharacterisationAlgorithm.cc line#: 221
Begin processing the 64th record. run: 1109 subRun: 1 event: 73664 at 26-Nov-2025 14:43:23 UTC
Begin processing the 65th record. run: 1109 subRun: 1 event: 73665 at 26-Nov-2025 14:43:25 UTC
Begin processing the 66th record. run: 1109 subRun: 1 event: 73666 at 26-Nov-2025 14:43:27 UTC
Begin processing the 67th record. run: 1109 subRun: 1 event: 73667 at 26-Nov-2025 14:43:31 UTC
Begin processing the 68th record. run: 1109 subRun: 1 event: 73668 at 26-Nov-2025 14:43:32 UTC
Begin processing the 69th record. run: 1109 subRun: 1 event: 73669 at 26-Nov-2025 14:43:34 UTC
Begin processing the 70th record. run: 1109 subRun: 1 event: 73670 at 26-Nov-2025 14:43:35 UTC
Begin processing the 71st record. run: 1109 subRun: 1 event: 73671 at 26-Nov-2025 14:43:37 UTC
Begin processing the 72nd record. run: 1109 subRun: 1 event: 73672 at 26-Nov-2025 14:43:39 UTC
Begin processing the 73rd record. run: 1109 subRun: 1 event: 73673 at 26-Nov-2025 14:43:52 UTC
Begin processing the 74th record. run: 1109 subRun: 1 event: 73674 at 26-Nov-2025 14:43:54 UTC
Begin processing the 75th record. run: 1109 subRun: 1 event: 73675 at 26-Nov-2025 14:43:56 UTC
Begin processing the 76th record. run: 1109 subRun: 1 event: 73676 at 26-Nov-2025 14:43:57 UTC
Begin processing the 77th record. run: 1109 subRun: 1 event: 73677 at 26-Nov-2025 14:43:59 UTC
Begin processing the 78th record. run: 1109 subRun: 1 event: 73678 at 26-Nov-2025 14:44:00 UTC
Begin processing the 79th record. run: 1109 subRun: 1 event: 73679 at 26-Nov-2025 14:44:04 UTC
Begin processing the 80th record. run: 1109 subRun: 1 event: 73680 at 26-Nov-2025 14:44:06 UTC
Begin processing the 81st record. run: 1109 subRun: 1 event: 73681 at 26-Nov-2025 14:44:07 UTC
Begin processing the 82nd record. run: 1109 subRun: 1 event: 73682 at 26-Nov-2025 14:44:09 UTC
Begin processing the 83rd record. run: 1109 subRun: 1 event: 73683 at 26-Nov-2025 14:44:11 UTC
Begin processing the 84th record. run: 1109 subRun: 1 event: 73684 at 26-Nov-2025 14:44:14 UTC
Begin processing the 85th record. run: 1109 subRun: 1 event: 73685 at 26-Nov-2025 14:44:16 UTC
Begin processing the 86th record. run: 1109 subRun: 1 event: 73686 at 26-Nov-2025 14:44:17 UTC
Begin processing the 87th record. run: 1109 subRun: 1 event: 73687 at 26-Nov-2025 14:44:19 UTC
Begin processing the 88th record. run: 1109 subRun: 1 event: 73688 at 26-Nov-2025 14:44:20 UTC
Begin processing the 89th record. run: 1109 subRun: 1 event: 73689 at 26-Nov-2025 14:44:22 UTC
Begin processing the 90th record. run: 1109 subRun: 1 event: 73690 at 26-Nov-2025 14:44:24 UTC
Begin processing the 91st record. run: 1109 subRun: 1 event: 73691 at 26-Nov-2025 14:44:26 UTC
Begin processing the 92nd record. run: 1109 subRun: 1 event: 73692 at 26-Nov-2025 14:44:28 UTC
Begin processing the 93rd record. run: 1109 subRun: 1 event: 73693 at 26-Nov-2025 14:44:29 UTC
Begin processing the 94th record. run: 1109 subRun: 1 event: 73694 at 26-Nov-2025 14:44:31 UTC
Begin processing the 95th record. run: 1109 subRun: 1 event: 73695 at 26-Nov-2025 14:44:33 UTC
Begin processing the 96th record. run: 1109 subRun: 1 event: 73696 at 26-Nov-2025 14:44:35 UTC
Begin processing the 97th record. run: 1109 subRun: 1 event: 73697 at 26-Nov-2025 14:44:37 UTC
Begin processing the 98th record. run: 1109 subRun: 1 event: 73698 at 26-Nov-2025 14:44:39 UTC
Begin processing the 99th record. run: 1109 subRun: 1 event: 73699 at 26-Nov-2025 14:44:41 UTC
Begin processing the 100th record. run: 1109 subRun: 1 event: 73700 at 26-Nov-2025 14:44:42 UTC
26-Nov-2025 14:44:44 UTC  Closed output file "nu_dune10kt_1x2x6_1109_736_20230827T003349Z_gen_g4_detsim_hitreco__20240229T163938Z_reco2_reco2.root"
26-Nov-2025 14:44:44 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/32/00/nu_dune10kt_1x2x6_1109_736_20230827T003349Z_gen_g4_detsim_hitreco__20240229T163938Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      1.36375       2.25016       24.9487       1.79995       2.58448        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00179452     0.019381      0.437099      0.0133786     0.0441787       100    
reco:pandora2:StandardPandora                   1.35133       2.23003       24.9307       1.78792       2.58477        100    
[art]:TriggerResults:TriggerResultInserter     1.063e-05    2.05435e-05   8.2296e-05    1.8871e-05    9.46298e-06      100    
end_path:out1:RootOutput                       2.405e-06    3.58731e-06   1.9116e-05     3.121e-06    1.89389e-06      100    
end_path:out1:RootOutput(write)               0.000174851   0.000541453   0.00259098    0.000284067   0.000574436      100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2278.19 MB
  Peak resident set size usage (VmHWM): 1293.17 MB
====================================================================================================
Art has completed and will exit with status 0.
lar exit code 0
total 2232
-rw-r--r-- 1 dunegli fnalgrid     210 Nov 26 14:40 all-input-dids.txt
-rw-r--r-- 1 dunegli fnalgrid       0 Nov 26 14:40 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid   42633 Nov 26 14:44 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid     184 Nov 26 14:44 justin-processed-pfns.txt
drwxr-xr-x 4 dunegli fnalgrid      60 Nov 26 14:40 larpandoracontent
-rw-r--r-- 1 dunegli fnalgrid  354964 Nov 26 14:44 nu_dune10kt_1x2x6_1109_736_20230827T003349Z_gen_g4_detsim_hitreco__20240229T163938Z_reco2_reco2.root
-rw-r--r-- 1 dunegli fnalgrid     519 Nov 26 14:44 reco2_hist.root
-rw-r--r-- 1 dunegli fnalgrid 1869067 Nov 26 14:44 trainingFile_nu_dune10kt_1x2x6_1109_736_20230827T003349Z_gen_g4_detsim_hitreco__20240229T163938Z_reco2.root
justIN time: 2026-02-08 07:16:18 UTC       justIN version: 01.06.00