justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 292862.1@dunegpschedd02.fnal.gov

Jobsub ID292862.1@dunegpschedd02.fnal.gov
Workflow ID12170
Stage ID1
User namelwhite86@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit3600 (1 hours)
Submitted time2026-01-21 15:06:52
SiteUS_UChicago
EntryEngage_US_MWT2_iut2_gk02_condce_mcore
Last heartbeat2026-01-21 15:17:44
From worker nodeHostnamecit2-c029.mwt2.org
cpuinfoIntel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-21 15:07:42
Input filesfardet-hd:nutau_dune10kt_1x2x6_1424_851_20230827T144455Z_gen_g4_detsim_hitreco__20240219T231840Z_reco2.root
JobscriptExit code0
Real time9m (586s)
CPU time8m (532s = 90%)
Max RSS bytes1539137536 (1467 MiB)
Outputting started2026-01-21 15:17:29
Output files
Finished2026-01-21 15:17:44
Saved logsjustin-logs:292862.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0th record. run: 1424 subRun: 1 event: 85170 at 21-Jan-2026 10:15:29 EST
TrackPID: -211, 0.176511, 0.507741, 0.30684, 0.00890736
TrackPID: 211, 0.0206536, 0.607175, 0.361093, 0.011079
TrackPID: 2212, 0.00028295, 0.00459995, 0.994286, 0.000830687
TrackPID: 211, 0.0211127, 0.767422, 0.200539, 0.010926
TrackPID: 2212, 0.00139901, 0.0189086, 0.977928, 0.00176415
TrackPID: -211, 0.0390765, 0.379452, 0.574834, 0.00663741
Begin processing the 71st record. run: 1424 subRun: 1 event: 85171 at 21-Jan-2026 10:15:33 EST
Begin processing the 72nd record. run: 1424 subRun: 1 event: 85172 at 21-Jan-2026 10:15:36 EST
TrackPID: 2212, 0.000328988, 0.00536963, 0.993404, 0.000897578
TrackPID: 211, 0.00277881, 0.019128, 0.974189, 0.00390475
TrackPID: 22, 0.0411628, 0.547049, 0.405771, 0.00601711
Begin processing the 73rd record. run: 1424 subRun: 1 event: 85173 at 21-Jan-2026 10:15:39 EST
TrackPID: -211, 0.0398451, 0.395231, 0.558153, 0.00677055
TrackPID: 2212, 0.0043728, 0.0244127, 0.96675, 0.00446433
Begin processing the 74th record. run: 1424 subRun: 1 event: 85174 at 21-Jan-2026 10:15:43 EST
PfoList empty... moving on.
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0097, LArDLTrackCharacterisation, STATUS_CODE_FAILURE
Begin processing the 75th record. run: 1424 subRun: 1 event: 85175 at 21-Jan-2026 10:15:47 EST
TrackPID: 2212, 0.0013676, 0.0379865, 0.959215, 0.00143129
TrackPID: 211, 0.0240786, 0.905925, 0.0631635, 0.00683321
TrackPID: 2212, 0.00444993, 0.081818, 0.909137, 0.00459566
TrackPID: 211, 0.024046, 0.736195, 0.231636, 0.00812262
Begin processing the 76th record. run: 1424 subRun: 1 event: 85176 at 21-Jan-2026 10:15:51 EST
PandoraContentApi::GetList(*this, m_trackPfoListName, pTrackPfoList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetAllTrackFeatures
    in file:     /exp/dune/app/users/lwhite86/DUNE-FD/pandoraPID/srcs/larpandoracontent/larpandoradlcontent/LArTrackShowerId/DlTrackCharacterisationAlgorithm.cc line#: 223
Begin processing the 77th record. run: 1424 subRun: 1 event: 85177 at 21-Jan-2026 10:15:54 EST
Begin processing the 78th record. run: 1424 subRun: 1 event: 85178 at 21-Jan-2026 10:15:57 EST
TrackPID: 22, 0.000215958, 0.00248704, 0.996433, 0.000863386
TrackPID: -211, 0.266245, 0.442417, 0.280498, 0.0108393
TrackPID: 22, 0.157975, 0.628527, 0.206054, 0.00744398
TrackPID: -211, 0.152915, 0.574736, 0.264268, 0.00808117
Begin processing the 79th record. run: 1424 subRun: 1 event: 85179 at 21-Jan-2026 10:16:03 EST
TrackPID: 2212, 0.000424757, 0.00111684, 0.99772, 0.000738188
Begin processing the 80th record. run: 1424 subRun: 1 event: 85180 at 21-Jan-2026 10:16:06 EST
TrackPID: 2212, 0.00407435, 0.0908193, 0.903006, 0.00210069
TrackPID: 2212, 9.45697e-05, 0.00291359, 0.996576, 0.000415978
Begin processing the 81st record. run: 1424 subRun: 1 event: 85181 at 21-Jan-2026 10:16:09 EST
PfoList empty... moving on.
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0097, LArDLTrackCharacterisation, STATUS_CODE_FAILURE
Begin processing the 82nd record. run: 1424 subRun: 1 event: 85182 at 21-Jan-2026 10:16:16 EST
PandoraContentApi::GetList(*this, m_trackPfoListName, pTrackPfoList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetAllTrackFeatures
    in file:     /exp/dune/app/users/lwhite86/DUNE-FD/pandoraPID/srcs/larpandoracontent/larpandoradlcontent/LArTrackShowerId/DlTrackCharacterisationAlgorithm.cc line#: 223
Begin processing the 83rd record. run: 1424 subRun: 1 event: 85183 at 21-Jan-2026 10:16:19 EST
TrackPID: -211, 0.106612, 0.72837, 0.159465, 0.00555294
Begin processing the 84th record. run: 1424 subRun: 1 event: 85184 at 21-Jan-2026 10:16:22 EST
TrackPID: 2212, 0.0152674, 0.281953, 0.698026, 0.00475376
TrackPID: 13, 0.998507, 0.0012569, 0.000190373, 4.56092e-05
Begin processing the 85th record. run: 1424 subRun: 1 event: 85185 at 21-Jan-2026 10:16:26 EST
TrackPID: 2212, 0.000260887, 0.0075916, 0.990885, 0.0012623
Begin processing the 86th record. run: 1424 subRun: 1 event: 85186 at 21-Jan-2026 10:16:29 EST
TrackPID: 2212, 1.87598e-05, 0.00415971, 0.994659, 0.00116231
TrackPID: 211, 0.0640372, 0.780035, 0.13217, 0.0237581
TrackPID: -211, 0.119763, 0.740518, 0.134411, 0.00530742
Begin processing the 87th record. run: 1424 subRun: 1 event: 85187 at 21-Jan-2026 10:16:33 EST
TrackPID: 2212, 0.662111, 0.263189, 0.0663964, 0.00830317
TrackPID: 2212, 0.00022939, 0.00101282, 0.998233, 0.000525143
TrackPID: 2212, 0.00254685, 0.222728, 0.765894, 0.00883141
TrackPID: -211, 0.242236, 0.619879, 0.117624, 0.0202607
Begin processing the 88th record. run: 1424 subRun: 1 event: 85188 at 21-Jan-2026 10:16:40 EST
Begin processing the 89th record. run: 1424 subRun: 1 event: 85189 at 21-Jan-2026 10:16:43 EST
Begin processing the 90th record. run: 1424 subRun: 1 event: 85190 at 21-Jan-2026 10:16:46 EST
Begin processing the 91st record. run: 1424 subRun: 1 event: 85191 at 21-Jan-2026 10:16:49 EST
TrackPID: 211, 0.0131113, 0.637585, 0.339345, 0.00995856
TrackPID: 211, 0.0964346, 0.746941, 0.151045, 0.0055792
TrackPID: 2212, 0.0136519, 0.562241, 0.41572, 0.00838714
TrackPID: 13, 0.953866, 0.0387071, 0.00678093, 0.000645476
TrackPID: 211, 0.370544, 0.530443, 0.0682824, 0.0307309
TrackPID: 2212, 0.00299652, 0.0640261, 0.93009, 0.00288687
TrackPID: 211, 0.131839, 0.693232, 0.169794, 0.00513483
Begin processing the 92nd record. run: 1424 subRun: 1 event: 85192 at 21-Jan-2026 10:16:54 EST
TrackPID: 2212, 6.46106e-05, 0.00276976, 0.996666, 0.000499252
TrackPID: -211, 0.523444, 0.421965, 0.0454529, 0.00913821
TrackPID: -211, 0.0221651, 0.81072, 0.159381, 0.00773415
TrackPID: 211, 0.0843478, 0.895678, 0.0181562, 0.00181829
Begin processing the 93rd record. run: 1424 subRun: 1 event: 85193 at 21-Jan-2026 10:16:58 EST
Begin processing the 94th record. run: 1424 subRun: 1 event: 85194 at 21-Jan-2026 10:17:01 EST
Begin processing the 95th record. run: 1424 subRun: 1 event: 85195 at 21-Jan-2026 10:17:04 EST
Begin processing the 96th record. run: 1424 subRun: 1 event: 85196 at 21-Jan-2026 10:17:07 EST
TrackPID: 2212, 0.00216024, 0.166784, 0.814095, 0.0169603
TrackPID: -211, 0.0136489, 0.120083, 0.85997, 0.00629801
Begin processing the 97th record. run: 1424 subRun: 1 event: 85197 at 21-Jan-2026 10:17:13 EST
TrackPID: 2212, 0.000442885, 0.005068, 0.992932, 0.00155689
Begin processing the 98th record. run: 1424 subRun: 1 event: 85198 at 21-Jan-2026 10:17:16 EST
TrackPID: -211, 0.0219802, 0.712657, 0.257042, 0.00832044
TrackPID: -211, 0.0141095, 0.253036, 0.729511, 0.00334402
Begin processing the 99th record. run: 1424 subRun: 1 event: 85199 at 21-Jan-2026 10:17:19 EST
TrackPID: -211, 0.148073, 0.822355, 0.0270527, 0.00251919
Begin processing the 100th record. run: 1424 subRun: 1 event: 85200 at 21-Jan-2026 10:17:23 EST
21-Jan-2026 10:17:27 EST  Closed output file "nutau_dune10kt_1x2x6_1424_851_20230827T144455Z_gen_g4_detsim_hitreco__20240219T231840Z_reco2_reco2.root"
21-Jan-2026 10:17:27 EST  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/ff/0d/nutau_dune10kt_1x2x6_1424_851_20230827T144455Z_gen_g4_detsim_hitreco__20240219T231840Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.0819239      4.53161       24.0997       3.1528        4.03308        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0114361     0.0306761     0.0572854     0.0292466     0.010373        100    
reco:pandora2:StandardPandora                  0.0688708      4.4998        24.0695       3.12744       4.03225        100    
[art]:TriggerResults:TriggerResultInserter    1.9825e-05    3.83322e-05   0.000115932   3.6489e-05    1.55592e-05      100    
end_path:out1:RootOutput                       3.382e-06    6.32333e-06    2.598e-05    5.0225e-06    3.06412e-06      100    
end_path:out1:RootOutput(write)               0.000221459   0.000803114   0.00613032    0.000588131   0.000781584      100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2521.43 MB
  Peak resident set size usage (VmHWM): 1539.14 MB
====================================================================================================
Art has completed and will exit with status 0.
lar exit code 0
total 428
-rw-r--r--. 1 dune osgvo    216 Jan 21 10:07 all-input-dids.txt
-rw-r--r--. 1 dune osgvo      0 Jan 21 10:08 debugprod.log
-rw-r--r--. 1 dune osgvo  60747 Jan 21 10:17 jobscript.log
-rw-r--r--. 1 dune osgvo    187 Jan 21 10:17 justin-processed-pfns.txt
drwxr-xr-x. 4 dune osgvo   4096 Jan 21 10:07 larpandoracontent
-rw-r--r--. 1 dune osgvo 351355 Jan 21 10:17 nutau_dune10kt_1x2x6_1424_851_20230827T144455Z_gen_g4_detsim_hitreco__20240219T231840Z_reco2_reco2.root
-rw-r--r--. 1 dune osgvo    519 Jan 21 10:17 reco2_hist.root
justIN time: 2026-02-05 12:45:48 UTC       justIN version: 01.06.00