justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 292861.2@dunegpschedd02.fnal.gov

Jobsub ID292861.2@dunegpschedd02.fnal.gov
Workflow ID12170
Stage ID1
User namelwhite86@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit3600 (1 hours)
Submitted time2026-01-21 15:04:52
SiteUK_QMUL
EntryDUNE_UK_London_QMUL_arcce03
Last heartbeat2026-01-21 15:14:06
From worker nodeHostnamecn008.htc.esc.qmul
cpuinfoIntel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2026-01-21 15:05:35
Input filesfardet-hd:nu_dune10kt_1x2x6_1413_336_20230826T183327Z_gen_g4_detsim_hitreco__20240109T222531Z_reco2.root
JobscriptExit code0
Real time6m (361s)
CPU time5m (334s = 92%)
Max RSS bytes1588596736 (1515 MiB)
Outputting started2026-01-21 15:11:37
Output files
Finished2026-01-21 15:14:06
Saved logsjustin-logs:292861.2-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

77 at 21-Jan-2026 15:10:39 GMT
TrackPID: 2212, 0.0384994, 0.107572, 0.84823, 0.00569805
TrackPID: 2212, 0.000295895, 0.00157357, 0.997406, 0.000724116
TrackPID: -211, 0.0243902, 0.348988, 0.620595, 0.00602645
Begin processing the 78th record. run: 1413 subRun: 1 event: 33678 at 21-Jan-2026 15:10:41 GMT
PfoList empty... moving on.
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0097, LArDLTrackCharacterisation, STATUS_CODE_FAILURE
Begin processing the 79th record. run: 1413 subRun: 1 event: 33679 at 21-Jan-2026 15:10:42 GMT
TrackPID: 2212, 0.000109162, 0.00138347, 0.998055, 0.000452473
TrackPID: -211, 0.303104, 0.499187, 0.18829, 0.00941872
TrackPID: -211, 0.0559894, 0.486477, 0.450836, 0.0066976
TrackPID: 211, 0.0426301, 0.664507, 0.281225, 0.0116374
TrackPID: 2212, 0.00185282, 0.0323545, 0.963792, 0.00200075
TrackPID: -211, 0.0228117, 0.904349, 0.0680824, 0.00475719
Begin processing the 80th record. run: 1413 subRun: 1 event: 33680 at 21-Jan-2026 15:10:45 GMT
TrackPID: 2212, 0.0023235, 0.0633455, 0.932058, 0.00227262
TrackPID: 2212, 0.0147042, 0.64737, 0.323584, 0.0143412
TrackPID: 13, 0.292514, 0.366218, 0.333518, 0.0077495
TrackPID: 2212, 0.0536224, 0.712833, 0.20724, 0.0263045
TrackPID: 211, 0.00596282, 0.468565, 0.517483, 0.00798897
TrackPID: 2212, 0.000176322, 0.0107402, 0.988023, 0.00106098
Begin processing the 81st record. run: 1413 subRun: 1 event: 33681 at 21-Jan-2026 15:10:47 GMT
TrackPID: 2212, 7.10531e-05, 0.00108803, 0.998374, 0.000467118
TrackPID: 211, 0.00132745, 0.0191806, 0.97667, 0.00282217
TrackPID: 2212, 0.000313743, 0.0628654, 0.931924, 0.00489685
TrackPID: 2212, 0.000162196, 0.0018497, 0.997273, 0.000714736
Begin processing the 82nd record. run: 1413 subRun: 1 event: 33682 at 21-Jan-2026 15:10:48 GMT
PfoList empty... moving on.
iter->second->Run() throw STATUS_CODE_FAILURE
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0097, LArDLTrackCharacterisation, STATUS_CODE_FAILURE
Begin processing the 83rd record. run: 1413 subRun: 1 event: 33683 at 21-Jan-2026 15:10:50 GMT
TrackPID: 13, 0.0274148, 0.238832, 0.726882, 0.0068719
TrackPID: 211, 0.191638, 0.494765, 0.297385, 0.0162128
TrackPID: 22, 0.0478223, 0.225298, 0.720119, 0.00676082
TrackPID: 22, 0.16654, 0.661834, 0.166484, 0.00514235
TrackPID: 211, 0.0293699, 0.79995, 0.150591, 0.0200895
Begin processing the 84th record. run: 1413 subRun: 1 event: 33684 at 21-Jan-2026 15:10:53 GMT
TrackPID: 13, 0.727831, 0.160391, 0.107032, 0.00474617
TrackPID: 2212, 0.0105195, 0.171221, 0.815057, 0.00320201
Begin processing the 85th record. run: 1413 subRun: 1 event: 33685 at 21-Jan-2026 15:10:55 GMT
TrackPID: 13, 0.0293334, 0.245943, 0.718092, 0.00663138
TrackPID: 2212, 0.000170928, 0.00328065, 0.995992, 0.000556182
Begin processing the 86th record. run: 1413 subRun: 1 event: 33686 at 21-Jan-2026 15:10:57 GMT
TrackPID: 2212, 0.000993667, 0.0259049, 0.971897, 0.00120457
TrackPID: -211, 0.0569194, 0.577509, 0.359105, 0.0064667
TrackPID: -211, 0.146008, 0.803008, 0.0472853, 0.00369832
TrackPID: -211, 0.0313189, 0.783176, 0.176834, 0.00867125
Begin processing the 87th record. run: 1413 subRun: 1 event: 33687 at 21-Jan-2026 15:10:59 GMT
TrackPID: 2212, 0.00078871, 0.00446186, 0.99343, 0.00131961
TrackPID: -211, 0.0151143, 0.696598, 0.271724, 0.0165639
TrackPID: -211, 0.144967, 0.669235, 0.179218, 0.00657993
TrackPID: 13, 0.999964, 2.84007e-05, 4.97413e-06, 2.36838e-06
TrackPID: 22, 0.087915, 0.764192, 0.142581, 0.00531143
Begin processing the 88th record. run: 1413 subRun: 1 event: 33688 at 21-Jan-2026 15:11:02 GMT
TrackPID: 2212, 0.00167723, 0.0122997, 0.983936, 0.00208699
TrackPID: 13, 0.276585, 0.707077, 0.0138071, 0.00253122
Begin processing the 89th record. run: 1413 subRun: 1 event: 33689 at 21-Jan-2026 15:11:04 GMT
TrackPID: -211, 0.0292044, 0.337398, 0.626382, 0.00701566
TrackPID: 2212, 0.00136705, 0.0195825, 0.976163, 0.00288756
TrackPID: 2212, 0.014078, 0.277879, 0.703232, 0.00481143
Begin processing the 90th record. run: 1413 subRun: 1 event: 33690 at 21-Jan-2026 15:11:06 GMT
TrackPID: 211, 0.106708, 0.747901, 0.14082, 0.00457083
TrackPID: 13, 0.121298, 0.732278, 0.14034, 0.00608484
TrackPID: 13, 0.712814, 0.247752, 0.0345407, 0.00489349
TrackPID: 2212, 0.021947, 0.180718, 0.788799, 0.00853623
TrackPID: 211, 0.43102, 0.43993, 0.108283, 0.0207665
TrackPID: 2212, 0.000268585, 0.00429217, 0.994704, 0.00073523
Begin processing the 91st record. run: 1413 subRun: 1 event: 33691 at 21-Jan-2026 15:11:11 GMT
TrackPID: 2212, 0.0761618, 0.516857, 0.400254, 0.0067272
TrackPID: 2212, 0.000109056, 0.0012763, 0.998183, 0.000431473
Begin processing the 92nd record. run: 1413 subRun: 1 event: 33692 at 21-Jan-2026 15:11:12 GMT
TrackPID: 2212, 0.00516247, 0.126656, 0.864043, 0.00413825
TrackPID: 211, 0.12587, 0.592661, 0.266164, 0.0153061
TrackPID: 211, 0.138577, 0.688352, 0.159427, 0.0136442
TrackPID: 2212, 0.00035028, 0.0119743, 0.98655, 0.00112496
Begin processing the 93rd record. run: 1413 subRun: 1 event: 33693 at 21-Jan-2026 15:11:15 GMT
TrackPID: 13, 0.88308, 0.0779591, 0.0367581, 0.00220291
Begin processing the 94th record. run: 1413 subRun: 1 event: 33694 at 21-Jan-2026 15:11:17 GMT
TrackPID: 2212, 0.000517769, 0.0348404, 0.960116, 0.00452603
TrackPID: 211, 0.0366618, 0.729002, 0.223813, 0.0105234
TrackPID: 211, 0.107758, 0.719892, 0.167056, 0.00529456
TrackPID: 13, 0.99697, 0.00200096, 0.000961653, 6.80234e-05
TrackPID: 211, 0.129146, 0.762043, 0.103939, 0.00487215
TrackPID: 2212, 0.000915852, 0.0364124, 0.96129, 0.00138166
TrackPID: 2212, 0.0472973, 0.170797, 0.773484, 0.00842123
Begin processing the 95th record. run: 1413 subRun: 1 event: 33695 at 21-Jan-2026 15:11:20 GMT
TrackPID: 2212, 0.00378268, 0.0380952, 0.953261, 0.00486118
TrackPID: 211, 0.122095, 0.591218, 0.270444, 0.0162418
Begin processing the 96th record. run: 1413 subRun: 1 event: 33696 at 21-Jan-2026 15:11:22 GMT
TrackPID: 22, 0.0918679, 0.80442, 0.0873899, 0.0163219
Begin processing the 97th record. run: 1413 subRun: 1 event: 33697 at 21-Jan-2026 15:11:25 GMT
TrackPID: 2212, 0.00444336, 0.085518, 0.906937, 0.00310196
TrackPID: 2212, 0.000136393, 0.00254919, 0.996585, 0.000729177
TrackPID: 13, 0.884031, 0.0867084, 0.0273577, 0.00190324
TrackPID: 2212, 0.00689112, 0.45107, 0.53049, 0.0115487
Begin processing the 98th record. run: 1413 subRun: 1 event: 33698 at 21-Jan-2026 15:11:26 GMT
TrackPID: 2212, 0.000624307, 0.0202169, 0.978047, 0.00111203
TrackPID: 211, 0.1646, 0.631753, 0.196582, 0.00706521
Begin processing the 99th record. run: 1413 subRun: 1 event: 33699 at 21-Jan-2026 15:11:28 GMT
TrackPID: 2212, 0.000657206, 0.0078082, 0.989415, 0.00211968
TrackPID: -13, 0.999327, 0.000396282, 0.000238985, 3.75772e-05
Begin processing the 100th record. run: 1413 subRun: 1 event: 33700 at 21-Jan-2026 15:11:32 GMT
21-Jan-2026 15:11:34 GMT  Closed output file "nu_dune10kt_1x2x6_1413_336_20230826T183327Z_gen_g4_detsim_hitreco__20240109T222531Z_reco2_reco2.root"
21-Jan-2026 15:11:34 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/fd/9a/nu_dune10kt_1x2x6_1413_336_20230826T183327Z_gen_g4_detsim_hitreco__20240109T222531Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.0663069      2.83152       26.4764       1.99409       3.66199        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00840967     0.023486      0.062167      0.025832     0.00966402       100    
reco:pandora2:StandardPandora                  0.0552636      2.80687       26.4575       1.96969       3.66127        100    
[art]:TriggerResults:TriggerResultInserter    1.4519e-05    5.96981e-05   0.000622332   2.78685e-05   0.000113617      100    
end_path:out1:RootOutput                       2.464e-06    3.49298e-06   2.6939e-05    3.2285e-06    2.40876e-06      100    
end_path:out1:RootOutput(write)               0.000203211   0.000874519   0.00602691    0.000305326   0.00116475       100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2572.81 MB
  Peak resident set size usage (VmHWM): 1588.6 MB
====================================================================================================
Art has completed and will exit with status 0.
lar exit code 0
total 420
-rw-r--r-- 1 pildune32 pildune    210 Jan 21 15:05 all-input-dids.txt
-rw-r--r-- 1 pildune32 pildune      0 Jan 21 15:06 debugprod.log
-rw-r--r-- 1 pildune32 pildune  60135 Jan 21 15:11 jobscript.log
-rw-r--r-- 1 pildune32 pildune    168 Jan 21 15:11 justin-processed-pfns.txt
drwxr-xr-x 4 pildune32 pildune     48 Jan 21 15:05 larpandoracontent
-rw-r--r-- 1 pildune32 pildune 355028 Jan 21 15:11 nu_dune10kt_1x2x6_1413_336_20230826T183327Z_gen_g4_detsim_hitreco__20240109T222531Z_reco2_reco2.root
-rw-r--r-- 1 pildune32 pildune    519 Jan 21 15:11 reco2_hist.root
justIN time: 2026-02-05 12:44:27 UTC       justIN version: 01.06.00