justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258316.0@dunegpschedd02.fnal.gov

Jobsub ID258316.0@dunegpschedd02.fnal.gov
Workflow ID11029
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-06 00:50:37
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-06 10:07:46
From worker nodeHostnamedunegli-7988755-0-fnpc18006.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-06 01:41:12
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T010243Z_gen_002920_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time8h (29939s)
CPU time8h (29267s = 97%)
Max RSS bytes3046195200 (2905 MiB)
Outputting started2025-12-06 10:00:11
Output files
Finished2025-12-06 10:07:46
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0.088804, 
Output 1: 0.2103, 0.131175, 0.316648, 0.341877, 
Output 2: 0.0229963, 0.0352124, 0.938091, 0.00370072, 
Output 3: 0.929733, 0.0655708, 0.00236809, 0.00232795, 
Output 4: 0.426899, 0.443286, 0.091753, 0.0380621, 
Output 5: 0.193221, 0.417482, 0.257236, 0.132061, 
Output 6: 0.255116, 0.337049, 0.131039, 0.276795, 

Begin processing the 10th record. run: 5027 subRun: 0 event: 29200 at 06-Dec-2025 09:08:29 UTC
Boundary wire vector sizes: 6627, 6498, 6615
minwire 0: 72
minwire 1: 1
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 711, 1210
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0853513, 
Output 1: 0.0781366, 0.0708539, 0.138008, 0.713001, 
Output 2: 0.0286846, 0.133665, 0.820156, 0.017494, 
Output 3: 0.965361, 0.0177852, 0.0041233, 0.0127306, 
Output 4: 0.486196, 0.418906, 0.0808615, 0.0140361, 
Output 5: 0.656131, 0.231057, 0.0821759, 0.0306356, 
Output 6: 0.0279636, 0.0637832, 0.0694042, 0.838849, 

06-Dec-2025 09:58:19 UTC  Closed output file "matt_reco2.root"
06-Dec-2025 09:58:19 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/fc/55/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T010243Z_gen_002920_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2722.83       2954.03       3280.57       2927.75       201.096        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00326454    0.00953236     0.0199538    0.00767969    0.00517279       10     
reco:pandora:StandardPandora                            2718.79       2949.35       3276.16       2923.82       200.215        10     
reco:pandoraTrack:LArPandoraTrackCreation              0.0129665     0.0539367     0.239744      0.043763      0.0641608       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00194434     0.010215      0.0463947    0.00279852     0.0131974       10     
reco:pandoracalo:Calorimetry                          0.000526513   0.00199236    0.00889614    0.000733962   0.00257834       10     
reco:pandorapid:Chi2ParticleID                        0.000150714   0.000716641   0.00352812    0.00020177    0.00108435       10     
reco:cvnmap:CVNMapper                                  0.0520255     0.0778913     0.217042      0.0603215     0.0472528       10     
reco:cvneva:CVNEvaluator                                2.26145       3.64383       8.20058       3.17733       1.57881        10     
reco:energyrecnumu:EnergyReco                         0.00755948     0.0947975     0.325618      0.0741522     0.0970853       10     
reco:energyrecnue:EnergyReco                          0.00223624    0.00413165     0.0161169    0.00269273    0.00401858       10     
reco:energyrecnc:EnergyReco                            0.0152035     0.0922146     0.334271      0.0600288     0.096783        10     
[art]:TriggerResults:TriggerResultInserter            3.0597e-05    5.57949e-05   0.000214254   3.92185e-05   5.31284e-05      10     
end_path:out1:RootOutput                              1.1262e-05    1.74571e-05   6.2187e-05    1.2449e-05    1.49436e-05      10     
end_path:out1:RootOutput(write)                        0.609264       0.68955      0.923651      0.632443      0.0995308       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21744 MB
  Peak resident set size usage (VmHWM): 3046.2 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
Completed MCParticle Loop
True Neutrino Energy: 28.9012
 Size of all hits: 44
hit initial : 38.3124 | hit correction : 1.02643
hit initial : 137.798 | hit correction : 1.0234
hit initial : 377.833 | hit correction : 1.02344
hit initial : 320.597 | hit correction : 1.02356
hit initial : 50.4306 | hit correction : 1.02363
hit initial : 124.224 | hit correction : 1.02426
hit initial : 148.522 | hit correction : 1.02431
hit initial : 55.0978 | hit correction : 1.02482
hit initial : 69.583 | hit correction : 1.02491
hit initial : 122.658 | hit correction : 1.02559
hit initial : 112.179 | hit correction : 1.02579
hit initial : 270.683 | hit correction : 1.02622
hit initial : 48.6002 | hit correction : 1.0263
hit initial : 65.4339 | hit correction : 1.02626
hit initial : 187.496 | hit correction : 1.02635
 *** Total Charge : 2182.08
 Looking at view: 0 | Number of hits in view: 15
hit initial : 27.6366 | hit correction : 1.02629
hit initial : 62.9623 | hit correction : 1.02336
hit initial : 174.737 | hit correction : 1.02337
hit initial : 217.808 | hit correction : 1.02342
hit initial : 197.882 | hit correction : 1.02351
hit initial : 192.981 | hit correction : 1.02361
hit initial : 87.0973 | hit correction : 1.0237
hit initial : 32.5433 | hit correction : 1.02381
hit initial : 265.816 | hit correction : 1.02423
hit initial : 423.127 | hit correction : 1.02468
hit initial : 425.592 | hit correction : 1.02539
hit initial : 315.518 | hit correction : 1.02591
hit initial : 272.797 | hit correction : 1.02622
hit initial : 209.181 | hit correction : 1.02635
hit initial : 58.9937 | hit correction : 1.0264
 *** Total Charge : 3038.19
 Looking at view: 1 | Number of hits in view: 15
hit initial : 17.9153 | hit correction : 1.0256
hit initial : 110.513 | hit correction : 1.02737
hit initial : 81.7481 | hit correction : 1.02692
hit initial : 272.354 | hit correction : 1.02336
hit initial : 219.126 | hit correction : 1.02343
hit initial : 263.391 | hit correction : 1.02355
hit initial : 206.162 | hit correction : 1.02363
hit initial : 249.4 | hit correction : 1.02395
hit initial : 517.84 | hit correction : 1.02462
hit initial : 300.764 | hit correction : 1.0253
hit initial : 182.453 | hit correction : 1.02571
hit initial : 245.686 | hit correction : 1.02597
hit initial : 537.469 | hit correction : 1.02631
hit initial : 42.2965 | hit correction : 1.02632
 *** Total Charge : 3328.03
 Looking at view: 2 | Number of hits in view: 14
Begin processing the 10th record. run: 5027 subRun: 0 event: 29200 at 06-Dec-2025 10:00:03 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  15
Added hit list length: 15 | Max was set to: -3.40282e+38
True Neutron Energy: 939.568
True Electron Energy: 11.8793
Completed MCParticle Loop
True Neutrino Energy: 15.1664
 Size of all hits: 15
hit initial : 36.4896 | hit correction : 1.15905
hit initial : 47.8068 | hit correction : 1.15906
hit initial : 68.0723 | hit correction : 1.15952
hit initial : 59.4981 | hit correction : 1.16198
hit initial : 151.408 | hit correction : 1.16208
hit initial : 221.68 | hit correction : 1.16234
hit initial : 289.984 | hit correction : 1.16265
hit initial : 148.18 | hit correction : 1.16279
 *** Total Charge : 1188.84
 Looking at view: 1 | Number of hits in view: 8
hit initial : 17.5286 | hit correction : 1.1645
hit initial : 35.3251 | hit correction : 1.16449
hit initial : 42.9639 | hit correction : 1.15954
hit initial : 326.287 | hit correction : 1.16208
hit initial : 252.184 | hit correction : 1.16248
hit initial : 184.754 | hit correction : 1.16277
hit initial : 114.968 | hit correction : 1.16299
 *** Total Charge : 1132.23
 Looking at view: 2 | Number of hits in view: 7
06-Dec-2025 10:00:10 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.167866      0.224115      0.451276      0.203701      0.0791306       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000678028   0.00104923    0.00263435    0.000829515   0.00054694       10     
end_path:ana:AnalyzeEvents         0.166859      0.222651      0.450139      0.202499      0.0792798       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19473.3 MB
  Peak resident set size usage (VmHWM): 1535.94 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 19:10:23 UTC       justIN version: 01.06.00