justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258328.3@dunegpschedd02.fnal.gov

Jobsub ID258328.3@dunegpschedd02.fnal.gov
Workflow ID11028
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-06 01:22:33
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-06 10:30:02
From worker nodeHostnamedunegli-7832153-0-fnpc18009.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-06 01:41:55
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T022243Z_gen_003971_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time8h (31279s)
CPU time8h (30751s = 98%)
Max RSS bytes3044454400 (2903 MiB)
Outputting started2025-12-06 10:23:14
Output files
Finished2025-12-06 10:30:02
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

4: 0.862425, 0.12293, 0.0130041, 0.00164137, 
Output 5: 0.623912, 0.262317, 0.0921719, 0.0215985, 
Output 6: 0.0128564, 0.0286355, 0.044139, 0.914369, 

Begin processing the 10th record. run: 5027 subRun: 0 event: 39710 at 06-Dec-2025 09:39:13 UTC
Boundary wire vector sizes: 6703, 6461, 6567
minwire 0: 43
minwire 1: 1
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 402, 901
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.10598, 
Output 1: 0.132473, 0.124134, 0.22899, 0.514403, 
Output 2: 0.0316915, 0.110689, 0.843088, 0.0145311, 
Output 3: 0.960477, 0.0302215, 0.00391651, 0.00538457, 
Output 4: 0.594777, 0.34241, 0.0525382, 0.0102752, 
Output 5: 0.345351, 0.419355, 0.183443, 0.0518509, 
Output 6: 0.0871261, 0.155666, 0.108347, 0.648861, 

06-Dec-2025 10:22:18 UTC  Closed output file "matt_reco2.root"
06-Dec-2025 10:22:18 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/bc/96/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T022243Z_gen_003971_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2527.38       3079.51       6097.83       2708.02       1036.12        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00318194     0.0110727     0.0415572     0.0067819     0.0111423       10     
reco:pandora:StandardPandora                            2523.75       3069.01       6020.15       2705.18       1014.35        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00197177     0.0485638     0.306151      0.0120765     0.0887621       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000609951   0.00923292     0.0446367    0.00334024     0.0124101       10     
reco:pandoracalo:Calorimetry                          0.000385778   0.00359852     0.0114536    0.000984698   0.00435976       10     
reco:pandorapid:Chi2ParticleID                        0.000105479   0.000675528   0.00486732    0.000197232   0.00140065       10     
reco:cvnmap:CVNMapper                                  0.0484989     0.0629134     0.131514      0.0572207     0.0232334       10     
reco:cvneva:CVNEvaluator                                1.9229        2.74329       7.12128       2.05861       1.50139        10     
reco:energyrecnumu:EnergyReco                          0.0144668     0.0860486     0.146096      0.110889      0.048056        10     
reco:energyrecnue:EnergyReco                          0.00155545    0.00604089     0.0150947    0.00252537    0.00561541       10     
reco:energyrecnc:EnergyReco                           0.00947995      6.88267        68.38       0.0150009      20.4992        10     
[art]:TriggerResults:TriggerResultInserter            2.3124e-05    5.0356e-05    0.000219384   3.05325e-05   5.67875e-05      10     
end_path:out1:RootOutput                               7.184e-06    1.68678e-05   6.4982e-05    1.14365e-05   1.62317e-05      10     
end_path:out1:RootOutput(write)                        0.372302      0.638364       1.58293      0.572089       0.32536        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21777.1 MB
  Peak resident set size usage (VmHWM): 3044.45 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 619.333 | hit correction : 1.08403
hit initial : 381.367 | hit correction : 1.08431
hit initial : 138.752 | hit correction : 1.08465
hit initial : 30.2284 | hit correction : 1.08481
hit initial : 121.354 | hit correction : 1.08443
hit initial : 140.266 | hit correction : 1.08442
hit initial : 57.4531 | hit correction : 1.08889
hit initial : 214.351 | hit correction : 1.08886
hit initial : 182.496 | hit correction : 1.08885
hit initial : 41.4158 | hit correction : 1.08885
hit initial : 105.558 | hit correction : 1.08952
hit initial : 143.333 | hit correction : 1.09006
hit initial : 121.053 | hit correction : 1.08954
hit initial : 213.905 | hit correction : 1.09014
hit initial : 36.0639 | hit correction : 1.08957
hit initial : 28.1582 | hit correction : 1.09132
hit initial : 93.7466 | hit correction : 1.09028
hit initial : 44.9441 | hit correction : 1.09141
hit initial : 25.8358 | hit correction : 1.09153
hit initial : 191.579 | hit correction : 1.09232
hit initial : 546.617 | hit correction : 1.09234
hit initial : 410.502 | hit correction : 1.09246
hit initial : 243.021 | hit correction : 1.09263
hit initial : 267.669 | hit correction : 1.0929
hit initial : 295.806 | hit correction : 1.09328
hit initial : 255.001 | hit correction : 1.09347
hit initial : 205.825 | hit correction : 1.0938
hit initial : 218.913 | hit correction : 1.094
hit initial : 233.62 | hit correction : 1.09421
hit initial : 240.427 | hit correction : 1.09446
hit initial : 225.962 | hit correction : 1.09465
hit initial : 137.178 | hit correction : 1.0948
 *** Total Charge : 7544.26
 Looking at view: 1 | Number of hits in view: 37
hit initial : 37.3734 | hit correction : 1.09583
hit initial : 17.5928 | hit correction : 1.09167
hit initial : 31.5399 | hit correction : 1.09165
hit initial : 45.138 | hit correction : 1.0859
hit initial : 52.2908 | hit correction : 1.08589
hit initial : 36.9536 | hit correction : 1.08569
hit initial : 263.191 | hit correction : 1.08462
hit initial : 17.409 | hit correction : 1.08379
hit initial : 427.646 | hit correction : 1.08378
hit initial : 142.618 | hit correction : 1.08402
hit initial : 53.2062 | hit correction : 1.08464
hit initial : 130.426 | hit correction : 1.08399
hit initial : 170.875 | hit correction : 1.08403
hit initial : 198.085 | hit correction : 1.08462
hit initial : 65.6941 | hit correction : 1.08479
hit initial : 347.401 | hit correction : 1.08431
hit initial : 258.732 | hit correction : 1.08431
hit initial : 147.398 | hit correction : 1.0844
hit initial : 17.9685 | hit correction : 1.08978
hit initial : 37.647 | hit correction : 1.08983
hit initial : 159.46 | hit correction : 1.08895
hit initial : 53.4702 | hit correction : 1.08884
hit initial : 275.045 | hit correction : 1.08881
hit initial : 193.844 | hit correction : 1.08954
hit initial : 259.545 | hit correction : 1.09008
hit initial : 137.235 | hit correction : 1.08982
hit initial : 64.8709 | hit correction : 1.0913
hit initial : 144.21 | hit correction : 1.09034
hit initial : 26.4848 | hit correction : 1.09153
hit initial : 24.9576 | hit correction : 1.09202
hit initial : 330.1 | hit correction : 1.09209
hit initial : 131.81 | hit correction : 1.09236
hit initial : 128.852 | hit correction : 1.09236
hit initial : 154.943 | hit correction : 1.09241
hit initial : 179.097 | hit correction : 1.0925
hit initial : 196.712 | hit correction : 1.09253
hit initial : 161.353 | hit correction : 1.09258
hit initial : 223.269 | hit correction : 1.09272
hit initial : 270.415 | hit correction : 1.09305
hit initial : 363.523 | hit correction : 1.09337
hit initial : 214.841 | hit correction : 1.0937
hit initial : 215.558 | hit correction : 1.09392
hit initial : 240.942 | hit correction : 1.09412
hit initial : 244.532 | hit correction : 1.09438
hit initial : 257.335 | hit correction : 1.09461
hit initial : 168.255 | hit correction : 1.09484
 *** Total Charge : 7976.53
 Looking at view: 2 | Number of hits in view: 46
06-Dec-2025 10:23:14 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.089535      0.114657      0.137907      0.113459      0.016312        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000440311   0.000558428   0.000861967   0.000533652   0.000111625      10     
end_path:ana:AnalyzeEvents         0.0887713     0.113866      0.137218      0.112728      0.0162674       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19477.8 MB
  Peak resident set size usage (VmHWM): 1539.26 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 19:09:13 UTC       justIN version: 01.06.00