justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 268037.1@dunegpschedd01.fnal.gov

Jobsub ID268037.1@dunegpschedd01.fnal.gov
Workflow ID11168
Stage ID1
User nameosbiston@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-13 13:06:38
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-13 20:46:08
From worker nodeHostnamedunegli-7890507-0-fnpc17010.fnal.gov
cpuinfoIntel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-13 13:08:21
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T232044Z_gen_001493_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time7h (27454s)
CPU time7h (27147s = 98%)
Max RSS bytes2832400384 (2701 MiB)
Outputting started2025-12-13 20:45:55
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/osbiston/fnal/11168/1/001/analysisLowE_20251213T130822Z.root
Finished2025-12-13 20:46:08
Saved logsjustin-logs:268037.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

piImpl.cc line#: 235
Failure in algorithm Alg0015, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  13-Dec-2025 20:44:33 UTC run: 5027 subRun: 0 event: 14930
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 6638, 6527, 6693
minwire 0: 33
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1154, 1653
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.110485, 
Output 1: 0.0665812, 0.0643553, 0.0868519, 0.782212, 
Output 2: 0.0863439, 0.252195, 0.631241, 0.0302206, 
Output 3: 0.978969, 0.0144414, 0.00132754, 0.00526166, 
Output 4: 0.52155, 0.438556, 0.0361997, 0.0036943, 
Output 5: 0.479987, 0.43825, 0.0682179, 0.0135448, 
Output 6: 0.0213745, 0.0571616, 0.0722215, 0.849243, 

13-Dec-2025 20:44:40 UTC  Closed output file "matt_reco2.root"
13-Dec-2025 20:44:40 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/a1/c7/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T232044Z_gen_001493_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2201.74       2713.14       4096.39       2538.18       503.246        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00104375     0.0051563     0.0229577    0.00134118    0.00783676       10     
reco:pandora:StandardPandora                            2199.38       2709.96       4088.23       2535.85       501.684        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000946994    0.0849826     0.153786      0.0937759     0.037968        10     
reco:pandoraShower:LArPandoraModularShowerCreation     0.0032915     0.0128724     0.0507186      0.01022      0.0131788       10     
reco:pandoracalo:Calorimetry                          0.000500269   0.00223888     0.0116761    0.00112883    0.00319451       10     
reco:pandorapid:Chi2ParticleID                        0.000115095    0.0006885    0.00506535    0.000205798   0.00145973       10     
reco:cvnmap:CVNMapper                                  0.0836426     0.139152       0.25572      0.115863      0.0566007       10     
reco:cvneva:CVNEvaluator                                1.21052       2.24157       6.96185       1.80566       1.59598        10     
reco:energyrecnumu:EnergyReco                          0.0988974     0.156996      0.327756       0.12926      0.0661349       10     
reco:energyrecnue:EnergyReco                           0.0016353    0.00268613    0.00646783    0.00231971    0.00130247       10     
reco:energyrecnc:EnergyReco                           0.00828278     0.0131721     0.0194748     0.0117616    0.00344738       10     
[art]:TriggerResults:TriggerResultInserter            1.7676e-05    6.44215e-05   0.000385405   3.04535e-05   0.00010714       10     
end_path:out1:RootOutput                               7.853e-06    1.94352e-05   6.3538e-05    1.1635e-05    1.74026e-05      10     
end_path:out1:RootOutput(write)                        0.341319      0.518641      0.576592      0.544088      0.0659119       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 11906.7 MB
  Peak resident set size usage (VmHWM): 2832.4 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
 Looking at view: 1 | Number of hits in view: 17
hit initial : 70.4332 | hit correction : 1.22985
hit initial : 222.515 | hit correction : 1.22983
hit initial : 61.2836 | hit correction : 1.23366
hit initial : 26.6621 | hit correction : 1.23485
hit initial : 89.4541 | hit correction : 1.23487
hit initial : 132.519 | hit correction : 1.23572
hit initial : 187.294 | hit correction : 1.23572
hit initial : 397.301 | hit correction : 1.23598
hit initial : 202.472 | hit correction : 1.23589
hit initial : 35.6946 | hit correction : 1.23596
hit initial : 246.796 | hit correction : 1.23647
hit initial : 158.537 | hit correction : 1.23665
hit initial : 116.583 | hit correction : 1.23679
hit initial : 117.134 | hit correction : 1.23688
hit initial : 114.251 | hit correction : 1.237
hit initial : 111.406 | hit correction : 1.23711
hit initial : 109.332 | hit correction : 1.23724
hit initial : 115.372 | hit correction : 1.23733
hit initial : 25.7717 | hit correction : 1.23738
 *** Total Charge : 3139.21
 Looking at view: 2 | Number of hits in view: 19
Begin processing the 10th record. run: 5027 subRun: 0 event: 14930 at 13-Dec-2025 20:45:48 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  4
This pfp has size:  43
Added hit list length: 43 | Max was set to: -3.40282e+38
This pfp has size:  4
True Neutron Energy: 939.577
True Electron Energy: 30.6903
Completed MCParticle Loop
True Neutrino Energy: 34.4366
 Size of all hits: 43
hit initial : 63.0174 | hit correction : 1.0226
hit initial : 245.529 | hit correction : 1.02265
hit initial : 447.583 | hit correction : 1.02273
hit initial : 342.212 | hit correction : 1.02291
hit initial : 231.436 | hit correction : 1.02296
hit initial : 247.571 | hit correction : 1.02292
hit initial : 255.774 | hit correction : 1.02294
hit initial : 255.282 | hit correction : 1.02304
hit initial : 213.992 | hit correction : 1.0232
hit initial : 226.47 | hit correction : 1.0234
hit initial : 231.232 | hit correction : 1.02353
hit initial : 259.981 | hit correction : 1.02357
hit initial : 298.988 | hit correction : 1.02359
hit initial : 256.759 | hit correction : 1.0236
hit initial : 186.803 | hit correction : 1.02356
hit initial : 121.085 | hit correction : 1.02348
 *** Total Charge : 3973.64
 Looking at view: 0 | Number of hits in view: 16
hit initial : 47.5046 | hit correction : 1.02259
hit initial : 42.5754 | hit correction : 1.02259
hit initial : 17.7995 | hit correction : 1.0233
hit initial : 45.1378 | hit correction : 1.02329
hit initial : 350.452 | hit correction : 1.02289
hit initial : 937.724 | hit correction : 1.02289
hit initial : 839.643 | hit correction : 1.02304
hit initial : 290.565 | hit correction : 1.02351
hit initial : 288.032 | hit correction : 1.02358
hit initial : 354.761 | hit correction : 1.02359
hit initial : 292.231 | hit correction : 1.02357
hit initial : 148.399 | hit correction : 1.02352
 *** Total Charge : 3739.53
 Looking at view: 1 | Number of hits in view: 12
hit initial : 26.1366 | hit correction : 1.0305
hit initial : 121.756 | hit correction : 1.0226
hit initial : 87.0233 | hit correction : 1.02259
hit initial : 385.988 | hit correction : 1.02268
hit initial : 278.855 | hit correction : 1.0228
hit initial : 157.045 | hit correction : 1.02294
hit initial : 238.125 | hit correction : 1.02291
hit initial : 167.154 | hit correction : 1.02297
hit initial : 181.864 | hit correction : 1.02289
hit initial : 166.549 | hit correction : 1.023
hit initial : 230.2 | hit correction : 1.02301
hit initial : 211.269 | hit correction : 1.02316
hit initial : 212.349 | hit correction : 1.02335
hit initial : 1085.35 | hit correction : 1.02357
hit initial : 454.661 | hit correction : 1.0235
 *** Total Charge : 4097.25
 Looking at view: 2 | Number of hits in view: 15
13-Dec-2025 20:45:53 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.112944      0.143032      0.172365      0.141613      0.0158459       10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000602308   0.00077909    0.00145228    0.000684388   0.000240637      10     
end_path:ana:AnalyzeEvents         0.112057      0.141986      0.171435      0.140699      0.0157763       10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9809.51 MB
  Peak resident set size usage (VmHWM): 1533.16 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2026-02-07 12:05:56 UTC       justIN version: 01.06.00