justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 258262.1@dunegpschedd02.fnal.gov

Jobsub ID258262.1@dunegpschedd02.fnal.gov
Workflow ID11012
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-05 22:53:13
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-06 11:13:26
From worker nodeHostnamedunegli-7989276-0-fnpc18006.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-06 01:38:45
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T003658Z_gen_002550_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time9h (34050s)
CPU time8h (31165s = 91%)
Max RSS bytes3080216576 (2937 MiB)
Outputting started2025-12-06 11:06:15
Output files
Finished2025-12-06 11:13:26
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

tput 0: 0.0615885, 
Output 1: 0.0343627, 0.057482, 0.0688795, 0.839276, 
Output 2: 0.0647892, 0.347997, 0.561052, 0.0261617, 
Output 3: 0.899084, 0.0792646, 0.00970152, 0.0119497, 
Output 4: 0.31834, 0.548327, 0.1225, 0.0108331, 
Output 5: 0.830212, 0.154829, 0.0128373, 0.00212199, 
Output 6: 0.0138405, 0.0433997, 0.0654403, 0.87732, 

Begin processing the 10th record. run: 5027 subRun: 0 event: 25500 at 06-Dec-2025 10:21:47 UTC
Boundary wire vector sizes: 6727, 6652, 6664
minwire 0: 47
minwire 1: 1
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.087068, 
Output 1: 0.0608562, 0.0759318, 0.220974, 0.642238, 
Output 2: 0.0492944, 0.119563, 0.814383, 0.0167592, 
Output 3: 0.953356, 0.036145, 0.00424602, 0.00625265, 
Output 4: 0.435567, 0.490854, 0.0639611, 0.00961762, 
Output 5: 0.567735, 0.34842, 0.0669331, 0.0169122, 
Output 6: 0.0260174, 0.066968, 0.0672812, 0.839733, 

06-Dec-2025 11:04:36 UTC  Closed output file "matt_reco2.root"
06-Dec-2025 11:04:36 UTC  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/49/86/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T003658Z_gen_002550_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2561.26       3369.53       5110.94       2997.66       784.579        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00499321     0.026868      0.167539     0.00899104     0.0473164       10     
reco:pandora:StandardPandora                            2557.39       3364.59       5097.12       2993.88       782.254        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00122599     0.0633239     0.281253      0.0455008     0.0764333       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00163893    0.00932265     0.0481758    0.00319192     0.0139542       10     
reco:pandoracalo:Calorimetry                          0.000432625   0.00667445     0.0459086    0.00083281     0.0137114       10     
reco:pandorapid:Chi2ParticleID                        0.000142679   0.000719741   0.00537655    0.000178286   0.00155371       10     
reco:cvnmap:CVNMapper                                  0.0438246     0.0736066     0.155585      0.0550355     0.0346502       10     
reco:cvneva:CVNEvaluator                                2.02657       3.91691       11.9182       3.15728       2.70273        10     
reco:energyrecnumu:EnergyReco                         0.00863404     0.122066      0.312014      0.129082      0.0782347       10     
reco:energyrecnue:EnergyReco                          0.00226332    0.00434884     0.0159901    0.00272796    0.00397758       10     
reco:energyrecnc:EnergyReco                            0.0101196     0.0393593      0.14139      0.0170403     0.0461119       10     
[art]:TriggerResults:TriggerResultInserter            2.5418e-05    5.02917e-05   0.000198675   3.5366e-05    4.96926e-05      10     
end_path:out1:RootOutput                              1.3475e-05    2.29202e-05   5.5104e-05    1.5374e-05    1.58601e-05      10     
end_path:out1:RootOutput(write)                        0.430572      0.674256       1.03767      0.679592      0.160413        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21790.9 MB
  Peak resident set size usage (VmHWM): 3080.22 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
hit initial : 82.5368 | hit correction : 1.1055
hit initial : 14.1642 | hit correction : 1.10549
hit initial : 32.9447 | hit correction : 1.1063
hit initial : 25.1927 | hit correction : 1.10845
hit initial : 30.9394 | hit correction : 1.10824
hit initial : 200.772 | hit correction : 1.1082
hit initial : 430.775 | hit correction : 1.10747
hit initial : 243.785 | hit correction : 1.10702
hit initial : 221.037 | hit correction : 1.10684
hit initial : 183.847 | hit correction : 1.10665
hit initial : 154.016 | hit correction : 1.10655
hit initial : 146.241 | hit correction : 1.10644
hit initial : 135.039 | hit correction : 1.10639
hit initial : 178.217 | hit correction : 1.10629
hit initial : 178.078 | hit correction : 1.10609
hit initial : 152.269 | hit correction : 1.10594
hit initial : 155.734 | hit correction : 1.10582
hit initial : 112.828 | hit correction : 1.10573
 *** Total Charge : 3410.83
 Looking at view: 2 | Number of hits in view: 22
Begin processing the 9th record. run: 5027 subRun: 0 event: 25499 at 06-Dec-2025 11:06:04 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  0
True Electron Energy: 21.5868
Completed MCParticle Loop
True Neutrino Energy: 27.7987
 Size of all hits: 0
Begin processing the 10th record. run: 5027 subRun: 0 event: 25500 at 06-Dec-2025 11:06:09 UTC
*--------------------------- Running LowE Analysis --------------------------------------*
Number of reconstructed particles:  2
This pfp has size:  39
Added hit list length: 39 | Max was set to: -3.40282e+38
True Neutron Energy: 939.569
True Electron Energy: 26.594
Completed MCParticle Loop
True Neutrino Energy: 39.1826
 Size of all hits: 39
hit initial : 231.351 | hit correction : 1.08632
hit initial : 79.0767 | hit correction : 1.08544
hit initial : 66.6185 | hit correction : 1.08636
hit initial : 33.7507 | hit correction : 1.08637
hit initial : 134.149 | hit correction : 1.08351
hit initial : 310.012 | hit correction : 1.08346
hit initial : 327.441 | hit correction : 1.08349
hit initial : 261.631 | hit correction : 1.08359
hit initial : 206.715 | hit correction : 1.08365
hit initial : 213.807 | hit correction : 1.08361
hit initial : 208.736 | hit correction : 1.08368
hit initial : 299.105 | hit correction : 1.0841
hit initial : 311.654 | hit correction : 1.08454
hit initial : 269.467 | hit correction : 1.08493
hit initial : 274.959 | hit correction : 1.08533
hit initial : 338.987 | hit correction : 1.0858
 *** Total Charge : 3868.71
 Looking at view: 0 | Number of hits in view: 16
hit initial : 111.505 | hit correction : 1.08635
hit initial : 325.513 | hit correction : 1.08612
hit initial : 35.5222 | hit correction : 1.08189
hit initial : 23.9745 | hit correction : 1.08188
hit initial : 479.335 | hit correction : 1.08354
hit initial : 616.882 | hit correction : 1.08356
hit initial : 161.21 | hit correction : 1.0836
hit initial : 29.8941 | hit correction : 1.08426
hit initial : 162.621 | hit correction : 1.0849
hit initial : 466.244 | hit correction : 1.08543
 *** Total Charge : 2616.46
 Looking at view: 1 | Number of hits in view: 10
hit initial : 258.718 | hit correction : 1.08352
hit initial : 169.213 | hit correction : 1.08339
hit initial : 145.273 | hit correction : 1.08339
hit initial : 194.486 | hit correction : 1.0835
hit initial : 220.041 | hit correction : 1.08359
hit initial : 225.087 | hit correction : 1.08367
hit initial : 148.132 | hit correction : 1.0836
hit initial : 172.695 | hit correction : 1.08358
hit initial : 199.957 | hit correction : 1.08376
hit initial : 284.073 | hit correction : 1.08416
hit initial : 257.069 | hit correction : 1.08455
hit initial : 398.372 | hit correction : 1.08502
hit initial : 731.83 | hit correction : 1.08598
 *** Total Charge : 3692.25
 Looking at view: 2 | Number of hits in view: 13
06-Dec-2025 11:06:14 UTC  Closed input file "matt_reco2.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                         0.104895      0.217582      0.790442      0.150922      0.193226        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000595883   0.000870106   0.00131135    0.000795124   0.000213535      10     
end_path:ana:AnalyzeEvents         0.103767      0.216383      0.788035      0.149882      0.192831        10     
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19477.9 MB
  Peak resident set size usage (VmHWM): 1536.11 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-12-19 14:55:50 UTC       justIN version: 01.05.03