justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 12805.1@dunegpschedd02.fnal.gov

Jobsub ID12805.1@dunegpschedd02.fnal.gov
Workflow ID83
Stage ID1
User nameichong@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-07-30 12:45:35
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce16-multicore
Last heartbeat2025-07-30 13:35:20
From worker nodeHostnamehnode64.pic.es
cpuinfoAMD EPYC 7402P 24-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-07-30 12:47:47
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481123_444_20231201T150236Z_gen_g4_detsim_hitreco__20240507T203712Z_reco2.root
JobscriptExit code0
Real time41m (2485s)
CPU time5m (357s = 14%)
Max RSS bytes2037186560 (1942 MiB)
Outputting started2025-07-30 13:29:13
Output files
Finished2025-07-30 13:35:20
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ar log file ===
Renamed training1_CaloHitListU_graph.data -> graph_output_2025-07-30T_124801Z_8_training1_CaloHitListU_graph.data
Renamed training1_CaloHitListV_graph.data -> graph_output_2025-07-30T_124801Z_8_training1_CaloHitListV_graph.data
Renamed training1_CaloHitListW_graph.data -> graph_output_2025-07-30T_124801Z_8_training1_CaloHitListW_graph.data
Renamed training2_CaloHitListU_graph.data -> graph_output_2025-07-30T_124801Z_8_training2_CaloHitListU_graph.data
Renamed training2_CaloHitListV_graph.data -> graph_output_2025-07-30T_124801Z_8_training2_CaloHitListV_graph.data
Renamed training2_CaloHitListW_graph.data -> graph_output_2025-07-30T_124801Z_8_training2_CaloHitListW_graph.data
Renamed analysiseid.root -> graph_output_2025-07-30T_124801Z_8_analysiseid.root
Renamed ana_tree_hd.root -> graph_output_2025-07-30T_124801Z_8_ana_tree_hd.root
=== Start last 100 lines of lar log file ===
8 1.96032e+06
9 1.95813e+06
10 1.95694e+06
---MC-PARTICLE-MONITORING-----------------------------------------------------------------------

BeamNeutrinos: 

--Primary 0, MCPDG 2212, Energy 1.60536, Dist. 105.212, nMCHits 510 (171, 184, 155)
MCPDG 2212, Energy 1.60536, Dist. 105.212, nMCHits 372 (135, 133, 104)
\_ MCPDG 2212, Energy 1.13396, Dist. 24.969, nMCHits 138 (36, 51, 51)

--Primary 1, MCPDG -211, Energy 0.543481, Dist. 25.9785, nMCHits 312 (113, 105, 94)
MCPDG -211, Energy 0.543481, Dist. 25.9785, nMCHits 104 (18, 48, 38)
\_ MCPDG -211, Energy 0.456663, Dist. 21.0695, nMCHits 95 (36, 40, 19)
   \_ MCPDG 2212, Energy 0.949694, Dist. 0.170736, nMCHits 2 (0, 2, 0)
   \_ MCPDG -211, Energy 0.276454, Dist. 42.6335, nMCHits 106 (57, 14, 35)
      \_ MCPDG 22, Energy 2.04418e-05, Dist. 0.149307, nMCHits 5 (2, 1, 2)
------------------------------------------------------------------------------------------------
Operating in training mode.
The eid is 98
Graph saved to training1_CaloHitListW_graph.data
Size of file training1_CaloHitListW_graph.data is 1458304 bytes.
The eid is 98
Graph saved to training1_CaloHitListU_graph.data
Size of file training1_CaloHitListU_graph.data is 1934776 bytes.
The eid is 98
Graph saved to training1_CaloHitListV_graph.data
Size of file training1_CaloHitListV_graph.data is 1895492 bytes.
Operating in inference mode.
Operating in training mode.
The eid is -1
Graph saved to training2_CaloHitListW_graph.data
Size of file training2_CaloHitListW_graph.data is 1457900 bytes.
The eid is -1
Graph saved to training2_CaloHitListU_graph.data
Size of file training2_CaloHitListU_graph.data is 1933056 bytes.
The eid is -1
Graph saved to training2_CaloHitListV_graph.data
Size of file training2_CaloHitListV_graph.data is 1894684 bytes.
Boundary wire vector sizes: 331, 337, 297
minwire 0: 375
minwire 1: 2373
minwire 2: 84
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0313214, 
Output 1: 0.00168575, 0.000841104, 0.0147123, 0.982761, 
Output 2: 0.423054, 0.103666, 0.000867939, 0.472413, 
Output 3: 0.0633221, 0.891562, 0.0449272, 0.000188545, 
Output 4: 0.0200499, 0.971368, 0.0085645, 1.73384e-05, 
Output 5: 0.998517, 0.00137279, 5.07935e-05, 5.88774e-05, 
Output 6: 0.618328, 0.339507, 0.0337169, 0.00844884, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 179!
30-Jul-2025 15:21:58 CEST  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481123_444_20231201T150236Z_gen_g4_detsim_hitreco__20240507T203712Z_reco2_graph_2025-07-30T_124801Z.root"
30-Jul-2025 15:21:58 CEST  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/ea/b9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481123_444_20231201T150236Z_gen_g4_detsim_hitreco__20240507T203712Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.02292       3.6059        20.3851       3.33619       2.40948        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0511918     0.0770399     0.103959      0.0612238     0.0253133       100    
reco:gaushit:GausHitFinder                              0.16492      0.218492      0.811339      0.179147      0.111107        100    
reco:spsolve:SpacePointSolver                         9.6603e-05     0.105757       3.41258     0.00409059     0.456882        100    
reco:hitfd:DisambigFromSpacePoints                    0.000173335    0.0144374     0.310987     0.00127437     0.0469322       100    
reco:rns:RandomNumberSaver                             1.523e-05    2.40832e-05   0.000213077   2.0926e-05    1.97133e-05      100    
reco:pandora:StandardPandora                           0.161255       1.41113       13.1783       1.03131       1.57307        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000103663   0.00015155    0.00180504    0.000118989   0.000169154      100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000144035   0.000190658   0.00270925    0.00015947    0.000253822      100    
reco:pandoracalo:Calorimetry                          8.2513e-05    0.000119134   0.00132543    0.000102054   0.000122927      100    
reco:pandorapid:Chi2ParticleID                        3.1291e-05    4.1683e-05    0.00060076    3.4776e-05    5.63498e-05      100    
reco:cvnmap:CVNMapper                                  1.914e-05     0.0164119     0.0683118     0.0176766     0.0128951       100    
reco:cvneva:CVNEvaluator                              1.4621e-05     0.614673       3.51245      0.807337      0.467825        100    
reco:energyrecnumu:EnergyReco                         0.00178542    0.00376582     0.0105052    0.00279386    0.00216348       100    
reco:energyrecnue:EnergyReco                          0.00174746    0.00264173    0.00992098    0.00223694    0.00145269       100    
reco:energyrecnc:EnergyReco                            0.0017258    0.00258407    0.00995123    0.00209001    0.00145918       100    
reco:energyrecnumurange:EnergyReco                    0.00174066     0.0026122    0.00995563     0.0021098    0.00153296       100    
reco:energyrecnumumcs:EnergyReco                      0.00171883    0.00259973    0.00998967    0.00209716    0.00155216       100    
reco:opdec:Deconvolution                               0.159366      0.372478      0.581002      0.393196      0.092056        100    
reco:ophitspe:OpHitFinderDeco                          0.452547      0.462981      0.467095      0.463145     0.00204583       100    
reco:opflash:OpFlashFinder                            0.000100783   0.000321923   0.00215859    0.000299104   0.00021639       100    
reco:opslicer:OpSlicer                                3.0441e-05     0.0108366     0.0461713    0.00655804     0.0117414       100    
[art]:TriggerResults:TriggerResultInserter             9.73e-06     1.35162e-05   5.6602e-05    1.27505e-05   5.05717e-06      100    
end_path:out1:RootOutput                               2.43e-06     3.79052e-06   1.4691e-05     3.485e-06    1.33349e-06      100    
end_path:out1:RootOutput(write)                       0.00629383     0.285724      0.701543      0.350767      0.165726        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 10024.9 MB
  Peak resident set size usage (VmHWM): 2037.19 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Generated output files ===
12805.1_dunegpschedd02.fnal.gov.logs.tgz
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481123_444_20231201T150236Z_gen_g4_detsim_hitreco__20240507T203712Z_reco2_graph_2025-07-30T_124801Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481123_444_20231201T150236Z_gen_g4_detsim_hitreco__20240507T203712Z_reco2_graph_2025-07-30T_124801Z.root
debugprod.log
graph_output_2025-07-30T_124801Z_8_ana_tree_hd.root
graph_output_2025-07-30T_124801Z_8_analysiseid.root
graph_output_2025-07-30T_124801Z_8_training1_CaloHitListU_graph.data
graph_output_2025-07-30T_124801Z_8_training1_CaloHitListV_graph.data
graph_output_2025-07-30T_124801Z_8_training1_CaloHitListW_graph.data
graph_output_2025-07-30T_124801Z_8_training2_CaloHitListU_graph.data
graph_output_2025-07-30T_124801Z_8_training2_CaloHitListV_graph.data
graph_output_2025-07-30T_124801Z_8_training2_CaloHitListW_graph.data
jobscript.log
justin-processed-pfns.txt
reco_hist.root
secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481123_444_20231201T150236Z_gen_g4_detsim_hitreco__20240507T203712Z_reco2_graph_2025-07-30T_124801Z.log
third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481123_444_20231201T150236Z_gen_g4_detsim_hitreco__20240507T203712Z_reco2_graph_2025-07-30T_124801Z.log
training1_CaloHitListU.csv
training1_CaloHitListV.csv
training1_CaloHitListW.csv
training2_CaloHitListU.csv
training2_CaloHitListV.csv
training2_CaloHitListW.csv
justIN time: 2025-08-04 21:36:21 UTC       justIN version: 01.04.00