justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 257483.0@dunegpschedd02.fnal.gov

Jobsub ID257483.0@dunegpschedd02.fnal.gov
Workflow ID10938
Stage ID1
User nameichong@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-04 07:25:15
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_klomp
Last heartbeat2025-12-04 08:09:28
From worker nodeHostnamewn-sate-032.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-04 07:25:47
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1500_412_20231028T033424Z_gen_g4_detsim_hitreco__20240503T045256Z_reco2.root
JobscriptExit code0
Real time36m (2209s)
CPU time16m (999s = 45%)
Max RSS bytes2153070592 (2053 MiB)
Outputting started2025-12-04 08:02:37
Output files
Finished2025-12-04 08:09:28
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

==============
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6533.81 MB
  Peak resident set size usage (VmHWM): 1255.47 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of third lar log file ===
Renamed training1_CaloHitListU_graph.data -> graph_output_2025-12-04T_072600Z_5_training1_CaloHitListU_graph.data
Renamed training1_CaloHitListV_graph.data -> graph_output_2025-12-04T_072600Z_5_training1_CaloHitListV_graph.data
Renamed training1_CaloHitListW_graph.data -> graph_output_2025-12-04T_072600Z_5_training1_CaloHitListW_graph.data
Renamed training2_CaloHitListU_graph.data -> graph_output_2025-12-04T_072600Z_5_training2_CaloHitListU_graph.data
Renamed training2_CaloHitListV_graph.data -> graph_output_2025-12-04T_072600Z_5_training2_CaloHitListV_graph.data
Renamed training2_CaloHitListW_graph.data -> graph_output_2025-12-04T_072600Z_5_training2_CaloHitListW_graph.data
Renamed analysiseid.root -> graph_output_2025-12-04T_072600Z_5_analysiseid.root
Renamed ana_tree_hd.root -> graph_output_2025-12-04T_072600Z_5_ana_tree_hd.root
=== Start last 100 lines of lar log file ===
Iterating with no regularization...
Begin: 2.28495e+07
0 2.0356e+07
1 2.01878e+07
2 2.01689e+07
Now with regularization...
Begin: 1.41914e+07
0 1.41862e+07
---MC-PARTICLE-MONITORING-----------------------------------------------------------------------

BeamNeutrinos: 

--Primary 0, MCPDG 13, Energy 0.910111, Dist. 355.437, nMCHits 1052 (323, 563, 166)
MCPDG 13, Energy 0.910111, Dist. 355.437, nMCHits 997 (303, 541, 153)
\_ MCPDG 11, Energy 0.039855, Dist. 6.31238, nMCHits 55 (20, 22, 13)

--Primary 1, MCPDG 2212, Energy 1.14526, Dist. 27.4358, nMCHits 19 (5, 7, 7)
MCPDG 2212, Energy 1.14526, Dist. 27.4358, nMCHits 19 (5, 7, 7)
------------------------------------------------------------------------------------------------
Operating in training mode.
The eid is 198
Graph saved to training1_CaloHitListW_graph.data
Size of file training1_CaloHitListW_graph.data is 3306588 bytes.
The eid is 198
Graph saved to training1_CaloHitListU_graph.data
Size of file training1_CaloHitListU_graph.data is 4491468 bytes.
The eid is 198
Graph saved to training1_CaloHitListV_graph.data
Size of file training1_CaloHitListV_graph.data is 4856788 bytes.
Operating in inference mode.
Operating in training mode.
The eid is -1
Graph saved to training2_CaloHitListW_graph.data
Size of file training2_CaloHitListW_graph.data is 3301564 bytes.
The eid is -1
Graph saved to training2_CaloHitListU_graph.data
Size of file training2_CaloHitListU_graph.data is 4489748 bytes.
The eid is -1
Graph saved to training2_CaloHitListV_graph.data
Size of file training2_CaloHitListV_graph.data is 4854964 bytes.
Boundary wire vector sizes: 333, 573, 174
minwire 0: 994
minwire 1: 1380
minwire 2: 997
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0399607, 
Output 1: 0.896071, 0.00112239, 0.0251974, 0.0776093, 
Output 2: 0.677261, 0.0261127, 0.00199368, 0.294633, 
Output 3: 0.495024, 0.493871, 0.0108186, 0.000286353, 
Output 4: 0.899312, 0.100568, 0.000114359, 5.86251e-06, 
Output 5: 0.991267, 0.00838143, 0.000107077, 0.000244968, 
Output 6: 0.33322, 0.599656, 0.051803, 0.015321, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 247!
04-Dec-2025 08:57:11 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1500_412_20231028T033424Z_gen_g4_detsim_hitreco__20240503T045256Z_reco2_graph_2025-12-04T_072600Z.root"
04-Dec-2025 08:57:11 CET  Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/90/c2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1500_412_20231028T033424Z_gen_g4_detsim_hitreco__20240503T045256Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.361622       4.44641       30.7142       4.21282       3.02717        200    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0147116     0.0227963     0.0348759     0.0194786    0.00718594       200    
reco:gaushit:GausHitFinder                             0.0593154     0.154296       0.99785      0.0922804     0.156613        200    
reco:spsolve:SpacePointSolver                         0.000114305    0.114323       3.92477     0.00548313     0.370659        200    
reco:hitfd:DisambigFromSpacePoints                    0.000217049    0.0298526      0.88019     0.00136039     0.0950832       200    
reco:rns:RandomNumberSaver                            1.8095e-05    2.9781e-05    0.000436963   2.5253e-05    2.99534e-05      200    
reco:pandora:StandardPandora                           0.0521351      1.93486       21.2593       1.39184       1.92907        200    
reco:pandoraTrack:LArPandoraTrackCreation              0.0001379    0.000231288   0.00713967    0.000182994   0.000492605      200    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000190329   0.000235217   0.00332103    0.000212185   0.000221021      200    
reco:pandoracalo:Calorimetry                          0.000118263   0.000152856   0.00168772    0.000141661   0.000110535      200    
reco:pandorapid:Chi2ParticleID                         4.202e-05    5.49298e-05   0.000728461   4.9543e-05    4.96012e-05      200    
reco:cvnmap:CVNMapper                                  2.674e-05     0.0217331     0.060166      0.0266993     0.0141629       200    
reco:cvneva:CVNEvaluator                              1.9437e-05      1.30432       3.96325       1.7286       0.789622        200    
reco:energyrecnumu:EnergyReco                         0.00259326    0.00606319     0.0196007    0.00586088    0.00300061       200    
reco:energyrecnue:EnergyReco                          0.00248305    0.00368437     0.0114578    0.00303185    0.00160974       200    
reco:energyrecnc:EnergyReco                           0.00246988    0.00361694     0.0115937    0.00292689    0.00161412       200    
reco:energyrecnumurange:EnergyReco                    0.00247716    0.00372357     0.0197921    0.00293889    0.00202218       200    
reco:energyrecnumumcs:EnergyReco                      0.00244285     0.0036103     0.0114454    0.00290833    0.00162502       200    
reco:opdec:Deconvolution                               0.0744701     0.293856      0.500753      0.292311      0.0930485       200    
reco:ophitspe:OpHitFinderDeco                          0.128746      0.135171      0.173529      0.133922     0.00506062       200    
reco:opflash:OpFlashFinder                            0.00011178    0.000387458   0.00304252    0.000345179   0.000241494      200    
reco:opslicer:OpSlicer                                4.7249e-05     0.0172086     0.0783728    0.00830859     0.0192205       200    
[art]:TriggerResults:TriggerResultInserter            1.1812e-05    1.64098e-05   7.6374e-05    1.5344e-05    6.4533e-06       200    
end_path:out1:RootOutput                               2.805e-06    4.51426e-06   2.3785e-05     4.784e-06    1.87205e-06      200    
end_path:out1:RootOutput(write)                        0.0131459     0.394947       1.04568      0.462737       0.21998        200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7991.43 MB
  Peak resident set size usage (VmHWM): 2153.07 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Generated output files ===
257483.0_dunegpschedd02.fnal.gov.logs.tgz
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1500_412_20231028T033424Z_gen_g4_detsim_hitreco__20240503T045256Z_reco2_graph_2025-12-04T_072600Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1500_412_20231028T033424Z_gen_g4_detsim_hitreco__20240503T045256Z_reco2_graph_2025-12-04T_072600Z.root
debugprod.log
graph_output_2025-12-04T_072600Z_5_ana_tree_hd.root
graph_output_2025-12-04T_072600Z_5_analysiseid.root
graph_output_2025-12-04T_072600Z_5_training1_CaloHitListU_graph.data
graph_output_2025-12-04T_072600Z_5_training1_CaloHitListV_graph.data
graph_output_2025-12-04T_072600Z_5_training1_CaloHitListW_graph.data
graph_output_2025-12-04T_072600Z_5_training2_CaloHitListU_graph.data
graph_output_2025-12-04T_072600Z_5_training2_CaloHitListV_graph.data
graph_output_2025-12-04T_072600Z_5_training2_CaloHitListW_graph.data
jobscript.log
justin-processed-pfns.txt
reco_hist.root
secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1500_412_20231028T033424Z_gen_g4_detsim_hitreco__20240503T045256Z_reco2_graph_2025-12-04T_072600Z.log
third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1500_412_20231028T033424Z_gen_g4_detsim_hitreco__20240503T045256Z_reco2_graph_2025-12-04T_072600Z.log
training1_CaloHitListU.csv
training1_CaloHitListV.csv
training1_CaloHitListW.csv
training2_CaloHitListU.csv
training2_CaloHitListV.csv
training2_CaloHitListW.csv
justIN time: 2025-12-19 15:40:47 UTC       justIN version: 01.05.03