justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 257480.1@dunegpschedd02.fnal.gov

Jobsub ID257480.1@dunegpschedd02.fnal.gov
Workflow ID10938
Stage ID1
User nameichong@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-04 06:27:13
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_klomp
Last heartbeat2025-12-04 07:14:01
From worker nodeHostnamewn-sate-031.farm.nikhef.nl
cpuinfoAMD EPYC 7551P 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-04 06:28:38
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1500_412_20231028T033424Z_gen_g4_detsim_hitreco__20240503T045256Z_reco2.root
JobscriptExit code0
Real time38m (2281s)
CPU time16m (1012s = 44%)
Max RSS bytes2154692608 (2054 MiB)
Outputting started2025-12-04 07:06:40
Output files
Finished2025-12-04 07:14:01
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6533.8 MB
  Peak resident set size usage (VmHWM): 1255.6 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of third lar log file ===
Renamed training1_CaloHitListU_graph.data -> graph_output_2025-12-04T_062901Z_5_training1_CaloHitListU_graph.data
Renamed training1_CaloHitListV_graph.data -> graph_output_2025-12-04T_062901Z_5_training1_CaloHitListV_graph.data
Renamed training1_CaloHitListW_graph.data -> graph_output_2025-12-04T_062901Z_5_training1_CaloHitListW_graph.data
Renamed training2_CaloHitListU_graph.data -> graph_output_2025-12-04T_062901Z_5_training2_CaloHitListU_graph.data
Renamed training2_CaloHitListV_graph.data -> graph_output_2025-12-04T_062901Z_5_training2_CaloHitListV_graph.data
Renamed training2_CaloHitListW_graph.data -> graph_output_2025-12-04T_062901Z_5_training2_CaloHitListW_graph.data
Renamed analysiseid.root -> graph_output_2025-12-04T_062901Z_5_analysiseid.root
Renamed ana_tree_hd.root -> graph_output_2025-12-04T_062901Z_5_ana_tree_hd.root
=== Start last 100 lines of lar log file ===
Iterating with no regularization...
Begin: 2.28495e+07
0 2.0356e+07
1 2.01878e+07
2 2.01689e+07
Now with regularization...
Begin: 1.41914e+07
0 1.41862e+07
---MC-PARTICLE-MONITORING-----------------------------------------------------------------------

BeamNeutrinos: 

--Primary 0, MCPDG 13, Energy 0.910111, Dist. 355.437, nMCHits 1052 (323, 563, 166)
MCPDG 13, Energy 0.910111, Dist. 355.437, nMCHits 997 (303, 541, 153)
\_ MCPDG 11, Energy 0.039855, Dist. 6.31238, nMCHits 55 (20, 22, 13)

--Primary 1, MCPDG 2212, Energy 1.14526, Dist. 27.4358, nMCHits 19 (5, 7, 7)
MCPDG 2212, Energy 1.14526, Dist. 27.4358, nMCHits 19 (5, 7, 7)
------------------------------------------------------------------------------------------------
Operating in training mode.
The eid is 198
Graph saved to training1_CaloHitListW_graph.data
Size of file training1_CaloHitListW_graph.data is 3306580 bytes.
The eid is 198
Graph saved to training1_CaloHitListU_graph.data
Size of file training1_CaloHitListU_graph.data is 4491324 bytes.
The eid is 198
Graph saved to training1_CaloHitListV_graph.data
Size of file training1_CaloHitListV_graph.data is 4856804 bytes.
Operating in inference mode.
Operating in training mode.
The eid is -1
Graph saved to training2_CaloHitListW_graph.data
Size of file training2_CaloHitListW_graph.data is 3301572 bytes.
The eid is -1
Graph saved to training2_CaloHitListU_graph.data
Size of file training2_CaloHitListU_graph.data is 4489604 bytes.
The eid is -1
Graph saved to training2_CaloHitListV_graph.data
Size of file training2_CaloHitListV_graph.data is 4854988 bytes.
Boundary wire vector sizes: 333, 573, 174
minwire 0: 994
minwire 1: 1380
minwire 2: 997
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0399607, 
Output 1: 0.896071, 0.00112239, 0.0251974, 0.0776093, 
Output 2: 0.677261, 0.0261127, 0.00199368, 0.294633, 
Output 3: 0.495024, 0.493871, 0.0108186, 0.000286353, 
Output 4: 0.899312, 0.100568, 0.000114359, 5.86251e-06, 
Output 5: 0.991267, 0.00838143, 0.000107077, 0.000244968, 
Output 6: 0.33322, 0.599656, 0.051803, 0.015321, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 247!
04-Dec-2025 08:00:52 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1500_412_20231028T033424Z_gen_g4_detsim_hitreco__20240503T045256Z_reco2_graph_2025-12-04T_062901Z.root"
04-Dec-2025 08:00:52 CET  Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/90/c2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1500_412_20231028T033424Z_gen_g4_detsim_hitreco__20240503T045256Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.358561       4.59271       32.2775       4.28083        3.175         200    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0153441     0.0311961     0.0722846     0.0309594     0.0116412       200    
reco:gaushit:GausHitFinder                             0.0594976     0.149008      0.946215      0.0814282     0.153742        200    
reco:spsolve:SpacePointSolver                         0.000117431    0.122689       4.13235     0.00616418     0.398659        200    
reco:hitfd:DisambigFromSpacePoints                    0.000208883    0.0302524     0.869495     0.00146306     0.0951945       200    
reco:rns:RandomNumberSaver                            1.9647e-05    3.34676e-05   0.000481366   2.7827e-05    3.96201e-05      200    
reco:pandora:StandardPandora                           0.0534039      2.01778       22.3809       1.45207       2.02007        200    
reco:pandoraTrack:LArPandoraTrackCreation             0.000138981   0.000237021    0.0072523    0.000188269   0.000510879      200    
reco:pandoraShower:LArPandoraModularShowerCreation    0.00018616    0.000231817   0.00297643    0.000205947   0.000203749      200    
reco:pandoracalo:Calorimetry                          0.000115227   0.000152204   0.00139302    0.000143765   8.96221e-05      200    
reco:pandorapid:Chi2ParticleID                        4.1678e-05    5.38032e-05   0.000672425   4.5315e-05    5.96228e-05      200    
reco:cvnmap:CVNMapper                                 2.5939e-05     0.0226734     0.0716577     0.0274037     0.014951        200    
reco:cvneva:CVNEvaluator                              1.9266e-05      1.32938       4.29829       1.70308      0.831463        200    
reco:energyrecnumu:EnergyReco                         0.00263777     0.0068277     0.0474516    0.00620968    0.00498052       200    
reco:energyrecnue:EnergyReco                          0.00263987    0.00434367     0.0416124     0.0030169    0.00438229       200    
reco:energyrecnc:EnergyReco                           0.00255109     0.0040546     0.0333225    0.00296531    0.00334426       200    
reco:energyrecnumurange:EnergyReco                    0.00260372    0.00415631     0.0369636    0.00300369    0.00360509       200    
reco:energyrecnumumcs:EnergyReco                      0.00249094    0.00388865     0.0291883    0.00296797    0.00265751       200    
reco:opdec:Deconvolution                               0.069056      0.298836       1.13158      0.284947       0.11933        200    
reco:ophitspe:OpHitFinderDeco                          0.131295      0.136956      0.156752      0.136023     0.00437368       200    
reco:opflash:OpFlashFinder                            0.00011651    0.000447103    0.0112344    0.000352408   0.000788788      200    
reco:opslicer:OpSlicer                                4.5084e-05     0.0184424     0.0871166    0.00813585     0.0207647       200    
[art]:TriggerResults:TriggerResultInserter            1.2784e-05    1.68363e-05   7.6374e-05     1.626e-05    4.91297e-06      200    
end_path:out1:RootOutput                               2.996e-06    4.6625e-06    2.2833e-05     3.932e-06     1.84e-06        200    
end_path:out1:RootOutput(write)                        0.012749      0.409767       1.37855      0.469148      0.231561        200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7985.66 MB
  Peak resident set size usage (VmHWM): 2154.69 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Generated output files ===
257480.1_dunegpschedd02.fnal.gov.logs.tgz
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1500_412_20231028T033424Z_gen_g4_detsim_hitreco__20240503T045256Z_reco2_graph_2025-12-04T_062901Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1500_412_20231028T033424Z_gen_g4_detsim_hitreco__20240503T045256Z_reco2_graph_2025-12-04T_062901Z.root
debugprod.log
graph_output_2025-12-04T_062901Z_5_ana_tree_hd.root
graph_output_2025-12-04T_062901Z_5_analysiseid.root
graph_output_2025-12-04T_062901Z_5_training1_CaloHitListU_graph.data
graph_output_2025-12-04T_062901Z_5_training1_CaloHitListV_graph.data
graph_output_2025-12-04T_062901Z_5_training1_CaloHitListW_graph.data
graph_output_2025-12-04T_062901Z_5_training2_CaloHitListU_graph.data
graph_output_2025-12-04T_062901Z_5_training2_CaloHitListV_graph.data
graph_output_2025-12-04T_062901Z_5_training2_CaloHitListW_graph.data
jobscript.log
justin-processed-pfns.txt
reco_hist.root
secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1500_412_20231028T033424Z_gen_g4_detsim_hitreco__20240503T045256Z_reco2_graph_2025-12-04T_062901Z.log
third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1500_412_20231028T033424Z_gen_g4_detsim_hitreco__20240503T045256Z_reco2_graph_2025-12-04T_062901Z.log
training1_CaloHitListU.csv
training1_CaloHitListV.csv
training1_CaloHitListW.csv
training2_CaloHitListU.csv
training2_CaloHitListV.csv
training2_CaloHitListW.csv
justIN time: 2026-02-04 13:39:22 UTC       justIN version: 01.06.00