justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 256474.1@dunegpschedd02.fnal.gov

Jobsub ID256474.1@dunegpschedd02.fnal.gov
Workflow ID10938
Stage ID1
User nameichong@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-02 17:12:13
SiteBR_CBPF
EntryDUNE_BR_CBPF_ce03
Last heartbeat2025-12-02 23:41:40
From worker nodeHostnamewn81
cpuinfoAMD EPYC 7713P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-02 17:14:15
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50253389_320_20231120T134814Z_gen_g4_detsim_hitreco__20240503T060637Z_reco2.root
JobscriptExit code0
Real time5h (19160s)
CPU time16m (974s = 5%)
Max RSS bytes1952399360 (1861 MiB)
Outputting started2025-12-02 22:33:38
Output files
Finished2025-12-02 23:41:40
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

sed)

  Peak virtual memory usage (VmPeak)  : 19156.1 MB
  Peak resident set size usage (VmHWM): 971.469 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of third lar log file ===
Renamed training1_CaloHitListU_graph.data -> graph_output_2025-12-02T_171434Z_2_training1_CaloHitListU_graph.data
Renamed training1_CaloHitListV_graph.data -> graph_output_2025-12-02T_171434Z_2_training1_CaloHitListV_graph.data
Renamed training1_CaloHitListW_graph.data -> graph_output_2025-12-02T_171434Z_2_training1_CaloHitListW_graph.data
Renamed training2_CaloHitListU_graph.data -> graph_output_2025-12-02T_171434Z_2_training2_CaloHitListU_graph.data
Renamed training2_CaloHitListV_graph.data -> graph_output_2025-12-02T_171434Z_2_training2_CaloHitListV_graph.data
Renamed training2_CaloHitListW_graph.data -> graph_output_2025-12-02T_171434Z_2_training2_CaloHitListW_graph.data
Renamed analysiseid.root -> graph_output_2025-12-02T_171434Z_2_analysiseid.root
Renamed ana_tree_hd.root -> graph_output_2025-12-02T_171434Z_2_ana_tree_hd.root
=== Start last 100 lines of lar log file ===
Iterating with no regularization...
Begin: 1.1857e+07
0 1.10152e+07
1 1.0998e+07
2 1.09966e+07
Now with regularization...
Begin: 7.60672e+06
0 7.6054e+06
---MC-PARTICLE-MONITORING-----------------------------------------------------------------------

BeamNeutrinos: 

--Primary 0, MCPDG 22, Energy 0.253446, Dist. 2.5872, nMCHits 321 (132, 103, 86)
MCPDG 22, Energy 0.253446, Dist. 2.5872, nMCHits 321 (132, 103, 86)

--Primary 1, MCPDG 2212, Energy 1.18097, Dist. 35.4274, nMCHits 172 (50, 59, 63)
MCPDG 2212, Energy 1.18097, Dist. 35.4274, nMCHits 172 (50, 59, 63)
------------------------------------------------------------------------------------------------
Operating in training mode.
The eid is 199
Graph saved to training1_CaloHitListW_graph.data
Size of file training1_CaloHitListW_graph.data is 2239216 bytes.
The eid is 199
Graph saved to training1_CaloHitListU_graph.data
Size of file training1_CaloHitListU_graph.data is 2874260 bytes.
The eid is 199
Graph saved to training1_CaloHitListV_graph.data
Size of file training1_CaloHitListV_graph.data is 3020088 bytes.
Operating in inference mode.
Operating in training mode.
The eid is -1
Graph saved to training2_CaloHitListW_graph.data
Size of file training2_CaloHitListW_graph.data is 2235372 bytes.
The eid is -1
Graph saved to training2_CaloHitListU_graph.data
Size of file training2_CaloHitListU_graph.data is 2868932 bytes.
The eid is -1
Graph saved to training2_CaloHitListV_graph.data
Size of file training2_CaloHitListV_graph.data is 3013344 bytes.
Boundary wire vector sizes: 186, 168, 152
minwire 0: 194
minwire 1: 2221
minwire 2: 204
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.00966839, 
Output 1: 0.000748222, 0.0544986, 0.0193611, 0.925392, 
Output 2: 0.80472, 0.152937, 0.00526642, 0.0370768, 
Output 3: 0.00477405, 0.986437, 0.00878748, 1.65998e-06, 
Output 4: 0.989793, 0.00963663, 0.000442427, 0.00012761, 
Output 5: 0.238716, 0.752568, 0.00835685, 0.000359497, 
Output 6: 0.657757, 0.331079, 0.010663, 0.000501651, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 123!
02-Dec-2025 18:32:08 -03  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50253389_320_20231120T134814Z_gen_g4_detsim_hitreco__20240503T060637Z_reco2_graph_2025-12-02T_171434Z.root"
02-Dec-2025 18:32:08 -03  Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/4f/9d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50253389_320_20231120T134814Z_gen_g4_detsim_hitreco__20240503T060637Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              5.64881       9.42836       31.1056       9.36541       3.08672        200    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.232105      0.359387       0.50618      0.267059      0.116554        200    
reco:gaushit:GausHitFinder                             0.716391      0.823871       1.84277      0.757975      0.170876        200    
reco:spsolve:SpacePointSolver                         0.000142797    0.261125       20.8295      0.007937       1.72737        200    
reco:hitfd:DisambigFromSpacePoints                    0.000279833    0.0236021     0.944728      0.0018336     0.0853953       200    
reco:rns:RandomNumberSaver                            2.2741e-05    5.35792e-05   0.00408307    2.9881e-05    0.000286825      200    
reco:pandora:StandardPandora                            1.85307       2.27849       12.7061       1.95616       1.09877        200    
reco:pandoraTrack:LArPandoraTrackCreation             0.000170747   0.000236301   0.00271148    0.000216124   0.000180262      200    
reco:pandoraShower:LArPandoraModularShowerCreation    0.00022511    0.000319543   0.00900048    0.000272993   0.00061589       200    
reco:pandoracalo:Calorimetry                          0.000134196   0.00018795    0.00183487    0.000176132   0.000118651      200    
reco:pandorapid:Chi2ParticleID                        4.7372e-05    6.96327e-05   0.000787365   6.4603e-05    5.29393e-05      200    
reco:cvnmap:CVNMapper                                 3.1771e-05     0.0186132     0.0638255     0.0151129     0.0145681       200    
reco:cvneva:CVNEvaluator                              2.2801e-05      1.16464       4.10107       1.54859      0.731802        200    
reco:energyrecnumu:EnergyReco                         0.00339385    0.00740959     0.0228328    0.00583157    0.00457169       200    
reco:energyrecnue:EnergyReco                          0.00230078    0.00523688     0.0208093    0.00408834    0.00300522       200    
reco:energyrecnc:EnergyReco                           0.00212368    0.00552148     0.0253874    0.00406901    0.00374025       200    
reco:energyrecnumurange:EnergyReco                    0.00309367    0.00508918     0.0208522    0.00405075    0.00286055       200    
reco:energyrecnumumcs:EnergyReco                      0.00295247    0.00480951     0.0170744    0.00398943    0.00221645       200    
reco:opdec:Deconvolution                                0.71341       1.74348       2.78612       1.83819      0.356155        200    
reco:ophitspe:OpHitFinderDeco                           2.07937       2.10671       3.57951       2.09322      0.118198        200    
reco:opflash:OpFlashFinder                            0.000168268   0.000493466   0.00374229    0.000460575   0.000286683      200    
reco:opslicer:OpSlicer                                5.1842e-05     0.0276504     0.151748      0.0150146     0.0324039       200    
[art]:TriggerResults:TriggerResultInserter            1.3901e-05    2.16475e-05   7.3333e-05    2.07505e-05   6.62287e-06      200    
end_path:out1:RootOutput                                3.2e-06     4.34165e-06   2.6991e-05    4.2005e-06    1.64991e-06      200    
end_path:out1:RootOutput(write)                        0.0126277     0.589987       1.87916      0.685125      0.364586        200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 20698.6 MB
  Peak resident set size usage (VmHWM): 1952.4 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Generated output files ===
256474.1_dunegpschedd02.fnal.gov.logs.tgz
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50253389_320_20231120T134814Z_gen_g4_detsim_hitreco__20240503T060637Z_reco2_graph_2025-12-02T_171434Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50253389_320_20231120T134814Z_gen_g4_detsim_hitreco__20240503T060637Z_reco2_graph_2025-12-02T_171434Z.root
debugprod.log
graph_output_2025-12-02T_171434Z_2_ana_tree_hd.root
graph_output_2025-12-02T_171434Z_2_analysiseid.root
graph_output_2025-12-02T_171434Z_2_training1_CaloHitListU_graph.data
graph_output_2025-12-02T_171434Z_2_training1_CaloHitListV_graph.data
graph_output_2025-12-02T_171434Z_2_training1_CaloHitListW_graph.data
graph_output_2025-12-02T_171434Z_2_training2_CaloHitListU_graph.data
graph_output_2025-12-02T_171434Z_2_training2_CaloHitListV_graph.data
graph_output_2025-12-02T_171434Z_2_training2_CaloHitListW_graph.data
jobscript.log
justin-processed-pfns.txt
reco_hist.root
secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50253389_320_20231120T134814Z_gen_g4_detsim_hitreco__20240503T060637Z_reco2_graph_2025-12-02T_171434Z.log
third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50253389_320_20231120T134814Z_gen_g4_detsim_hitreco__20240503T060637Z_reco2_graph_2025-12-02T_171434Z.log
training1_CaloHitListU.csv
training1_CaloHitListV.csv
training1_CaloHitListW.csv
training2_CaloHitListU.csv
training2_CaloHitListV.csv
training2_CaloHitListW.csv
justIN time: 2025-12-19 07:12:01 UTC       justIN version: 01.05.03