Jobsub ID 14065.69@dunegpschedd02.fnal.gov
| Jobsub ID | 14065.69@dunegpschedd02.fnal.gov | |
| Workflow ID | 253 | |
| Stage ID | 1 | |
| User name | ichong@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-08-02 22:37:47 | |
| Site | NL_SURFsara | |
| Entry | DUNE_SurfSARA_arc01 | |
| Last heartbeat | 2025-08-02 22:46:37 | |
| From worker node | Hostname | wn-da-25.gina.surf.nl |
| cpuinfo | AMD EPYC 7702P 64-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 129600 (36 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | finished | |
| Started | 2025-08-02 22:38:54 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_1129_20231204T080326Z_gen_g4_detsim_hitreco__20240509T214123Z_reco2.root | |
| Jobscript | Exit code | 0 |
| Real time | 6m (410s) | |
| CPU time | 6m (386s = 94%) | |
| Max RSS bytes | 2011684864 (1918 MiB) | |
| Outputting started | 2025-08-02 22:45:45 | |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223902Z_3_training1_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223902Z_3_training1_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223902Z_3_training1_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223902Z_3_training2_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223902Z_3_training2_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223902Z_3_training2_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223902Z_3_analysiseid.root https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223902Z_3_ana_tree_hd.root | |
| Finished | 2025-08-02 22:46:37 | |
| Saved logs | justin-logs:14065.69-dunegpschedd02.fnal.gov.logs.tgz | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
============ MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 10071.1 MB Peak resident set size usage (VmHWM): 1030.89 MB ==================================================================================================== Art has completed and will exit with status 0. === End last 100 lines of third lar log file === Renamed training1_CaloHitListU_graph.data -> graph_output_2025-08-02T_223902Z_3_training1_CaloHitListU_graph.data Renamed training1_CaloHitListV_graph.data -> graph_output_2025-08-02T_223902Z_3_training1_CaloHitListV_graph.data Renamed training1_CaloHitListW_graph.data -> graph_output_2025-08-02T_223902Z_3_training1_CaloHitListW_graph.data Renamed training2_CaloHitListU_graph.data -> graph_output_2025-08-02T_223902Z_3_training2_CaloHitListU_graph.data Renamed training2_CaloHitListV_graph.data -> graph_output_2025-08-02T_223902Z_3_training2_CaloHitListV_graph.data Renamed training2_CaloHitListW_graph.data -> graph_output_2025-08-02T_223902Z_3_training2_CaloHitListW_graph.data Renamed analysiseid.root -> graph_output_2025-08-02T_223902Z_3_analysiseid.root Renamed ana_tree_hd.root -> graph_output_2025-08-02T_223902Z_3_ana_tree_hd.root === Start last 100 lines of lar log file === Iterating with no regularization... Begin: 2.82451e+06 0 2.52123e+06 1 2.49344e+06 2 2.48924e+06 3 2.48881e+06 Now with regularization... Begin: 1.22582e+06 0 1.22558e+06 ---MC-PARTICLE-MONITORING----------------------------------------------------------------------- BeamNeutrinos: --Primary 0, MCPDG 13, Energy 0.855723, Dist. 81.8568, nMCHits 353 (58, 163, 132) MCPDG 13, Energy 0.855723, Dist. 81.8568, nMCHits 353 (58, 163, 132) --Primary 1, MCPDG 2212, Energy 1.05705, Dist. 10.3633, nMCHits 32 (6, 15, 11) MCPDG 2212, Energy 1.05705, Dist. 10.3633, nMCHits 32 (6, 15, 11) ------------------------------------------------------------------------------------------------ Operating in training mode. The eid is 98 Graph saved to training1_CaloHitListW_graph.data Size of file training1_CaloHitListW_graph.data is 1107108 bytes. The eid is 98 Graph saved to training1_CaloHitListU_graph.data Size of file training1_CaloHitListU_graph.data is 1404836 bytes. The eid is 98 Graph saved to training1_CaloHitListV_graph.data Size of file training1_CaloHitListV_graph.data is 1438172 bytes. Operating in inference mode. Operating in training mode. The eid is -1 Graph saved to training2_CaloHitListW_graph.data Size of file training2_CaloHitListW_graph.data is 1106276 bytes. The eid is -1 Graph saved to training2_CaloHitListU_graph.data Size of file training2_CaloHitListU_graph.data is 1401304 bytes. The eid is -1 Graph saved to training2_CaloHitListV_graph.data Size of file training2_CaloHitListV_graph.data is 1435216 bytes. Boundary wire vector sizes: 83, 196, 161 minwire 0: 135 minwire 1: 2059 minwire 2: 0 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Classifier summary: Output 0: 0.0231216, Output 1: 0.0134508, 0.000409985, 0.0105064, 0.975633, Output 2: 0.4876, 0.134355, 0.00481165, 0.373233, Output 3: 0.987242, 0.0108305, 0.000733768, 0.00119382, Output 4: 0.008638, 0.989568, 0.00179059, 3.61699e-06, Output 5: 0.997534, 0.00245325, 7.4099e-06, 5.38493e-06, Output 6: 0.209183, 0.749204, 0.0372255, 0.00438802, Running Ophitfinder with InputDigiType = 'recob' Found hits: 76! 03-Aug-2025 00:44:43 CEST Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_1129_20231204T080326Z_gen_g4_detsim_hitreco__20240509T214123Z_reco2_graph_2025-08-02T_223902Z.root" 03-Aug-2025 00:44:43 CEST Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/b9/9d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_1129_20231204T080326Z_gen_g4_detsim_hitreco__20240509T214123Z_reco2.root" ======================================================================================================================================== TimeTracker printout (sec) Min Avg Max Median RMS nEvts ======================================================================================================================================== Full event 0.068359 2.56427 13.2145 2.60639 1.71654 100 ---------------------------------------------------------------------------------------------------------------------------------------- source:RootInput(read) 0.00163418 0.00328275 0.016296 0.00320753 0.00160024 100 reco:gaushit:GausHitFinder 0.0165774 0.0622996 0.692861 0.0263202 0.0988292 100 reco:spsolve:SpacePointSolver 9.5051e-05 0.0537561 1.31397 0.00289712 0.18688 100 reco:hitfd:DisambigFromSpacePoints 0.000191465 0.0117816 0.34369 0.000881803 0.0422353 100 reco:rns:RandomNumberSaver 1.6571e-05 2.56937e-05 0.000310922 2.135e-05 2.91293e-05 100 reco:pandora:StandardPandora 0.0121573 1.24644 9.26284 1.01257 1.02581 100 reco:pandoraTrack:LArPandoraTrackCreation 0.000123144 0.000191573 0.00292054 0.000140322 0.000303793 100 reco:pandoraShower:LArPandoraModularShowerCreation 0.000166217 0.00021327 0.00318269 0.000179086 0.000298701 100 reco:pandoracalo:Calorimetry 0.000102044 0.000135971 0.00142822 0.000117484 0.000131025 100 reco:pandorapid:Chi2ParticleID 3.5988e-05 4.60169e-05 0.000732344 3.8789e-05 6.90055e-05 100 reco:cvnmap:CVNMapper 2.109e-05 0.020941 0.0498955 0.0294285 0.0151344 100 reco:cvneva:CVNEvaluator 1.6641e-05 0.662827 3.21687 0.940515 0.516965 100 reco:energyrecnumu:EnergyReco 0.00195975 0.0045189 0.0190495 0.00311615 0.00351966 100 reco:energyrecnue:EnergyReco 0.00187664 0.00299543 0.016839 0.00247645 0.00188947 100 reco:energyrecnc:EnergyReco 0.00198086 0.00294536 0.0177435 0.00235847 0.00197183 100 reco:energyrecnumurange:EnergyReco 0.00192751 0.00278271 0.00942519 0.00236834 0.00123813 100 reco:energyrecnumumcs:EnergyReco 0.00188691 0.00316075 0.0170261 0.00233154 0.00258663 100 reco:opdec:Deconvolution 0.0081725 0.157904 0.32003 0.152502 0.0664061 100 reco:ophitspe:OpHitFinderDeco 0.0104756 0.0162365 0.0772655 0.0154193 0.0068404 100 reco:opflash:OpFlashFinder 0.000107133 0.000320649 0.00313269 0.00026567 0.0003032 100 reco:opslicer:OpSlicer 3.3754e-05 0.0112231 0.059622 0.00506199 0.0147238 100 [art]:TriggerResults:TriggerResultInserter 9.989e-06 1.27628e-05 5.3652e-05 1.16675e-05 4.84443e-06 100 end_path:out1:RootOutput 3.256e-06 3.90946e-06 1.6802e-05 3.702e-06 1.34793e-06 100 end_path:out1:RootOutput(write) 0.00346694 0.299352 0.651666 0.38962 0.186521 100 ======================================================================================================================================== ==================================================================================================== MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 11600 MB Peak resident set size usage (VmHWM): 2011.68 MB ==================================================================================================== Art has completed and will exit with status 0. === End last 100 lines of lar log file === === Generated output files === 14065.69_dunegpschedd02.fnal.gov.logs.tgz atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_1129_20231204T080326Z_gen_g4_detsim_hitreco__20240509T214123Z_reco2_graph_2025-08-02T_223902Z.log atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_1129_20231204T080326Z_gen_g4_detsim_hitreco__20240509T214123Z_reco2_graph_2025-08-02T_223902Z.root debugprod.log graph_output_2025-08-02T_223902Z_3_ana_tree_hd.root graph_output_2025-08-02T_223902Z_3_analysiseid.root graph_output_2025-08-02T_223902Z_3_training1_CaloHitListU_graph.data graph_output_2025-08-02T_223902Z_3_training1_CaloHitListV_graph.data graph_output_2025-08-02T_223902Z_3_training1_CaloHitListW_graph.data graph_output_2025-08-02T_223902Z_3_training2_CaloHitListU_graph.data graph_output_2025-08-02T_223902Z_3_training2_CaloHitListV_graph.data graph_output_2025-08-02T_223902Z_3_training2_CaloHitListW_graph.data jobscript.log justin-processed-pfns.txt reco_hist.root secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_1129_20231204T080326Z_gen_g4_detsim_hitreco__20240509T214123Z_reco2_graph_2025-08-02T_223902Z.log third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_1129_20231204T080326Z_gen_g4_detsim_hitreco__20240509T214123Z_reco2_graph_2025-08-02T_223902Z.log training1_CaloHitListU.csv training1_CaloHitListV.csv training1_CaloHitListW.csv training2_CaloHitListU.csv training2_CaloHitListV.csv training2_CaloHitListW.csv