Jobsub ID 20538.1@dunegpschedd01.fnal.gov
| Jobsub ID | 20538.1@dunegpschedd01.fnal.gov | |
| Workflow ID | 253 | |
| Stage ID | 1 | |
| User name | ichong@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-08-02 23:11:48 | |
| Site | NL_SURFsara | |
| Entry | DUNE_SurfSARA_arc02 | |
| Last heartbeat | 2025-08-02 23:24:27 | |
| From worker node | Hostname | wn-lb-01.gina.surf.nl |
| cpuinfo | AMD EPYC 9754 128-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 129600 (36 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | finished | |
| Started | 2025-08-02 23:13:49 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_780_20231202T162805Z_gen_g4_detsim_hitreco__20240508T045723Z_reco2.root | |
| Jobscript | Exit code | 0 |
| Real time | 9m (566s) | |
| CPU time | 8m (523s = 92%) | |
| Max RSS bytes | 2184474624 (2083 MiB) | |
| Outputting started | 2025-08-02 23:23:16 | |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_231402Z_6_training1_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_231402Z_6_training1_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_231402Z_6_training1_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_231402Z_6_training2_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_231402Z_6_training2_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_231402Z_6_training2_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_231402Z_6_analysiseid.root https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_231402Z_6_ana_tree_hd.root | |
| Finished | 2025-08-02 23:24:27 | |
| Saved logs | justin-logs:20538.1-dunegpschedd01.fnal.gov.logs.tgz | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
tListV_graph.data -> graph_output_2025-08-02T_231402Z_6_training1_CaloHitListV_graph.data Renamed training1_CaloHitListW_graph.data -> graph_output_2025-08-02T_231402Z_6_training1_CaloHitListW_graph.data Renamed training2_CaloHitListU_graph.data -> graph_output_2025-08-02T_231402Z_6_training2_CaloHitListU_graph.data Renamed training2_CaloHitListV_graph.data -> graph_output_2025-08-02T_231402Z_6_training2_CaloHitListV_graph.data Renamed training2_CaloHitListW_graph.data -> graph_output_2025-08-02T_231402Z_6_training2_CaloHitListW_graph.data Renamed analysiseid.root -> graph_output_2025-08-02T_231402Z_6_analysiseid.root Renamed ana_tree_hd.root -> graph_output_2025-08-02T_231402Z_6_ana_tree_hd.root === Start last 100 lines of lar log file === \_ MCPDG 2212, Energy 1.03702, Dist. 7.75252, nMCHits 15 (0, 10, 5) \_ MCPDG -211, Energy 0.542889, Dist. 161.655, nMCHits 660 (327, 85, 248) \_ MCPDG 2212, Energy 0.999059, Dist. 3.24137, nMCHits 24 (10, 6, 8) \_ MCPDG -211, Energy 0.164871, Dist. 2.92422, nMCHits 12 (3, 4, 5) \_ MCPDG 1000170370, Energy 34.4253, Dist. 0.00014314, nMCHits 2 (1, 1, 0) \_ MCPDG 11, Energy 0.000515634, Dist. 3.05176e-05, nMCHits 2 (1, 1, 0) --Primary 2, MCPDG -211, Energy 0.434161, Dist. 17.8835, nMCHits 207 (102, 40, 65) MCPDG -211, Energy 0.434161, Dist. 17.8835, nMCHits 56 (29, 14, 13) \_ MCPDG -211, Energy 0.266031, Dist. 37.1177, nMCHits 150 (72, 26, 52) \_ MCPDG 2212, Energy 0.946859, Dist. 0.102987, nMCHits 1 (1, 0, 0) --Primary 3, MCPDG 2212, Energy 1.20183, Dist. 41.9001, nMCHits 103 (50, 46, 7) MCPDG 2212, Energy 1.20183, Dist. 41.9001, nMCHits 103 (50, 46, 7) --Primary 4, MCPDG 211, Energy 0.165118, Dist. 2.96991, nMCHits 53 (30, 11, 12) MCPDG 211, Energy 0.165118, Dist. 2.96991, nMCHits 7 (2, 2, 3) \_ MCPDG -11, Energy 0.0425359, Dist. 14.3966, nMCHits 46 (28, 9, 9) ------------------------------------------------------------------------------------------------ Operating in training mode. The eid is 98 Graph saved to training1_CaloHitListW_graph.data Size of file training1_CaloHitListW_graph.data is 1220632 bytes. The eid is 98 Graph saved to training1_CaloHitListU_graph.data Size of file training1_CaloHitListU_graph.data is 1640472 bytes. The eid is 98 Graph saved to training1_CaloHitListV_graph.data Size of file training1_CaloHitListV_graph.data is 1617576 bytes. Operating in inference mode. Operating in training mode. The eid is -1 Graph saved to training2_CaloHitListW_graph.data Size of file training2_CaloHitListW_graph.data is 1220632 bytes. The eid is -1 Graph saved to training2_CaloHitListU_graph.data Size of file training2_CaloHitListU_graph.data is 1638952 bytes. The eid is -1 Graph saved to training2_CaloHitListV_graph.data Size of file training2_CaloHitListV_graph.data is 1616480 bytes. Boundary wire vector sizes: 2254, 1920, 728 minwire 0: 126 minwire 1: 1717 minwire 2: 23 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Classifier summary: Output 0: 0.123124, Output 1: 0.926274, 0.0154484, 0.0208581, 0.0374195, Output 2: 0.0026463, 0.790292, 0.199904, 0.00715825, Output 3: 0.333746, 0.196886, 0.18382, 0.285547, Output 4: 0.156478, 0.618036, 0.215083, 0.0104031, Output 5: 0.669629, 0.3247, 0.00557863, 9.25528e-05, Output 6: 0.00909142, 0.097798, 0.224182, 0.668929, Running Ophitfinder with InputDigiType = 'recob' Found hits: 335! 03-Aug-2025 01:21:51 CEST Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_780_20231202T162805Z_gen_g4_detsim_hitreco__20240508T045723Z_reco2_graph_2025-08-02T_231402Z.root" 03-Aug-2025 01:21:51 CEST Closed input file "root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/69/c6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_780_20231202T162805Z_gen_g4_detsim_hitreco__20240508T045723Z_reco2.root" ======================================================================================================================================== TimeTracker printout (sec) Min Avg Max Median RMS nEvts ======================================================================================================================================== Full event 0.0902212 3.44349 22.4317 3.2481 2.5347 100 ---------------------------------------------------------------------------------------------------------------------------------------- source:RootInput(read) 0.00292947 0.00550658 0.0249128 0.00521447 0.00251171 100 reco:gaushit:GausHitFinder 0.0193574 0.0754141 0.733759 0.0298332 0.115818 100 reco:spsolve:SpacePointSolver 0.000135694 0.126568 4.42632 0.00476592 0.51317 100 reco:hitfd:DisambigFromSpacePoints 0.000260251 0.0154739 0.37232 0.00121485 0.0517394 100 reco:rns:RandomNumberSaver 2.4827e-05 3.85632e-05 0.000411107 3.1948e-05 3.83978e-05 100 reco:pandora:StandardPandora 0.0177187 1.69178 14.3188 1.33433 1.52075 100 reco:pandoraTrack:LArPandoraTrackCreation 0.00015371 0.0002403 0.00401604 0.000190035 0.000383225 100 reco:pandoraShower:LArPandoraModularShowerCreation 0.000202193 0.00025555 0.00324105 0.000219489 0.000301003 100 reco:pandoracalo:Calorimetry 0.000132319 0.000177312 0.00163819 0.000157631 0.000148583 100 reco:pandorapid:Chi2ParticleID 4.2433e-05 5.48657e-05 0.000775554 4.59585e-05 7.26711e-05 100 reco:cvnmap:CVNMapper 2.8513e-05 0.0140641 0.0613009 0.0140257 0.0106384 100 reco:cvneva:CVNEvaluator 2.0831e-05 0.869933 3.44055 1.09525 0.541945 100 reco:energyrecnumu:EnergyReco 0.00215859 0.00577794 0.0131238 0.00584833 0.00289788 100 reco:energyrecnue:EnergyReco 0.00217855 0.00308746 0.0160934 0.00253091 0.00186906 100 reco:energyrecnc:EnergyReco 0.00216054 0.00309134 0.0297657 0.00240736 0.00294486 100 reco:energyrecnumurange:EnergyReco 0.00215208 0.00286476 0.011194 0.00240456 0.0013354 100 reco:energyrecnumumcs:EnergyReco 0.00215038 0.00312241 0.0157634 0.00238938 0.00208358 100 reco:opdec:Deconvolution 0.0117137 0.182483 0.346851 0.177053 0.0751291 100 reco:ophitspe:OpHitFinderDeco 0.0185309 0.023449 0.0470539 0.0229256 0.00360061 100 reco:opflash:OpFlashFinder 0.000142925 0.000443618 0.00357899 0.00037311 0.000354457 100 reco:opslicer:OpSlicer 4.2944e-05 0.0152467 0.0967498 0.00678615 0.0199785 100 [art]:TriggerResults:TriggerResultInserter 1.3881e-05 1.75229e-05 7.6654e-05 1.62395e-05 6.7952e-06 100 end_path:out1:RootOutput 5.467e-06 6.83465e-06 2.2383e-05 6.45e-06 1.84949e-06 100 end_path:out1:RootOutput(write) 0.00459897 0.403228 0.911561 0.476407 0.210798 100 ======================================================================================================================================== ==================================================================================================== MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 19889.3 MB Peak resident set size usage (VmHWM): 2184.47 MB ==================================================================================================== Art has completed and will exit with status 0. === End last 100 lines of lar log file === === Generated output files === 20538.1_dunegpschedd01.fnal.gov.logs.tgz atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_780_20231202T162805Z_gen_g4_detsim_hitreco__20240508T045723Z_reco2_graph_2025-08-02T_231402Z.log atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_780_20231202T162805Z_gen_g4_detsim_hitreco__20240508T045723Z_reco2_graph_2025-08-02T_231402Z.root debugprod.log graph_output_2025-08-02T_231402Z_6_ana_tree_hd.root graph_output_2025-08-02T_231402Z_6_analysiseid.root graph_output_2025-08-02T_231402Z_6_training1_CaloHitListU_graph.data graph_output_2025-08-02T_231402Z_6_training1_CaloHitListV_graph.data graph_output_2025-08-02T_231402Z_6_training1_CaloHitListW_graph.data graph_output_2025-08-02T_231402Z_6_training2_CaloHitListU_graph.data graph_output_2025-08-02T_231402Z_6_training2_CaloHitListV_graph.data graph_output_2025-08-02T_231402Z_6_training2_CaloHitListW_graph.data jobscript.log justin-processed-pfns.txt reco_hist.root secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_780_20231202T162805Z_gen_g4_detsim_hitreco__20240508T045723Z_reco2_graph_2025-08-02T_231402Z.log third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_780_20231202T162805Z_gen_g4_detsim_hitreco__20240508T045723Z_reco2_graph_2025-08-02T_231402Z.log training1_CaloHitListU.csv training1_CaloHitListV.csv training1_CaloHitListW.csv training2_CaloHitListU.csv training2_CaloHitListV.csv training2_CaloHitListW.csv