Jobsub ID 14065.60@dunegpschedd02.fnal.gov
| Jobsub ID | 14065.60@dunegpschedd02.fnal.gov | |
| Workflow ID | 253 | |
| Stage ID | 1 | |
| User name | ichong@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-08-02 22:37:47 | |
| Site | UK_RAL-Tier1 | |
| Entry | LIGO_UK_RAL_arc_ce03 | |
| Last heartbeat | 2025-08-02 22:55:14 | |
| From worker node | Hostname | dune001-7882708.0-lcg2601.gridpp.rl.ac.uk |
| cpuinfo | AMD EPYC 7763 64-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 216000 (60 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | finished | |
| Started | 2025-08-02 22:39:11 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_521_20231203T175734Z_gen_g4_detsim_hitreco__20240509T215645Z_reco2.root | |
| Jobscript | Exit code | 0 |
| Real time | 15m (909s) | |
| CPU time | 11m (688s = 75%) | |
| Max RSS bytes | 1888817152 (1801 MiB) | |
| Outputting started | 2025-08-02 22:54:21 | |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223922Z_3_training1_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223922Z_3_training1_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223922Z_3_training1_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223922Z_3_training2_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223922Z_3_training2_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223922Z_3_training2_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223922Z_3_analysiseid.root https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223922Z_3_ana_tree_hd.root | |
| Finished | 2025-08-02 22:55:14 | |
| Saved logs | justin-logs:14065.60-dunegpschedd02.fnal.gov.logs.tgz | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
================================================================== MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 36422.8 MB Peak resident set size usage (VmHWM): 991.261 MB ==================================================================================================== Art has completed and will exit with status 0. === End last 100 lines of third lar log file === Renamed training1_CaloHitListU_graph.data -> graph_output_2025-08-02T_223922Z_3_training1_CaloHitListU_graph.data Renamed training1_CaloHitListV_graph.data -> graph_output_2025-08-02T_223922Z_3_training1_CaloHitListV_graph.data Renamed training1_CaloHitListW_graph.data -> graph_output_2025-08-02T_223922Z_3_training1_CaloHitListW_graph.data Renamed training2_CaloHitListU_graph.data -> graph_output_2025-08-02T_223922Z_3_training2_CaloHitListU_graph.data Renamed training2_CaloHitListV_graph.data -> graph_output_2025-08-02T_223922Z_3_training2_CaloHitListV_graph.data Renamed training2_CaloHitListW_graph.data -> graph_output_2025-08-02T_223922Z_3_training2_CaloHitListW_graph.data Renamed analysiseid.root -> graph_output_2025-08-02T_223922Z_3_analysiseid.root Renamed ana_tree_hd.root -> graph_output_2025-08-02T_223922Z_3_ana_tree_hd.root === Start last 100 lines of lar log file === Begin: 1.58529e+06 0 1.04884e+06 1 983753 2 966039 3 960112 4 959272 Now with regularization... Begin: 615442 0 614229 1 613899 ---MC-PARTICLE-MONITORING----------------------------------------------------------------------- BeamNeutrinos: --Primary 0, MCPDG -11, Energy 0.0699128, Dist. 12.8483, nMCHits 96 (43, 27, 26) MCPDG -11, Energy 0.0699128, Dist. 12.8483, nMCHits 96 (43, 27, 26) ------------------------------------------------------------------------------------------------ Operating in training mode. The eid is 99 Graph saved to training1_CaloHitListW_graph.data Size of file training1_CaloHitListW_graph.data is 1059384 bytes. The eid is 99 Graph saved to training1_CaloHitListU_graph.data Size of file training1_CaloHitListU_graph.data is 1534220 bytes. The eid is 99 Graph saved to training1_CaloHitListV_graph.data Size of file training1_CaloHitListV_graph.data is 1420624 bytes. Operating in inference mode. Operating in training mode. The eid is -1 Graph saved to training2_CaloHitListW_graph.data Size of file training2_CaloHitListW_graph.data is 1059384 bytes. The eid is -1 Graph saved to training2_CaloHitListU_graph.data Size of file training2_CaloHitListU_graph.data is 1533696 bytes. The eid is -1 Graph saved to training2_CaloHitListV_graph.data Size of file training2_CaloHitListV_graph.data is 1419776 bytes. Boundary wire vector sizes: 53, 71, 50 minwire 0: 1294 minwire 1: 711 minwire 2: 1497 Used alternate method to get min and max wires due to vertex determination failure: 0, 499 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Used alternate method to get min and max wires due to vertex determination failure: 0, 499 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Classifier summary: Output 0: 0.108166, Output 1: 0.000413783, 0.004931, 0.00293082, 0.991724, Output 2: 0.372797, 0.481396, 0.00531502, 0.140492, Output 3: 0.801079, 0.196101, 0.002604, 0.000215362, Output 4: 0.991769, 0.00810652, 0.000111876, 1.21997e-05, Output 5: 0.0508296, 0.928507, 0.0202803, 0.000382814, Output 6: 0.0366543, 0.749267, 0.190419, 0.02366, Running Ophitfinder with InputDigiType = 'recob' Found hits: 74! 02-Aug-2025 22:52:15 UTC Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_521_20231203T175734Z_gen_g4_detsim_hitreco__20240509T215645Z_reco2_graph_2025-08-02T_223922Z.root" 02-Aug-2025 22:52:15 UTC Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/47/36/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_521_20231203T175734Z_gen_g4_detsim_hitreco__20240509T215645Z_reco2.root" ======================================================================================================================================== TimeTracker printout (sec) Min Avg Max Median RMS nEvts ======================================================================================================================================== Full event 1.40192 4.98889 19.241 4.72344 2.93591 100 ---------------------------------------------------------------------------------------------------------------------------------------- source:RootInput(read) 0.00540779 0.0105112 0.0177225 0.0097371 0.00305089 100 reco:gaushit:GausHitFinder 0.0312829 0.110798 0.801706 0.0600879 0.133683 100 reco:spsolve:SpacePointSolver 0.000175643 0.300037 11.9251 0.0051285 1.3613 100 reco:hitfd:DisambigFromSpacePoints 0.000317256 0.0267595 0.900343 0.00151246 0.104946 100 reco:rns:RandomNumberSaver 2.844e-05 5.39808e-05 0.000679702 4.3351e-05 6.46482e-05 100 reco:pandora:StandardPandora 1.04135 2.12791 12.1818 1.84323 1.30581 100 reco:pandoraTrack:LArPandoraTrackCreation 0.000150173 0.000461939 0.0145297 0.000258974 0.00143254 100 reco:pandoraShower:LArPandoraModularShowerCreation 0.000183123 0.000435176 0.00872017 0.000314416 0.000867637 100 reco:pandoracalo:Calorimetry 0.000130182 0.000268448 0.0037283 0.000219804 0.000356626 100 reco:pandorapid:Chi2ParticleID 4.4271e-05 9.62133e-05 0.00218124 7.0541e-05 0.000210523 100 reco:cvnmap:CVNMapper 6.6481e-05 0.0623245 0.16273 0.0615825 0.0456788 100 reco:cvneva:CVNEvaluator 2.085e-05 1.47187 6.52549 1.72668 0.985671 100 reco:energyrecnumu:EnergyReco 0.0020399 0.00684612 0.021746 0.00604937 0.00368247 100 reco:energyrecnue:EnergyReco 0.00182136 0.00441465 0.010933 0.0040625 0.00162183 100 reco:energyrecnc:EnergyReco 0.00173082 0.004266 0.0111442 0.00392223 0.00158659 100 reco:energyrecnumurange:EnergyReco 0.00175048 0.00423136 0.0102671 0.00381528 0.00156608 100 reco:energyrecnumumcs:EnergyReco 0.00176703 0.00422995 0.0114334 0.00377762 0.00159952 100 reco:opdec:Deconvolution 0.0705377 0.223416 0.421802 0.213462 0.0790356 100 reco:ophitspe:OpHitFinderDeco 0.0332549 0.0462361 0.101247 0.0447568 0.00785361 100 reco:opflash:OpFlashFinder 0.000262694 0.000738355 0.0128223 0.00057591 0.00123569 100 reco:opslicer:OpSlicer 6.7631e-05 0.0175807 0.0964559 0.0106275 0.0215481 100 [art]:TriggerResults:TriggerResultInserter 1.642e-05 3.16988e-05 0.000187524 2.6846e-05 1.97704e-05 100 end_path:out1:RootOutput 4.49e-06 9.16553e-06 8.3562e-05 6.295e-06 9.51789e-06 100 end_path:out1:RootOutput(write) 0.029661 0.563676 0.986521 0.678881 0.297874 100 ======================================================================================================================================== ==================================================================================================== MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 37952 MB Peak resident set size usage (VmHWM): 1888.82 MB ==================================================================================================== Art has completed and will exit with status 0. === End last 100 lines of lar log file === === Generated output files === 14065.60_dunegpschedd02.fnal.gov.logs.tgz atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_521_20231203T175734Z_gen_g4_detsim_hitreco__20240509T215645Z_reco2_graph_2025-08-02T_223922Z.log atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_521_20231203T175734Z_gen_g4_detsim_hitreco__20240509T215645Z_reco2_graph_2025-08-02T_223922Z.root debugprod.log graph_output_2025-08-02T_223922Z_3_ana_tree_hd.root graph_output_2025-08-02T_223922Z_3_analysiseid.root graph_output_2025-08-02T_223922Z_3_training1_CaloHitListU_graph.data graph_output_2025-08-02T_223922Z_3_training1_CaloHitListV_graph.data graph_output_2025-08-02T_223922Z_3_training1_CaloHitListW_graph.data graph_output_2025-08-02T_223922Z_3_training2_CaloHitListU_graph.data graph_output_2025-08-02T_223922Z_3_training2_CaloHitListV_graph.data graph_output_2025-08-02T_223922Z_3_training2_CaloHitListW_graph.data jobscript.log justin-processed-pfns.txt reco_hist.root secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_521_20231203T175734Z_gen_g4_detsim_hitreco__20240509T215645Z_reco2_graph_2025-08-02T_223922Z.log third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_521_20231203T175734Z_gen_g4_detsim_hitreco__20240509T215645Z_reco2_graph_2025-08-02T_223922Z.log training1_CaloHitListU.csv training1_CaloHitListV.csv training1_CaloHitListW.csv training2_CaloHitListU.csv training2_CaloHitListV.csv training2_CaloHitListW.csv