Jobsub ID 13546.6@dunegpschedd02.fnal.gov
| Jobsub ID | 13546.6@dunegpschedd02.fnal.gov | |
| Workflow ID | 83 | |
| Stage ID | 1 | |
| User name | ichong@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-08-01 02:23:30 | |
| Site | NL_SURFsara | |
| Entry | DUNE_SurfSARA_arc03 | |
| Last heartbeat | 2025-08-01 02:31:41 | |
| From worker node | Hostname | wn-da-23.gina.surf.nl |
| cpuinfo | AMD EPYC 7702P 64-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 129600 (36 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | finished | |
| Started | 2025-08-01 02:24:00 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_121_20231202T093900Z_gen_g4_detsim_hitreco__20240507T195639Z_reco2.root | |
| Jobscript | Exit code | 0 |
| Real time | 6m (417s) | |
| CPU time | 6m (384s = 92%) | |
| Max RSS bytes | 2013712384 (1920 MiB) | |
| Outputting started | 2025-08-01 02:30:57 | |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00083/1/024/graph_output_2025-08-01T_022411Z_8_training1_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00083/1/024/graph_output_2025-08-01T_022411Z_8_training1_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00083/1/024/graph_output_2025-08-01T_022411Z_8_training1_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00083/1/024/graph_output_2025-08-01T_022411Z_8_training2_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00083/1/024/graph_output_2025-08-01T_022411Z_8_training2_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00083/1/024/graph_output_2025-08-01T_022411Z_8_training2_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00083/1/024/graph_output_2025-08-01T_022411Z_8_analysiseid.root https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00083/1/024/graph_output_2025-08-01T_022411Z_8_ana_tree_hd.root | |
| Finished | 2025-08-01 02:31:41 | |
| Saved logs | justin-logs:13546.6-dunegpschedd02.fnal.gov.logs.tgz | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
======= MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 10078 MB Peak resident set size usage (VmHWM): 1036.11 MB ==================================================================================================== Art has completed and will exit with status 0. === End last 100 lines of third lar log file === Renamed training1_CaloHitListU_graph.data -> graph_output_2025-08-01T_022411Z_8_training1_CaloHitListU_graph.data Renamed training1_CaloHitListV_graph.data -> graph_output_2025-08-01T_022411Z_8_training1_CaloHitListV_graph.data Renamed training1_CaloHitListW_graph.data -> graph_output_2025-08-01T_022411Z_8_training1_CaloHitListW_graph.data Renamed training2_CaloHitListU_graph.data -> graph_output_2025-08-01T_022411Z_8_training2_CaloHitListU_graph.data Renamed training2_CaloHitListV_graph.data -> graph_output_2025-08-01T_022411Z_8_training2_CaloHitListV_graph.data Renamed training2_CaloHitListW_graph.data -> graph_output_2025-08-01T_022411Z_8_training2_CaloHitListW_graph.data Renamed analysiseid.root -> graph_output_2025-08-01T_022411Z_8_analysiseid.root Renamed ana_tree_hd.root -> graph_output_2025-08-01T_022411Z_8_ana_tree_hd.root === Start last 100 lines of lar log file === Neighbour search... 10793 tests to find 5124 neighbours Iterating with no regularization... Begin: 5.3585e+07 0 4.80535e+07 1 4.73786e+07 2 4.73079e+07 3 4.73008e+07 Now with regularization... Begin: 4.09846e+07 0 4.09665e+07 ---MC-PARTICLE-MONITORING----------------------------------------------------------------------- BeamNeutrinos: --Primary 0, MCPDG -11, Energy 0.737575, Dist. 39.5161, nMCHits 796 (257, 326, 213) MCPDG -11, Energy 0.737575, Dist. 39.5161, nMCHits 796 (257, 326, 213) ------------------------------------------------------------------------------------------------ Operating in training mode. The eid is 99 Graph saved to training1_CaloHitListW_graph.data Size of file training1_CaloHitListW_graph.data is 907524 bytes. The eid is 99 Graph saved to training1_CaloHitListU_graph.data Size of file training1_CaloHitListU_graph.data is 1166972 bytes. The eid is 99 Graph saved to training1_CaloHitListV_graph.data Size of file training1_CaloHitListV_graph.data is 1095792 bytes. Operating in inference mode. Operating in training mode. The eid is -1 Graph saved to training2_CaloHitListW_graph.data Size of file training2_CaloHitListW_graph.data is 906892 bytes. The eid is -1 Graph saved to training2_CaloHitListU_graph.data Size of file training2_CaloHitListU_graph.data is 1164744 bytes. The eid is -1 Graph saved to training2_CaloHitListV_graph.data Size of file training2_CaloHitListV_graph.data is 1093368 bytes. Boundary wire vector sizes: 279, 340, 221 minwire 0: 1820 minwire 1: 904 minwire 2: 1766 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Classifier summary: Output 0: 0.260186, Output 1: 0.0140355, 0.888258, 0.059708, 0.037999, Output 2: 0.896258, 0.0245824, 0.0160284, 0.0631308, Output 3: 0.995124, 0.00420371, 5.53928e-05, 0.00061671, Output 4: 0.997833, 0.00120129, 9.15565e-05, 0.000873857, Output 5: 0.999192, 0.00064284, 2.14398e-05, 0.000143307, Output 6: 0.284951, 0.608029, 0.0813213, 0.025699, Running Ophitfinder with InputDigiType = 'recob' Found hits: 245! 01-Aug-2025 04:30:04 CEST Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_121_20231202T093900Z_gen_g4_detsim_hitreco__20240507T195639Z_reco2_graph_2025-08-01T_022411Z.root" 01-Aug-2025 04:30:04 CEST Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/53/a7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_121_20231202T093900Z_gen_g4_detsim_hitreco__20240507T195639Z_reco2.root" ======================================================================================================================================== TimeTracker printout (sec) Min Avg Max Median RMS nEvts ======================================================================================================================================== Full event 1.00849 2.54201 15.8256 2.55813 1.55383 100 ---------------------------------------------------------------------------------------------------------------------------------------- source:RootInput(read) 0.00171528 0.0033031 0.0131335 0.00323911 0.00141862 100 reco:gaushit:GausHitFinder 0.0160387 0.0525655 0.706979 0.0289034 0.0820899 100 reco:spsolve:SpacePointSolver 9.6686e-05 0.0450312 1.70296 0.00415747 0.185894 100 reco:hitfd:DisambigFromSpacePoints 0.000202862 0.00839998 0.442942 0.00128946 0.0443887 100 reco:rns:RandomNumberSaver 1.6953e-05 2.81318e-05 0.000310939 2.3265e-05 2.899e-05 100 reco:pandora:StandardPandora 0.892811 1.13645 10.7546 0.983064 0.98878 100 reco:pandoraTrack:LArPandoraTrackCreation 0.000121373 0.000187222 0.00279583 0.000140696 0.000271485 100 reco:pandoraShower:LArPandoraModularShowerCreation 0.000168405 0.000231392 0.00400055 0.000187341 0.000379508 100 reco:pandoracalo:Calorimetry 0.000101035 0.000144613 0.00165759 0.000123282 0.000153517 100 reco:pandorapid:Chi2ParticleID 3.7312e-05 5.05409e-05 0.000831702 4.086e-05 7.8729e-05 100 reco:cvnmap:CVNMapper 2.2974e-05 0.0172581 0.0586807 0.020889 0.0104944 100 reco:cvneva:CVNEvaluator 1.5951e-05 0.747858 3.17098 0.935462 0.465301 100 reco:energyrecnumu:EnergyReco 0.00206151 0.00485794 0.0185793 0.00402584 0.0031911 100 reco:energyrecnue:EnergyReco 0.0019699 0.00274286 0.0130939 0.00249302 0.00120243 100 reco:energyrecnc:EnergyReco 0.00201864 0.00268063 0.0129911 0.00235085 0.00119892 100 reco:energyrecnumurange:EnergyReco 0.0020211 0.00269061 0.0131896 0.00241195 0.00121562 100 reco:energyrecnumumcs:EnergyReco 0.00199657 0.00301823 0.0186657 0.00241569 0.00241162 100 reco:opdec:Deconvolution 0.0299035 0.158334 0.320236 0.155908 0.0559588 100 reco:ophitspe:OpHitFinderDeco 0.0100038 0.0168946 0.0468984 0.016195 0.00481665 100 reco:opflash:OpFlashFinder 0.000173034 0.000348032 0.0032618 0.000303619 0.000307514 100 reco:opslicer:OpSlicer 4.3243e-05 0.00995647 0.0589871 0.00552124 0.0113179 100 [art]:TriggerResults:TriggerResultInserter 1.018e-05 1.47486e-05 7.5195e-05 1.32305e-05 6.93774e-06 100 end_path:out1:RootOutput 3.226e-06 4.72129e-06 2.1612e-05 4.528e-06 1.82995e-06 100 end_path:out1:RootOutput(write) 0.0141431 0.327943 0.746015 0.391248 0.162557 100 ======================================================================================================================================== ==================================================================================================== MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 11604.4 MB Peak resident set size usage (VmHWM): 2013.71 MB ==================================================================================================== Art has completed and will exit with status 0. === End last 100 lines of lar log file === === Generated output files === 13546.6_dunegpschedd02.fnal.gov.logs.tgz atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_121_20231202T093900Z_gen_g4_detsim_hitreco__20240507T195639Z_reco2_graph_2025-08-01T_022411Z.log atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_121_20231202T093900Z_gen_g4_detsim_hitreco__20240507T195639Z_reco2_graph_2025-08-01T_022411Z.root debugprod.log graph_output_2025-08-01T_022411Z_8_ana_tree_hd.root graph_output_2025-08-01T_022411Z_8_analysiseid.root graph_output_2025-08-01T_022411Z_8_training1_CaloHitListU_graph.data graph_output_2025-08-01T_022411Z_8_training1_CaloHitListV_graph.data graph_output_2025-08-01T_022411Z_8_training1_CaloHitListW_graph.data graph_output_2025-08-01T_022411Z_8_training2_CaloHitListU_graph.data graph_output_2025-08-01T_022411Z_8_training2_CaloHitListV_graph.data graph_output_2025-08-01T_022411Z_8_training2_CaloHitListW_graph.data jobscript.log justin-processed-pfns.txt reco_hist.root secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_121_20231202T093900Z_gen_g4_detsim_hitreco__20240507T195639Z_reco2_graph_2025-08-01T_022411Z.log third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_121_20231202T093900Z_gen_g4_detsim_hitreco__20240507T195639Z_reco2_graph_2025-08-01T_022411Z.log training1_CaloHitListU.csv training1_CaloHitListV.csv training1_CaloHitListW.csv training2_CaloHitListU.csv training2_CaloHitListV.csv training2_CaloHitListW.csv