Jobsub ID 14065.24@dunegpschedd02.fnal.gov
| Jobsub ID | 14065.24@dunegpschedd02.fnal.gov | |
| Workflow ID | 253 | |
| Stage ID | 1 | |
| User name | ichong@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-08-02 22:37:47 | |
| Site | NL_SURFsara | |
| Entry | DUNE_SurfSARA_arc01 | |
| Last heartbeat | 2025-08-02 22:48:09 | |
| From worker node | Hostname | wn-la-03.gina.surf.nl |
| cpuinfo | AMD EPYC 9754 128-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 129600 (36 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | finished | |
| Started | 2025-08-02 22:38:51 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481123_457_20231201T150239Z_gen_g4_detsim_hitreco__20240507T203609Z_reco2.root | |
| Jobscript | Exit code | 0 |
| Real time | 8m (501s) | |
| CPU time | 7m (463s = 92%) | |
| Max RSS bytes | 2156564480 (2056 MiB) | |
| Outputting started | 2025-08-02 22:47:13 | |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223902Z_4_training1_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223902Z_4_training1_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223902Z_4_training1_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223902Z_4_training2_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223902Z_4_training2_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223902Z_4_training2_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223902Z_4_analysiseid.root https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223902Z_4_ana_tree_hd.root | |
| Finished | 2025-08-02 22:48:09 | |
| Saved logs | justin-logs:14065.24-dunegpschedd02.fnal.gov.logs.tgz | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
cker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 18033.5 MB Peak resident set size usage (VmHWM): 949.576 MB ==================================================================================================== Art has completed and will exit with status 0. === End last 100 lines of third lar log file === Renamed training1_CaloHitListU_graph.data -> graph_output_2025-08-02T_223902Z_4_training1_CaloHitListU_graph.data Renamed training1_CaloHitListV_graph.data -> graph_output_2025-08-02T_223902Z_4_training1_CaloHitListV_graph.data Renamed training1_CaloHitListW_graph.data -> graph_output_2025-08-02T_223902Z_4_training1_CaloHitListW_graph.data Renamed training2_CaloHitListU_graph.data -> graph_output_2025-08-02T_223902Z_4_training2_CaloHitListU_graph.data Renamed training2_CaloHitListV_graph.data -> graph_output_2025-08-02T_223902Z_4_training2_CaloHitListV_graph.data Renamed training2_CaloHitListW_graph.data -> graph_output_2025-08-02T_223902Z_4_training2_CaloHitListW_graph.data Renamed analysiseid.root -> graph_output_2025-08-02T_223902Z_4_analysiseid.root Renamed ana_tree_hd.root -> graph_output_2025-08-02T_223902Z_4_ana_tree_hd.root === Start last 100 lines of lar log file === 129 potential space points Neighbour search... 2065 tests to find 984 neighbours Iterating with no regularization... Begin: 6.48359e+06 0 5.53043e+06 1 5.48629e+06 2 5.48287e+06 Now with regularization... Begin: 4.08295e+06 0 4.08166e+06 ---MC-PARTICLE-MONITORING----------------------------------------------------------------------- BeamNeutrinos: --Primary 0, MCPDG 11, Energy 0.3088, Dist. 26.8387, nMCHits 359 (154, 95, 110) MCPDG 11, Energy 0.3088, Dist. 26.8387, nMCHits 359 (154, 95, 110) ------------------------------------------------------------------------------------------------ Operating in training mode. The eid is 99 Graph saved to training1_CaloHitListW_graph.data Size of file training1_CaloHitListW_graph.data is 1130684 bytes. The eid is 99 Graph saved to training1_CaloHitListU_graph.data Size of file training1_CaloHitListU_graph.data is 1559000 bytes. The eid is 99 Graph saved to training1_CaloHitListV_graph.data Size of file training1_CaloHitListV_graph.data is 1511492 bytes. Operating in inference mode. Operating in training mode. The eid is -1 Graph saved to training2_CaloHitListW_graph.data Size of file training2_CaloHitListW_graph.data is 1129220 bytes. The eid is -1 Graph saved to training2_CaloHitListU_graph.data Size of file training2_CaloHitListU_graph.data is 1556504 bytes. The eid is -1 Graph saved to training2_CaloHitListV_graph.data Size of file training2_CaloHitListV_graph.data is 1509384 bytes. Boundary wire vector sizes: 120, 177, 128 minwire 0: 1548 minwire 1: 694 minwire 2: 1758 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Classifier summary: Output 0: 0.0118979, Output 1: 0.00307416, 0.992053, 0.00194929, 0.00292363, Output 2: 0.975149, 0.0100731, 0.00757695, 0.00720102, Output 3: 0.745613, 0.251024, 0.00318846, 0.000174919, Output 4: 0.991844, 0.00696725, 0.000224564, 0.00096383, Output 5: 0.990154, 0.00947832, 4.87916e-05, 0.000319255, Output 6: 0.103804, 0.817274, 0.0657963, 0.0131256, Running Ophitfinder with InputDigiType = 'recob' Found hits: 189! 03-Aug-2025 00:46:05 CEST Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481123_457_20231201T150239Z_gen_g4_detsim_hitreco__20240507T203609Z_reco2_graph_2025-08-02T_223902Z.root" 03-Aug-2025 00:46:05 CEST Closed input file "root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/73/53/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481123_457_20231201T150239Z_gen_g4_detsim_hitreco__20240507T203609Z_reco2.root" ======================================================================================================================================== TimeTracker printout (sec) Min Avg Max Median RMS nEvts ======================================================================================================================================== Full event 1.29763 3.13123 13.125 3.06674 1.84969 100 ---------------------------------------------------------------------------------------------------------------------------------------- source:RootInput(read) 0.00199409 0.00445026 0.0174773 0.00409375 0.00180044 100 reco:gaushit:GausHitFinder 0.0165082 0.0689652 0.624413 0.0286976 0.102622 100 reco:spsolve:SpacePointSolver 0.000128661 0.106966 3.4876 0.00432459 0.401914 100 reco:hitfd:DisambigFromSpacePoints 0.000256513 0.0156852 0.412535 0.000897265 0.0536276 100 reco:rns:RandomNumberSaver 2.2423e-05 3.38074e-05 0.000463584 2.669e-05 4.37408e-05 100 reco:pandora:StandardPandora 1.18708 1.53126 8.1347 1.24856 0.963098 100 reco:pandoraTrack:LArPandoraTrackCreation 0.000152759 0.000327064 0.0139539 0.000175633 0.00136999 100 reco:pandoraShower:LArPandoraModularShowerCreation 0.000196674 0.000249362 0.00334839 0.000211142 0.000311893 100 reco:pandoracalo:Calorimetry 0.000129755 0.000167791 0.00156219 0.000145728 0.000141789 100 reco:pandorapid:Chi2ParticleID 4.1862e-05 5.53117e-05 0.000763333 4.5242e-05 7.16676e-05 100 reco:cvnmap:CVNMapper 2.9634e-05 0.0101132 0.0454483 0.0123767 0.00759966 100 reco:cvneva:CVNEvaluator 2.1312e-05 0.783486 3.17832 1.08174 0.553517 100 reco:energyrecnumu:EnergyReco 0.00210882 0.0053346 0.0206119 0.00463625 0.00351393 100 reco:energyrecnue:EnergyReco 0.00217586 0.00292376 0.00954675 0.00248463 0.0012239 100 reco:energyrecnc:EnergyReco 0.00202106 0.00282611 0.00958196 0.00238232 0.0012261 100 reco:energyrecnumurange:EnergyReco 0.00217571 0.0028793 0.00954373 0.00236323 0.00127051 100 reco:energyrecnumumcs:EnergyReco 0.00216867 0.00280981 0.00952489 0.00234212 0.00121858 100 reco:opdec:Deconvolution 0.0243808 0.181042 0.329963 0.1733 0.0736567 100 reco:ophitspe:OpHitFinderDeco 0.00996332 0.0214166 0.0364498 0.0214743 0.00483325 100 reco:opflash:OpFlashFinder 0.00012679 0.00043362 0.0057968 0.000352593 0.000558957 100 reco:opslicer:OpSlicer 3.8568e-05 0.016294 0.0964979 0.00661873 0.0214805 100 [art]:TriggerResults:TriggerResultInserter 1.3489e-05 1.75411e-05 9.8818e-05 1.5939e-05 8.81504e-06 100 end_path:out1:RootOutput 4.908e-06 6.24579e-06 3.342e-05 5.614e-06 3.78698e-06 100 end_path:out1:RootOutput(write) 0.0134775 0.372439 0.813497 0.46251 0.21448 100 ======================================================================================================================================== ==================================================================================================== MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 19866.1 MB Peak resident set size usage (VmHWM): 2156.56 MB ==================================================================================================== Art has completed and will exit with status 0. === End last 100 lines of lar log file === === Generated output files === 14065.24_dunegpschedd02.fnal.gov.logs.tgz atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481123_457_20231201T150239Z_gen_g4_detsim_hitreco__20240507T203609Z_reco2_graph_2025-08-02T_223902Z.log atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481123_457_20231201T150239Z_gen_g4_detsim_hitreco__20240507T203609Z_reco2_graph_2025-08-02T_223902Z.root debugprod.log graph_output_2025-08-02T_223902Z_4_ana_tree_hd.root graph_output_2025-08-02T_223902Z_4_analysiseid.root graph_output_2025-08-02T_223902Z_4_training1_CaloHitListU_graph.data graph_output_2025-08-02T_223902Z_4_training1_CaloHitListV_graph.data graph_output_2025-08-02T_223902Z_4_training1_CaloHitListW_graph.data graph_output_2025-08-02T_223902Z_4_training2_CaloHitListU_graph.data graph_output_2025-08-02T_223902Z_4_training2_CaloHitListV_graph.data graph_output_2025-08-02T_223902Z_4_training2_CaloHitListW_graph.data jobscript.log justin-processed-pfns.txt reco_hist.root secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481123_457_20231201T150239Z_gen_g4_detsim_hitreco__20240507T203609Z_reco2_graph_2025-08-02T_223902Z.log third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481123_457_20231201T150239Z_gen_g4_detsim_hitreco__20240507T203609Z_reco2_graph_2025-08-02T_223902Z.log training1_CaloHitListU.csv training1_CaloHitListV.csv training1_CaloHitListW.csv training2_CaloHitListU.csv training2_CaloHitListV.csv training2_CaloHitListW.csv