Jobsub ID 14065.4@dunegpschedd02.fnal.gov
| Jobsub ID | 14065.4@dunegpschedd02.fnal.gov | |
| Workflow ID | 253 | |
| Stage ID | 1 | |
| User name | ichong@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-08-02 22:37:47 | |
| Site | NL_SURFsara | |
| Entry | DUNE_SurfSARA_arc03 | |
| Last heartbeat | 2025-08-02 22:47:42 | |
| From worker node | Hostname | wn-da-22.gina.surf.nl |
| cpuinfo | AMD EPYC-Rome Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 129600 (36 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | finished | |
| Started | 2025-08-02 22:38:43 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_75_20231201T042313Z_gen_g4_detsim_hitreco__20240507T203618Z_reco2.root | |
| Jobscript | Exit code | 0 |
| Real time | 7m (474s) | |
| CPU time | 6m (413s = 87%) | |
| Max RSS bytes | 1979146240 (1887 MiB) | |
| Outputting started | 2025-08-02 22:46:38 | |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223854Z_8_training1_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223854Z_8_training1_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223854Z_8_training1_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223854Z_8_training2_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223854Z_8_training2_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223854Z_8_training2_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223854Z_8_analysiseid.root https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223854Z_8_ana_tree_hd.root | |
| Finished | 2025-08-02 22:47:42 | |
| Saved logs | justin-logs:14065.4-dunegpschedd02.fnal.gov.logs.tgz | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
_CaloHitListU_graph.data -> graph_output_2025-08-02T_223854Z_8_training1_CaloHitListU_graph.data Renamed training1_CaloHitListV_graph.data -> graph_output_2025-08-02T_223854Z_8_training1_CaloHitListV_graph.data Renamed training1_CaloHitListW_graph.data -> graph_output_2025-08-02T_223854Z_8_training1_CaloHitListW_graph.data Renamed training2_CaloHitListU_graph.data -> graph_output_2025-08-02T_223854Z_8_training2_CaloHitListU_graph.data Renamed training2_CaloHitListV_graph.data -> graph_output_2025-08-02T_223854Z_8_training2_CaloHitListV_graph.data Renamed training2_CaloHitListW_graph.data -> graph_output_2025-08-02T_223854Z_8_training2_CaloHitListW_graph.data Renamed analysiseid.root -> graph_output_2025-08-02T_223854Z_8_analysiseid.root Renamed ana_tree_hd.root -> graph_output_2025-08-02T_223854Z_8_ana_tree_hd.root === Start last 100 lines of lar log file === --Primary 0, MCPDG 22, Energy 0.707988, Dist. 28.035, nMCHits 1102 (350, 371, 381) MCPDG 22, Energy 0.707988, Dist. 28.035, nMCHits 1102 (350, 371, 381) --Primary 1, MCPDG -13, Energy 0.728995, Dist. 273.042, nMCHits 884 (182, 463, 239) MCPDG -13, Energy 0.728995, Dist. 273.042, nMCHits 829 (158, 448, 223) \_ MCPDG -11, Energy 0.0329914, Dist. 10.1631, nMCHits 55 (24, 15, 16) --Primary 2, MCPDG 2212, Energy 1.45634, Dist. 121.337, nMCHits 543 (159, 179, 205) MCPDG 2212, Energy 1.45634, Dist. 121.337, nMCHits 543 (159, 179, 205) --Primary 3, MCPDG 22, Energy 0.224595, Dist. 57.7115, nMCHits 360 (106, 126, 128) MCPDG 22, Energy 0.224595, Dist. 57.7115, nMCHits 360 (106, 126, 128) --Primary 4, MCPDG -211, Energy 0.817942, Dist. 58.8404, nMCHits 196 (30, 113, 53) MCPDG -211, Energy 0.817942, Dist. 58.8404, nMCHits 142 (18, 81, 43) \_ MCPDG 2212, Energy 1.10396, Dist. 18.8214, nMCHits 53 (12, 31, 10) \_ MCPDG 2212, Energy 0.946424, Dist. 0.0947036, nMCHits 1 (0, 1, 0) ------------------------------------------------------------------------------------------------ Operating in training mode. The eid is 97 Graph saved to training1_CaloHitListW_graph.data Size of file training1_CaloHitListW_graph.data is 958928 bytes. The eid is 97 Graph saved to training1_CaloHitListU_graph.data Size of file training1_CaloHitListU_graph.data is 1322552 bytes. The eid is 97 Graph saved to training1_CaloHitListV_graph.data Size of file training1_CaloHitListV_graph.data is 1213664 bytes. Operating in inference mode. Operating in training mode. The eid is -1 Graph saved to training2_CaloHitListW_graph.data Size of file training2_CaloHitListW_graph.data is 949940 bytes. The eid is -1 Graph saved to training2_CaloHitListU_graph.data Size of file training2_CaloHitListU_graph.data is 1309888 bytes. The eid is -1 Graph saved to training2_CaloHitListV_graph.data Size of file training2_CaloHitListV_graph.data is 1211736 bytes. Boundary wire vector sizes: 998, 1411, 1147 minwire 0: 188 minwire 1: 1922 minwire 2: 8 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Classifier summary: Output 0: 0.0501376, Output 1: 0.0973343, 0.256117, 0.0855023, 0.561046, Output 2: 0.125576, 0.428596, 0.425815, 0.0200117, Output 3: 0.948843, 0.044567, 0.00265832, 0.0039319, Output 4: 0.101756, 0.826827, 0.0677498, 0.00366674, Output 5: 0.677018, 0.316281, 0.00587597, 0.000824692, Output 6: 0.0430919, 0.537566, 0.236495, 0.182847, Running Ophitfinder with InputDigiType = 'recob' Found hits: 320! 03-Aug-2025 00:45:23 CEST Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_75_20231201T042313Z_gen_g4_detsim_hitreco__20240507T203618Z_reco2_graph_2025-08-02T_223854Z.root" 03-Aug-2025 00:45:23 CEST Closed input file "root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/f6/58/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_75_20231201T042313Z_gen_g4_detsim_hitreco__20240507T203618Z_reco2.root" ======================================================================================================================================== TimeTracker printout (sec) Min Avg Max Median RMS nEvts ======================================================================================================================================== Full event 0.0983528 2.7143 11.3922 2.7686 1.58696 100 ---------------------------------------------------------------------------------------------------------------------------------------- source:RootInput(read) 0.0021559 0.00514661 0.0453302 0.00366787 0.00636092 100 reco:gaushit:GausHitFinder 0.0189784 0.0676793 0.632555 0.036185 0.0962564 100 reco:spsolve:SpacePointSolver 0.000107485 0.0406497 0.745129 0.00452214 0.110114 100 reco:hitfd:DisambigFromSpacePoints 0.000198328 0.0129459 0.411212 0.00112218 0.0535548 100 reco:rns:RandomNumberSaver 1.9246e-05 2.81096e-05 0.000289592 2.40115e-05 2.68151e-05 100 reco:pandora:StandardPandora 0.0112492 1.18787 7.35783 0.988233 0.946584 100 reco:pandoraTrack:LArPandoraTrackCreation 0.000121451 0.000173027 0.00217684 0.000148352 0.000202592 100 reco:pandoraShower:LArPandoraModularShowerCreation 0.000170134 0.000229968 0.0034427 0.000192476 0.0003237 100 reco:pandoracalo:Calorimetry 0.000107796 0.000139697 0.00131538 0.000123861 0.000119286 100 reco:pandorapid:Chi2ParticleID 3.7422e-05 4.95083e-05 0.000689174 4.1163e-05 6.47997e-05 100 reco:cvnmap:CVNMapper 2.069e-05 0.0216459 0.0663441 0.020247 0.015279 100 reco:cvneva:CVNEvaluator 1.6292e-05 0.778204 2.79356 1.02026 0.493713 100 reco:energyrecnumu:EnergyReco 0.00236451 0.00572199 0.0199635 0.00607206 0.00294334 100 reco:energyrecnue:EnergyReco 0.00230253 0.00313489 0.0106212 0.00269838 0.00145345 100 reco:energyrecnc:EnergyReco 0.00231194 0.00308277 0.012892 0.00260845 0.00163737 100 reco:energyrecnumurange:EnergyReco 0.00226969 0.00332386 0.0163963 0.00258361 0.00230915 100 reco:energyrecnumumcs:EnergyReco 0.00227937 0.00313263 0.0158109 0.00258549 0.00184322 100 reco:opdec:Deconvolution 0.00900854 0.191082 0.357161 0.183903 0.0762748 100 reco:ophitspe:OpHitFinderDeco 0.0126761 0.0260839 0.154561 0.0198872 0.0233648 100 reco:opflash:OpFlashFinder 0.000116061 0.000341263 0.00349753 0.000298895 0.000334569 100 reco:opslicer:OpSlicer 3.6651e-05 0.0127866 0.0672934 0.00646371 0.0162431 100 [art]:TriggerResults:TriggerResultInserter 1.014e-05 1.30521e-05 7.0694e-05 1.1918e-05 6.13388e-06 100 end_path:out1:RootOutput 3.206e-06 4.33967e-06 2.0608e-05 4.158e-06 1.69145e-06 100 end_path:out1:RootOutput(write) 0.00444723 0.349873 0.696737 0.430118 0.187565 100 ======================================================================================================================================== ==================================================================================================== MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 11574.1 MB Peak resident set size usage (VmHWM): 1979.15 MB ==================================================================================================== Art has completed and will exit with status 0. === End last 100 lines of lar log file === === Generated output files === 14065.4_dunegpschedd02.fnal.gov.logs.tgz atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_75_20231201T042313Z_gen_g4_detsim_hitreco__20240507T203618Z_reco2_graph_2025-08-02T_223854Z.log atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_75_20231201T042313Z_gen_g4_detsim_hitreco__20240507T203618Z_reco2_graph_2025-08-02T_223854Z.root debugprod.log graph_output_2025-08-02T_223854Z_8_ana_tree_hd.root graph_output_2025-08-02T_223854Z_8_analysiseid.root graph_output_2025-08-02T_223854Z_8_training1_CaloHitListU_graph.data graph_output_2025-08-02T_223854Z_8_training1_CaloHitListV_graph.data graph_output_2025-08-02T_223854Z_8_training1_CaloHitListW_graph.data graph_output_2025-08-02T_223854Z_8_training2_CaloHitListU_graph.data graph_output_2025-08-02T_223854Z_8_training2_CaloHitListV_graph.data graph_output_2025-08-02T_223854Z_8_training2_CaloHitListW_graph.data jobscript.log justin-processed-pfns.txt reco_hist.root secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_75_20231201T042313Z_gen_g4_detsim_hitreco__20240507T203618Z_reco2_graph_2025-08-02T_223854Z.log third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_75_20231201T042313Z_gen_g4_detsim_hitreco__20240507T203618Z_reco2_graph_2025-08-02T_223854Z.log training1_CaloHitListU.csv training1_CaloHitListV.csv training1_CaloHitListW.csv training2_CaloHitListU.csv training2_CaloHitListV.csv training2_CaloHitListW.csv