Jobsub ID 13546.0@dunegpschedd02.fnal.gov
| Jobsub ID | 13546.0@dunegpschedd02.fnal.gov | |
| Workflow ID | 83 | |
| Stage ID | 1 | |
| User name | ichong@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-08-01 02:23:30 | |
| Site | NL_SURFsara | |
| Entry | DUNE_SurfSARA_arc03 | |
| Last heartbeat | 2025-08-01 02:31:59 | |
| From worker node | Hostname | wn-da-01.gina.surf.nl |
| cpuinfo | AMD EPYC 7702P 64-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 129600 (36 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | finished | |
| Started | 2025-08-01 02:24:15 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_535_20231202T102929Z_gen_g4_detsim_hitreco__20240507T200455Z_reco2.root | |
| Jobscript | Exit code | 0 |
| Real time | 6m (407s) | |
| CPU time | 6m (377s = 92%) | |
| Max RSS bytes | 1985093632 (1893 MiB) | |
| Outputting started | 2025-08-01 02:31:03 | |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00083/1/024/graph_output_2025-08-01T_022424Z_10_training1_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00083/1/024/graph_output_2025-08-01T_022424Z_10_training1_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00083/1/024/graph_output_2025-08-01T_022424Z_10_training1_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00083/1/024/graph_output_2025-08-01T_022424Z_10_training2_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00083/1/024/graph_output_2025-08-01T_022424Z_10_training2_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00083/1/024/graph_output_2025-08-01T_022424Z_10_training2_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00083/1/024/graph_output_2025-08-01T_022424Z_10_analysiseid.root https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00083/1/024/graph_output_2025-08-01T_022424Z_10_ana_tree_hd.root | |
| Finished | 2025-08-01 02:31:59 | |
| Saved logs | justin-logs:13546.0-dunegpschedd02.fnal.gov.logs.tgz | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
_10_training1_CaloHitListU_graph.data Renamed training1_CaloHitListV_graph.data -> graph_output_2025-08-01T_022424Z_10_training1_CaloHitListV_graph.data Renamed training1_CaloHitListW_graph.data -> graph_output_2025-08-01T_022424Z_10_training1_CaloHitListW_graph.data Renamed training2_CaloHitListU_graph.data -> graph_output_2025-08-01T_022424Z_10_training2_CaloHitListU_graph.data Renamed training2_CaloHitListV_graph.data -> graph_output_2025-08-01T_022424Z_10_training2_CaloHitListV_graph.data Renamed training2_CaloHitListW_graph.data -> graph_output_2025-08-01T_022424Z_10_training2_CaloHitListW_graph.data Renamed analysiseid.root -> graph_output_2025-08-01T_022424Z_10_analysiseid.root Renamed ana_tree_hd.root -> graph_output_2025-08-01T_022424Z_10_ana_tree_hd.root === Start last 100 lines of lar log file === --Primary 2, MCPDG 2212, Energy 1.27317, Dist. 30.7954, nMCHits 143 (48, 72, 23) MCPDG 2212, Energy 1.27317, Dist. 30.7954, nMCHits 72 (36, 31, 5) \_ MCPDG 2212, Energy 1.13879, Dist. 21.0759, nMCHits 63 (10, 37, 16) \_ MCPDG 2212, Energy 0.959191, Dist. 0.489004, nMCHits 8 (2, 4, 2) --Primary 3, MCPDG 2212, Energy 1.15525, Dist. 29.478, nMCHits 57 (31, 7, 19) MCPDG 2212, Energy 1.15525, Dist. 29.478, nMCHits 57 (31, 7, 19) --Primary 4, MCPDG 2212, Energy 1.1317, Dist. 18.5907, nMCHits 47 (11, 27, 9) MCPDG 2212, Energy 1.1317, Dist. 18.5907, nMCHits 31 (8, 18, 5) \_ MCPDG 1000180390, Energy 36.2867, Dist. 0.000235525, nMCHits 2 (0, 2, 0) \_ MCPDG 2212, Energy 1.00334, Dist. 3.70666, nMCHits 14 (3, 7, 4) --Primary 5, MCPDG 2212, Energy 1.09492, Dist. 17.4848, nMCHits 47 (30, 6, 11) MCPDG 2212, Energy 1.09492, Dist. 17.4848, nMCHits 47 (30, 6, 11) --Primary 6, MCPDG 2212, Energy 1.04101, Dist. 8.21063, nMCHits 24 (7, 9, 8) MCPDG 2212, Energy 1.04101, Dist. 8.21063, nMCHits 24 (7, 9, 8) ------------------------------------------------------------------------------------------------ Operating in training mode. The eid is 98 Graph saved to training1_CaloHitListW_graph.data Size of file training1_CaloHitListW_graph.data is 957420 bytes. The eid is 98 Graph saved to training1_CaloHitListU_graph.data Size of file training1_CaloHitListU_graph.data is 1263588 bytes. The eid is 98 Graph saved to training1_CaloHitListV_graph.data Size of file training1_CaloHitListV_graph.data is 1372124 bytes. Operating in inference mode. Operating in training mode. The eid is -1 Graph saved to training2_CaloHitListW_graph.data Size of file training2_CaloHitListW_graph.data is 957120 bytes. The eid is -1 Graph saved to training2_CaloHitListU_graph.data Size of file training2_CaloHitListU_graph.data is 1260728 bytes. The eid is -1 Graph saved to training2_CaloHitListV_graph.data Size of file training2_CaloHitListV_graph.data is 1370468 bytes. Boundary wire vector sizes: 643, 988, 787 minwire 0: 1878 minwire 1: 484 minwire 2: 1863 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499 Classifier summary: Output 0: 0.0258873, Output 1: 0.973479, 0.00375504, 0.00261201, 0.0201543, Output 2: 0.0252358, 0.833741, 0.0935249, 0.0474979, Output 3: 0.027634, 0.00477343, 0.0391054, 0.928487, Output 4: 0.841526, 0.131887, 0.0198578, 0.0067293, Output 5: 0.99919, 0.000787978, 1.51353e-05, 7.14713e-06, Output 6: 0.00613414, 0.0202274, 0.0586339, 0.915005, Running Ophitfinder with InputDigiType = 'recob' Found hits: 339! 01-Aug-2025 04:30:02 CEST Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_535_20231202T102929Z_gen_g4_detsim_hitreco__20240507T200455Z_reco2_graph_2025-08-01T_022424Z.root" 01-Aug-2025 04:30:02 CEST Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/e8/00/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_535_20231202T102929Z_gen_g4_detsim_hitreco__20240507T200455Z_reco2.root" ======================================================================================================================================== TimeTracker printout (sec) Min Avg Max Median RMS nEvts ======================================================================================================================================== Full event 0.0993108 2.51343 12.8388 2.60853 1.43996 100 ---------------------------------------------------------------------------------------------------------------------------------------- source:RootInput(read) 0.00186898 0.00300291 0.00656354 0.00310586 0.000808544 100 reco:gaushit:GausHitFinder 0.0170224 0.0572532 0.614021 0.0338212 0.075792 100 reco:spsolve:SpacePointSolver 9.5061e-05 0.0371534 0.98755 0.00388393 0.128564 100 reco:hitfd:DisambigFromSpacePoints 0.000197175 0.0153084 0.677125 0.0014586 0.0778495 100 reco:rns:RandomNumberSaver 1.7984e-05 2.54993e-05 0.000266756 2.13255e-05 2.47691e-05 100 reco:pandora:StandardPandora 0.0116426 1.14256 8.54205 1.0116 0.831322 100 reco:pandoraTrack:LArPandoraTrackCreation 0.00012633 0.000179887 0.00243502 0.000147537 0.000228999 100 reco:pandoraShower:LArPandoraModularShowerCreation 0.000174463 0.000225997 0.00304059 0.000193433 0.000283254 100 reco:pandoracalo:Calorimetry 0.000106061 0.000144073 0.00129407 0.000119542 0.00013617 100 reco:pandorapid:Chi2ParticleID 3.7531e-05 5.03716e-05 0.000693448 4.2656e-05 6.48897e-05 100 reco:cvnmap:CVNMapper 2.1772e-05 0.0175786 0.0492572 0.0215381 0.0108012 100 reco:cvneva:CVNEvaluator 1.7813e-05 0.713162 2.90285 0.917782 0.45644 100 reco:energyrecnumu:EnergyReco 0.0020722 0.00483865 0.0128287 0.00508075 0.00211437 100 reco:energyrecnue:EnergyReco 0.00203421 0.00275534 0.0107629 0.0025033 0.00104391 100 reco:energyrecnc:EnergyReco 0.00201736 0.0026624 0.0106819 0.00235849 0.00103931 100 reco:energyrecnumurange:EnergyReco 0.00203484 0.00264644 0.010527 0.00238783 0.00101385 100 reco:energyrecnumumcs:EnergyReco 0.00203699 0.00266028 0.0106419 0.002377 0.00103565 100 reco:opdec:Deconvolution 0.0281237 0.158611 0.28018 0.158001 0.056789 100 reco:ophitspe:OpHitFinderDeco 0.0115305 0.0165626 0.1061 0.0156096 0.00914804 100 reco:opflash:OpFlashFinder 0.000122492 0.000323251 0.00310492 0.000280268 0.000294654 100 reco:opslicer:OpSlicer 3.5809e-05 0.0101845 0.0587571 0.00623181 0.011806 100 [art]:TriggerResults:TriggerResultInserter 1.074e-05 1.35556e-05 4.6959e-05 1.26e-05 4.33789e-06 100 end_path:out1:RootOutput 3.738e-06 4.90996e-06 1.7764e-05 4.825e-06 1.36288e-06 100 end_path:out1:RootOutput(write) 0.0130321 0.324635 0.665118 0.397269 0.170244 100 ======================================================================================================================================== ==================================================================================================== MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 11579.7 MB Peak resident set size usage (VmHWM): 1985.09 MB ==================================================================================================== Art has completed and will exit with status 0. === End last 100 lines of lar log file === === Generated output files === 13546.0_dunegpschedd02.fnal.gov.logs.tgz atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_535_20231202T102929Z_gen_g4_detsim_hitreco__20240507T200455Z_reco2_graph_2025-08-01T_022424Z.log atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_535_20231202T102929Z_gen_g4_detsim_hitreco__20240507T200455Z_reco2_graph_2025-08-01T_022424Z.root debugprod.log graph_output_2025-08-01T_022424Z_10_ana_tree_hd.root graph_output_2025-08-01T_022424Z_10_analysiseid.root graph_output_2025-08-01T_022424Z_10_training1_CaloHitListU_graph.data graph_output_2025-08-01T_022424Z_10_training1_CaloHitListV_graph.data graph_output_2025-08-01T_022424Z_10_training1_CaloHitListW_graph.data graph_output_2025-08-01T_022424Z_10_training2_CaloHitListU_graph.data graph_output_2025-08-01T_022424Z_10_training2_CaloHitListV_graph.data graph_output_2025-08-01T_022424Z_10_training2_CaloHitListW_graph.data jobscript.log justin-processed-pfns.txt reco_hist.root secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_535_20231202T102929Z_gen_g4_detsim_hitreco__20240507T200455Z_reco2_graph_2025-08-01T_022424Z.log third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_535_20231202T102929Z_gen_g4_detsim_hitreco__20240507T200455Z_reco2_graph_2025-08-01T_022424Z.log training1_CaloHitListU.csv training1_CaloHitListV.csv training1_CaloHitListW.csv training2_CaloHitListU.csv training2_CaloHitListV.csv training2_CaloHitListW.csv