Jobsub ID 14065.48@dunegpschedd02.fnal.gov
| Jobsub ID | 14065.48@dunegpschedd02.fnal.gov | |
| Workflow ID | 253 | |
| Stage ID | 1 | |
| User name | ichong@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-08-02 22:37:47 | |
| Site | NL_SURFsara | |
| Entry | DUNE_SurfSARA_arc03 | |
| Last heartbeat | 2025-08-02 22:47:44 | |
| From worker node | Hostname | wn-la-24.gina.surf.nl |
| cpuinfo | AMD EPYC 9754 128-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 129600 (36 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | finished | |
| Started | 2025-08-02 22:39:03 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_730_20231207T140535Z_gen_g4_detsim_hitreco__20240510T045130Z_reco2.root | |
| Jobscript | Exit code | 0 |
| Real time | 7m (460s) | |
| CPU time | 7m (435s = 94%) | |
| Max RSS bytes | 2068250624 (1972 MiB) | |
| Outputting started | 2025-08-02 22:46:44 | |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223914Z_7_training1_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223914Z_7_training1_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223914Z_7_training1_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223914Z_7_training2_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223914Z_7_training2_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223914Z_7_training2_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223914Z_7_analysiseid.root https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00253/1/001/graph_output_2025-08-02T_223914Z_7_ana_tree_hd.root | |
| Finished | 2025-08-02 22:47:44 | |
| Saved logs | justin-logs:14065.48-dunegpschedd02.fnal.gov.logs.tgz | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
T_223914Z_7_training1_CaloHitListU_graph.data
Renamed training1_CaloHitListV_graph.data -> graph_output_2025-08-02T_223914Z_7_training1_CaloHitListV_graph.data
Renamed training1_CaloHitListW_graph.data -> graph_output_2025-08-02T_223914Z_7_training1_CaloHitListW_graph.data
Renamed training2_CaloHitListU_graph.data -> graph_output_2025-08-02T_223914Z_7_training2_CaloHitListU_graph.data
Renamed training2_CaloHitListV_graph.data -> graph_output_2025-08-02T_223914Z_7_training2_CaloHitListV_graph.data
Renamed training2_CaloHitListW_graph.data -> graph_output_2025-08-02T_223914Z_7_training2_CaloHitListW_graph.data
Renamed analysiseid.root -> graph_output_2025-08-02T_223914Z_7_analysiseid.root
Renamed ana_tree_hd.root -> graph_output_2025-08-02T_223914Z_7_ana_tree_hd.root
=== Start last 100 lines of lar log file ===
MCPDG -11, Energy 6.70827, Dist. 59.2197, nMCHits 3370 (1375, 1300, 695)
--Primary 1, MCPDG 211, Energy 3.05647, Dist. 99.8597, nMCHits 626 (267, 206, 153)
MCPDG 211, Energy 3.05647, Dist. 99.8597, nMCHits 273 (151, 64, 58)
\_ MCPDG 211, Energy 2.58091, Dist. 52.3772, nMCHits 243 (91, 94, 58)
\_ MCPDG 22, Energy 0.066154, Dist. 0, nMCHits 1 (0, 0, 1)
\_ MCPDG 2212, Energy 1.17802, Dist. 34.3206, nMCHits 109 (25, 48, 36)
--Primary 2, MCPDG 22, Energy 0.356059, Dist. 7.12121, nMCHits 350 (145, 130, 75)
MCPDG 22, Energy 0.356059, Dist. 7.12121, nMCHits 350 (145, 130, 75)
--Primary 3, MCPDG -211, Energy 0.987739, Dist. 20.1951, nMCHits 153 (32, 63, 58)
MCPDG -211, Energy 0.987739, Dist. 20.1951, nMCHits 57 (9, 25, 23)
\_ MCPDG -211, Energy 0.73081, Dist. 12.0148, nMCHits 55 (17, 17, 21)
\_ MCPDG 2212, Energy 1.07083, Dist. 12.8911, nMCHits 41 (6, 21, 14)
--Primary 4, MCPDG 22, Energy 0.0349597, Dist. 8.24384, nMCHits 56 (17, 21, 18)
MCPDG 22, Energy 0.0349597, Dist. 8.24384, nMCHits 56 (17, 21, 18)
------------------------------------------------------------------------------------------------
Operating in training mode.
The eid is 98
Graph saved to training1_CaloHitListW_graph.data
Size of file training1_CaloHitListW_graph.data is 798776 bytes.
The eid is 98
Graph saved to training1_CaloHitListU_graph.data
Size of file training1_CaloHitListU_graph.data is 1019384 bytes.
The eid is 98
Graph saved to training1_CaloHitListV_graph.data
Size of file training1_CaloHitListV_graph.data is 1103652 bytes.
Operating in inference mode.
Operating in training mode.
The eid is -1
Graph saved to training2_CaloHitListW_graph.data
Size of file training2_CaloHitListW_graph.data is 797980 bytes.
The eid is -1
Graph saved to training2_CaloHitListU_graph.data
Size of file training2_CaloHitListU_graph.data is 1018520 bytes.
The eid is -1
Graph saved to training2_CaloHitListV_graph.data
Size of file training2_CaloHitListV_graph.data is 1103060 bytes.
Boundary wire vector sizes: 1835, 1948, 1100
minwire 0: 625
minwire 1: 1701
minwire 2: 360
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary:
Output 0: 0.0453517,
Output 1: 0.0714273, 0.0352234, 0.039941, 0.853408,
Output 2: 0.0312412, 0.742486, 0.218576, 0.00769741,
Output 3: 0.320138, 0.55328, 0.101813, 0.0247688,
Output 4: 0.063585, 0.341807, 0.551937, 0.0426714,
Output 5: 0.947222, 0.0457963, 0.00633597, 0.000645984,
Output 6: 0.240793, 0.446656, 0.200535, 0.112016,
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 418!
03-Aug-2025 00:45:42 CEST Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_730_20231207T140535Z_gen_g4_detsim_hitreco__20240510T045130Z_reco2_graph_2025-08-02T_223914Z.root"
03-Aug-2025 00:45:42 CEST Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/73/9f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_730_20231207T140535Z_gen_g4_detsim_hitreco__20240510T045130Z_reco2.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 0.0862432 2.93168 9.85365 3.05896 1.16219 100
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.00235517 0.00418661 0.0247801 0.00364253 0.00230085 100
reco:gaushit:GausHitFinder 0.0179502 0.0585475 0.558778 0.0359806 0.0723846 100
reco:spsolve:SpacePointSolver 0.00012123 0.0379975 1.19675 0.0052737 0.131397 100
reco:hitfd:DisambigFromSpacePoints 0.000227981 0.00881333 0.268859 0.00124506 0.0334221 100
reco:rns:RandomNumberSaver 2.2372e-05 3.1317e-05 0.000349142 2.68205e-05 3.22552e-05 100
reco:pandora:StandardPandora 0.0129336 1.30062 5.58767 1.19 0.504144 100
reco:pandoraTrack:LArPandoraTrackCreation 0.00014717 0.000211951 0.00318874 0.000172292 0.00030196 100
reco:pandoraShower:LArPandoraModularShowerCreation 0.000191517 0.000244883 0.00319311 0.000212733 0.000296749 100
reco:pandoracalo:Calorimetry 0.000122754 0.000159231 0.00150004 0.000144605 0.000135599 100
reco:pandorapid:Chi2ParticleID 4.1342e-05 5.34812e-05 0.000723862 4.4822e-05 6.77485e-05 100
reco:cvnmap:CVNMapper 2.4857e-05 0.0111705 0.0360076 0.0125286 0.00649908 100
reco:cvneva:CVNEvaluator 1.9929e-05 0.881234 3.15588 1.07909 0.499137 100
reco:energyrecnumu:EnergyReco 0.00208696 0.00548737 0.0241773 0.00473903 0.00398929 100
reco:energyrecnue:EnergyReco 0.00213245 0.00283845 0.0205672 0.00245968 0.00186493 100
reco:energyrecnc:EnergyReco 0.00214975 0.00265293 0.00729674 0.00238333 0.00074875 100
reco:energyrecnumurange:EnergyReco 0.00215352 0.00286572 0.0181613 0.00237163 0.00214187 100
reco:energyrecnumumcs:EnergyReco 0.0021596 0.00283859 0.015403 0.00239436 0.0017236 100
reco:opdec:Deconvolution 0.0149933 0.17618 0.394817 0.177838 0.0639272 100
reco:ophitspe:OpHitFinderDeco 0.0168333 0.0221572 0.0502191 0.0217886 0.00370231 100
reco:opflash:OpFlashFinder 0.000131526 0.000371112 0.00296956 0.000332026 0.00028271 100
reco:opslicer:OpSlicer 4.1603e-05 0.0125347 0.0705502 0.00710217 0.0149035 100
[art]:TriggerResults:TriggerResultInserter 1.306e-05 1.83705e-05 0.00023399 1.5313e-05 2.21306e-05 100
end_path:out1:RootOutput 4.216e-06 5.305e-06 2.4036e-05 4.9025e-06 2.40378e-06 100
end_path:out1:RootOutput(write) 0.00445819 0.399456 0.699471 0.475474 0.182598 100
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 19773.9 MB
Peak resident set size usage (VmHWM): 2068.25 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Generated output files ===
14065.48_dunegpschedd02.fnal.gov.logs.tgz
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_730_20231207T140535Z_gen_g4_detsim_hitreco__20240510T045130Z_reco2_graph_2025-08-02T_223914Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_730_20231207T140535Z_gen_g4_detsim_hitreco__20240510T045130Z_reco2_graph_2025-08-02T_223914Z.root
debugprod.log
graph_output_2025-08-02T_223914Z_7_ana_tree_hd.root
graph_output_2025-08-02T_223914Z_7_analysiseid.root
graph_output_2025-08-02T_223914Z_7_training1_CaloHitListU_graph.data
graph_output_2025-08-02T_223914Z_7_training1_CaloHitListV_graph.data
graph_output_2025-08-02T_223914Z_7_training1_CaloHitListW_graph.data
graph_output_2025-08-02T_223914Z_7_training2_CaloHitListU_graph.data
graph_output_2025-08-02T_223914Z_7_training2_CaloHitListV_graph.data
graph_output_2025-08-02T_223914Z_7_training2_CaloHitListW_graph.data
jobscript.log
justin-processed-pfns.txt
reco_hist.root
secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_730_20231207T140535Z_gen_g4_detsim_hitreco__20240510T045130Z_reco2_graph_2025-08-02T_223914Z.log
third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_730_20231207T140535Z_gen_g4_detsim_hitreco__20240510T045130Z_reco2_graph_2025-08-02T_223914Z.log
training1_CaloHitListU.csv
training1_CaloHitListV.csv
training1_CaloHitListW.csv
training2_CaloHitListU.csv
training2_CaloHitListV.csv
training2_CaloHitListW.csv