Jobsub ID 12822.14@dunegpschedd02.fnal.gov
| Jobsub ID | 12822.14@dunegpschedd02.fnal.gov | |
| Workflow ID | 83 | |
| Stage ID | 1 | |
| User name | ichong@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-07-30 13:45:38 | |
| Site | ES_PIC | |
| Entry | DUNE_T1_ES_PIC_ce15-multicore | |
| Last heartbeat | 2025-07-30 14:43:32 | |
| From worker node | Hostname | tds421.pic.es |
| cpuinfo | AMD EPYC 7502 32-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 216000 (60 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | finished | |
| Started | 2025-07-30 13:46:24 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_498_20231207T204556Z_gen_g4_detsim_hitreco__20240510T052859Z_reco2.root | |
| Jobscript | Exit code | 0 |
| Real time | 49m (2952s) | |
| CPU time | 9m (540s = 18%) | |
| Max RSS bytes | 2137083904 (2038 MiB) | |
| Outputting started | 2025-07-30 14:35:38 | |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00083/1/024/graph_output_2025-07-30T_134710Z_3_ana_tree_hd.root https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00083/1/024/graph_output_2025-07-30T_134710Z_3_training1_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00083/1/024/graph_output_2025-07-30T_134710Z_3_training1_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00083/1/024/graph_output_2025-07-30T_134710Z_3_analysiseid.root https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00083/1/024/graph_output_2025-07-30T_134710Z_3_training2_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00083/1/024/graph_output_2025-07-30T_134710Z_3_training1_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00083/1/024/graph_output_2025-07-30T_134710Z_3_training2_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00083/1/024/graph_output_2025-07-30T_134710Z_3_training2_CaloHitListV_graph.data | |
| Finished | 2025-07-30 14:43:32 | |
| Saved logs | justin-logs:12822.14-dunegpschedd02.fnal.gov.logs.tgz | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
CaloHitListU_graph.data -> graph_output_2025-07-30T_134710Z_3_training1_CaloHitListU_graph.data
Renamed training1_CaloHitListV_graph.data -> graph_output_2025-07-30T_134710Z_3_training1_CaloHitListV_graph.data
Renamed training1_CaloHitListW_graph.data -> graph_output_2025-07-30T_134710Z_3_training1_CaloHitListW_graph.data
Renamed training2_CaloHitListU_graph.data -> graph_output_2025-07-30T_134710Z_3_training2_CaloHitListU_graph.data
Renamed training2_CaloHitListV_graph.data -> graph_output_2025-07-30T_134710Z_3_training2_CaloHitListV_graph.data
Renamed training2_CaloHitListW_graph.data -> graph_output_2025-07-30T_134710Z_3_training2_CaloHitListW_graph.data
Renamed analysiseid.root -> graph_output_2025-07-30T_134710Z_3_analysiseid.root
Renamed ana_tree_hd.root -> graph_output_2025-07-30T_134710Z_3_ana_tree_hd.root
=== Start last 100 lines of lar log file ===
C:0 T:2 12 XUs and 12 XVs -> 8 XUVs
C:0 T:10 30 XUs and 36 XVs -> 21 XUVs
33 XUVs total
18 collection wire objects
33 potential space points
Neighbour search...
251 tests to find 146 neighbours
Iterating with no regularization...
Begin: 451992
0 252476
1 138422
2 122833
3 120510
4 120068
5 119995
Now with regularization...
Begin: -100387
0 -100685
1 -100824
2 -100843
---MC-PARTICLE-MONITORING-----------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Operating in training mode.
this->CompleteMCHierarchy(mcToHitsMap, hierarchy) return STATUS_CODE_NOT_FOUND
in function: PrepareTrainingSample
in file: /exp/dune/app/users/ichong/larsoft_graph_V1_2025/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 81
iter->second->Run() throw STATUS_CODE_NOT_FOUND
in function: RunAlgorithm
in file: /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLVertexing, STATUS_CODE_NOT_FOUND
Operating in inference mode.
Operating in training mode.
this->CompleteMCHierarchy(mcToHitsMap, hierarchy) return STATUS_CODE_NOT_FOUND
in function: PrepareTrainingSample
in file: /exp/dune/app/users/ichong/larsoft_graph_V1_2025/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 81
iter->second->Run() throw STATUS_CODE_NOT_FOUND
in function: RunAlgorithm
in file: /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_FOUND
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 153!
Begin processing the 100th record. run: 50640579 subRun: 1 event: 49900 at 30-Jul-2025 16:26:48 CEST
---MC-PARTICLE-MONITORING-----------------------------------------------------------------------
BeamNeutrinos:
--Primary 0, MCPDG 2212, Energy 0.958911, Dist. 0.481572, nMCHits 16 (10, 5, 1)
MCPDG 2212, Energy 0.958911, Dist. 0.481572, nMCHits 16 (10, 5, 1)
------------------------------------------------------------------------------------------------
Operating in training mode.
The eid is 99
Graph saved to training1_CaloHitListU_graph.data
Size of file training1_CaloHitListU_graph.data is 1418444 bytes.
Operating in inference mode.
Operating in training mode.
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 61!
30-Jul-2025 16:26:52 CEST Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_498_20231207T204556Z_gen_g4_detsim_hitreco__20240510T052859Z_reco2_graph_2025-07-30T_134710Z.root"
30-Jul-2025 16:26:52 CEST Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/df/70/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_498_20231207T204556Z_gen_g4_detsim_hitreco__20240510T052859Z_reco2.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 1.84306 5.02326 54.0073 4.59794 5.35312 100
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.0337522 0.0575105 0.414237 0.060469 0.0408398 100
reco:gaushit:GausHitFinder 0.122985 0.320105 3.96069 0.221009 0.432064 100
reco:spsolve:SpacePointSolver 0.000142852 0.13207 6.78061 0.00466072 0.694553 100
reco:hitfd:DisambigFromSpacePoints 0.000265505 0.0347615 2.23793 0.00145955 0.225245 100
reco:rns:RandomNumberSaver 1.979e-05 3.34326e-05 0.000470669 2.74e-05 4.43966e-05 100
reco:pandora:StandardPandora 1.08279 1.97146 36.5091 1.4144 3.5686 100
reco:pandoraTrack:LArPandoraTrackCreation 0.000134702 0.000269177 0.00657712 0.000188939 0.000639522 100
reco:pandoraShower:LArPandoraModularShowerCreation 0.000177243 0.000290531 0.00393089 0.000247774 0.000367709 100
reco:pandoracalo:Calorimetry 0.000102312 0.000172876 0.00151247 0.000155838 0.00013666 100
reco:pandorapid:Chi2ParticleID 3.7761e-05 6.07241e-05 0.000661383 5.3941e-05 6.1226e-05 100
reco:cvnmap:CVNMapper 3.3101e-05 0.0156162 0.0903471 0.0184121 0.0125212 100
reco:cvneva:CVNEvaluator 1.949e-05 1.11614 6.89728 1.47096 0.884625 100
reco:energyrecnumu:EnergyReco 0.00226263 0.00559904 0.0257509 0.00396981 0.00341579 100
reco:energyrecnue:EnergyReco 0.00245464 0.00420076 0.0234082 0.00360519 0.0023586 100
reco:energyrecnc:EnergyReco 0.00234453 0.00414518 0.0236709 0.00356431 0.00237073 100
reco:energyrecnumurange:EnergyReco 0.00232962 0.00415347 0.0224853 0.00359312 0.00228973 100
reco:energyrecnumumcs:EnergyReco 0.00225256 0.00413982 0.0223359 0.00356457 0.00228053 100
reco:opdec:Deconvolution 0.158358 0.55163 1.37331 0.513474 0.229615 100
reco:ophitspe:OpHitFinderDeco 0.29938 0.322698 0.570104 0.308854 0.0546837 100
reco:opflash:OpFlashFinder 0.000160323 0.000460508 0.00442983 0.000409533 0.000428507 100
reco:opslicer:OpSlicer 6.0361e-05 0.0177011 0.102884 0.0068035 0.0233395 100
[art]:TriggerResults:TriggerResultInserter 1.184e-05 2.10952e-05 0.000221694 1.803e-05 2.14487e-05 100
end_path:out1:RootOutput 3.13e-06 4.00347e-06 2.1931e-05 3.79e-06 1.83915e-06 100
end_path:out1:RootOutput(write) 0.0193616 0.458738 1.26599 0.570489 0.265759 100
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 20896.1 MB
Peak resident set size usage (VmHWM): 2137.08 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Generated output files ===
12822.14_dunegpschedd02.fnal.gov.logs.tgz
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_498_20231207T204556Z_gen_g4_detsim_hitreco__20240510T052859Z_reco2_graph_2025-07-30T_134710Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_498_20231207T204556Z_gen_g4_detsim_hitreco__20240510T052859Z_reco2_graph_2025-07-30T_134710Z.root
debugprod.log
graph_output_2025-07-30T_134710Z_3_ana_tree_hd.root
graph_output_2025-07-30T_134710Z_3_analysiseid.root
graph_output_2025-07-30T_134710Z_3_training1_CaloHitListU_graph.data
graph_output_2025-07-30T_134710Z_3_training1_CaloHitListV_graph.data
graph_output_2025-07-30T_134710Z_3_training1_CaloHitListW_graph.data
graph_output_2025-07-30T_134710Z_3_training2_CaloHitListU_graph.data
graph_output_2025-07-30T_134710Z_3_training2_CaloHitListV_graph.data
graph_output_2025-07-30T_134710Z_3_training2_CaloHitListW_graph.data
jobscript.log
justin-processed-pfns.txt
reco_hist.root
secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_498_20231207T204556Z_gen_g4_detsim_hitreco__20240510T052859Z_reco2_graph_2025-07-30T_134710Z.log
third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_498_20231207T204556Z_gen_g4_detsim_hitreco__20240510T052859Z_reco2_graph_2025-07-30T_134710Z.log
training1_CaloHitListU.csv
training1_CaloHitListV.csv
training1_CaloHitListW.csv
training2_CaloHitListU.csv
training2_CaloHitListV.csv
training2_CaloHitListW.csv