Jobsub ID 15593.90@dunegpschedd02.fnal.gov
| Jobsub ID | 15593.90@dunegpschedd02.fnal.gov |
| Workflow ID | 684 |
| Stage ID | 1 |
| User name | ichong@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-08-08 13:04:58 |
| Site | CERN |
| Entry | CMSHTPC_T2_CH_CERN_ce511 |
| Last heartbeat | 2025-08-09 04:15:53 |
| From worker node | Hostname | b9p25p8074.cern.ch |
| cpuinfo | AMD EPYC 7302 16-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 343800 (95 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | finished |
| Started | 2025-08-08 13:08:11 |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_1069_20231205T105950Z_gen_g4_detsim_hitreco__20240509T222603Z_reco2.root
|
| Jobscript | Exit code | 0 |
| Real time | 15h (54104s) |
| CPU time | 14h (52490s = 97%) |
| Max RSS bytes | 6341861376 (6048 MiB) |
| Outputting started | 2025-08-09 04:09:56 |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00684/1/001/caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_1069_20231205T105950Z_gen_g4_detsim_hitreco__20240509T222603Z_reco2_truthvtx_2025-08-09T_040910Z_caf.root
|
| Finished | 2025-08-09 04:15:53 |
| Saved logs | justin-logs:15593.90-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
ct directory is /cvmfs/fifeuser3.opensciencegrid.org/sw/dune/1f8f8aaded68cc7c3e873196b083ca84b933f518/truth_vtx_larsoft/localProducts_larsoft_v10_04_06d00_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
Running LArSoft with FCL: /cvmfs/fifeuser3.opensciencegrid.org/sw/dune/aaa05dc551a4535ad0eeaa35b47a4bc39c7f55ae/atm-reco_truth_vtx.fcl
Running CAF step with FCL_SECONDARY: cafmaker_atmos_dune10kt_1x2x6_runreco-nuenergy-nuangular_geov5.fcl using reco output: atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_1069_20231205T105950Z_gen_g4_detsim_hitreco__20240509T222603Z_reco2_truthvtx_2025-08-08T_130818Z.root
Renamed caf.root -> caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_1069_20231205T105950Z_gen_g4_detsim_hitreco__20240509T222603Z_reco2_truthvtx_2025-08-09T_040910Z_caf.root
=== Start last 100 lines of lar log file ===
MCPDG 2212, Energy 1.06448, Dist. 11.7455, nMCHits 25 (15, 2, 8)
--Primary 5, MCPDG 2212, Energy 1.03742, Dist. 7.78886, nMCHits 21 (12, 4, 5)
MCPDG 2212, Energy 1.03742, Dist. 7.78886, nMCHits 21 (12, 4, 5)
------------------------------------------------------------------------------------------------
DlVertexingAlgorithm::Run() called. m_trainingMode = 0
DlVertexingAlgorithm::Run() calling Infer()
DlVertexingAlgorithm::Infer() called
DlVertexingAlgorithm: Outputting the truth vertex.
Truth vertex position: -7.33704, -475.913, 717.201
DlVertexingAlgorithm::Run() called. m_trainingMode = 0
DlVertexingAlgorithm::Run() calling Infer()
DlVertexingAlgorithm::Infer() called
DlVertexingAlgorithm: Outputting the truth vertex.
Truth vertex position: -7.33704, -475.913, 717.201
%MSG-e ShowerProducedPtrsHolder: LArPandoraModularShowerCreation:pandoraShower@BeginModule 09-Aug-2025 06:09:05 CEST run: 6430442 subRun: 1 event: 106999
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder: LArPandoraModularShowerCreation:pandoraShower@BeginModule 09-Aug-2025 06:09:05 CEST run: 6430442 subRun: 1 event: 106999
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder: LArPandoraModularShowerCreation:pandoraShower@BeginModule 09-Aug-2025 06:09:05 CEST run: 6430442 subRun: 1 event: 106999
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 960, 1223, 945
minwire 0: 1196
minwire 1: 1046
minwire 2: 970
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary:
Output 0: 0.00846572,
Output 1: 0.00336282, 0.335693, 0.00330656, 0.657638,
Output 2: 0.877919, 0.101316, 0.0162476, 0.00451715,
Output 3: 0.896268, 0.101144, 0.00165939, 0.000929549,
Output 4: 0.990256, 0.00659799, 0.00105554, 0.00209029,
Output 5: 0.286805, 0.70768, 0.00485872, 0.000656513,
Output 6: 0.660331, 0.315457, 0.0208747, 0.00333661,
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 387!
Begin processing the 100th record. run: 6430442 subRun: 1 event: 107000 at 09-Aug-2025 06:09:07 CEST
---MC-PARTICLE-MONITORING-----------------------------------------------------------------------
------------------------------------------------------------------------------------------------
DlVertexingAlgorithm::Run() called. m_trainingMode = 0
DlVertexingAlgorithm::Run() calling Infer()
DlVertexingAlgorithm::Infer() called
DlVertexingAlgorithm: Outputting the truth vertex.
Truth vertex position: -109.677, -329.731, 1141.83
DlVertexingAlgorithm::Run() called. m_trainingMode = 0
DlVertexingAlgorithm::Run() calling Infer()
DlVertexingAlgorithm::Infer() called
DlVertexingAlgorithm: Outputting the truth vertex.
Truth vertex position: -109.677, -329.731, 1141.83
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 17!
09-Aug-2025 06:09:09 CEST Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_1069_20231205T105950Z_gen_g4_detsim_hitreco__20240509T222603Z_reco2_truthvtx_2025-08-08T_130818Z.root"
09-Aug-2025 06:09:09 CEST Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/63/2d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_1069_20231205T105950Z_gen_g4_detsim_hitreco__20240509T222603Z_reco2.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 0.388503 532.816 53027 2.28103 5275.87 100
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.0163093 0.0349309 0.0661715 0.0338502 0.012362 100
reco:gaushit:GausHitFinder 0.0620936 0.110949 0.504688 0.0817984 0.0753923 100
reco:spsolve:SpacePointSolver 9.4901e-05 0.101908 7.33973 0.00332451 0.733162 100
reco:hitfd:DisambigFromSpacePoints 0.000232574 0.0109204 0.410617 0.0011351 0.0430726 100
reco:rns:RandomNumberSaver 1.691e-05 3.01058e-05 0.000332325 2.4705e-05 3.15165e-05 100
reco:pandora:StandardPandora 0.0533543 530.751 53021.1 0.0980636 5275.48 100
reco:pandoraTrack:LArPandoraTrackCreation 0.000110092 0.00503512 0.0594399 0.00128536 0.00935651 100
reco:pandoraShower:LArPandoraModularShowerCreation 0.000161882 0.014745 0.167755 0.00268046 0.030628 100
reco:pandoracalo:Calorimetry 9.3381e-05 0.0141736 0.579105 0.000965674 0.0602123 100
reco:pandorapid:Chi2ParticleID 3.6391e-05 0.000162826 0.00151302 0.000117087 0.000166714 100
reco:cvnmap:CVNMapper 2.444e-05 0.0194892 0.0882214 0.0201509 0.0152164 100
reco:cvneva:CVNEvaluator 1.5031e-05 0.851836 2.17902 1.12509 0.521996 100
reco:energyrecnumu:EnergyReco 0.000298484 0.0375899 0.587922 0.00739521 0.0915693 100
reco:energyrecnue:EnergyReco 0.000139572 0.00175878 0.0191839 0.000683455 0.00270073 100
reco:energyrecnc:EnergyReco 0.00176318 0.0039753 0.0218603 0.00296428 0.0028965 100
reco:energyrecnumurange:EnergyReco 0.000158683 0.0344982 0.585039 0.0026901 0.0914996 100
reco:energyrecnumumcs:EnergyReco 0.00132915 0.0360886 0.595595 0.00349444 0.0923622 100
reco:opdec:Deconvolution 0.0592352 0.243227 0.514376 0.25827 0.0922319 100
reco:ophitspe:OpHitFinderDeco 0.14083 0.149986 0.162392 0.149658 0.00317067 100
reco:opflash:OpFlashFinder 8.9701e-05 0.00044792 0.00357026 0.000402816 0.000358092 100
reco:opslicer:OpSlicer 2.866e-05 0.0155371 0.0821355 0.00617837 0.0207972 100
[art]:TriggerResults:TriggerResultInserter 1.0151e-05 1.74717e-05 4.2951e-05 1.692e-05 4.59511e-06 100
end_path:out1:RootOutput 3.41e-06 4.74598e-06 2.1301e-05 4.355e-06 1.95198e-06 100
end_path:out1:RootOutput(write) 0.00894675 0.375582 0.936551 0.452811 0.212223 100
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 16909.9 MB
Peak resident set size usage (VmHWM): 6341.86 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Generated output files ===
15593.90_dunegpschedd02.fnal.gov.logs.tgz
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_1069_20231205T105950Z_gen_g4_detsim_hitreco__20240509T222603Z_reco2_truthvtx_2025-08-08T_130818Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_1069_20231205T105950Z_gen_g4_detsim_hitreco__20240509T222603Z_reco2_truthvtx_2025-08-08T_130818Z.root
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_1069_20231205T105950Z_gen_g4_detsim_hitreco__20240509T222603Z_reco2_truthvtx_2025-08-09T_040910Z.log
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_1069_20231205T105950Z_gen_g4_detsim_hitreco__20240509T222603Z_reco2_truthvtx_2025-08-09T_040910Z.root
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_1069_20231205T105950Z_gen_g4_detsim_hitreco__20240509T222603Z_reco2_truthvtx_2025-08-09T_040910Z_caf.root
debugprod.log
flatcaf.root
jobscript.log
justin-processed-pfns.txt
reco_hist.root
