Jobsub ID 15982.1@dunegpschedd02.fnal.gov
| Jobsub ID | 15982.1@dunegpschedd02.fnal.gov |
| Workflow ID | 682 |
| Stage ID | 1 |
| User name | ichong@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-08-09 12:56:09 |
| Site | ES_PIC |
| Entry | DUNE_T1_ES_PIC_ce15-multicore |
| Last heartbeat | 2025-08-10 02:51:22 |
| From worker node | Hostname | hnode60.pic.es |
| cpuinfo | AMD EPYC 7402P 24-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | finished |
| Started | 2025-08-09 12:57:30 |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_1069_20231205T105950Z_gen_g4_detsim_hitreco__20240509T222603Z_reco2.root
|
| Jobscript | Exit code | 0 |
| Real time | 13h (50001s) |
| CPU time | 13h (49042s = 98%) |
| Max RSS bytes | 6559289344 (6255 MiB) |
| Outputting started | 2025-08-10 02:50:52 |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/00682/1/001/caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_1069_20231205T105950Z_gen_g4_detsim_hitreco__20240509T222603Z_reco2_truthvtx_2025-08-10T_025033Z_caf.root
|
| Finished | 2025-08-10 02:51:22 |
| Saved logs | justin-logs:15982.1-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
rid.org/sw/dune/1f8f8aaded68cc7c3e873196b083ca84b933f518/truth_vtx_larsoft/localProducts_larsoft_v10_04_06d00_e26_prof
PRODUCTS=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/1f8f8aaded68cc7c3e873196b083ca84b933f518/truth_vtx_larsoft/localProducts_larsoft_v10_04_06d00_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/1f8f8aaded68cc7c3e873196b083ca84b933f518/truth_vtx_larsoft/localProducts_larsoft_v10_04_06d00_e26_prof
local product directory is /cvmfs/fifeuser3.opensciencegrid.org/sw/dune/1f8f8aaded68cc7c3e873196b083ca84b933f518/truth_vtx_larsoft/localProducts_larsoft_v10_04_06d00_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
Running LArSoft with FCL: /cvmfs/fifeuser3.opensciencegrid.org/sw/dune/7e4ea945688a9906bc9086d3744735f9a7bd40fa/atm-reco_truth_vtx.fcl
Running CAF step with FCL_SECONDARY: cafmaker_atmos_dune10kt_1x2x6_runreco-nuenergy-nuangular_geov5.fcl using reco output: atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_1069_20231205T105950Z_gen_g4_detsim_hitreco__20240509T222603Z_reco2_truthvtx_2025-08-09T_125734Z.root
Renamed caf.root -> caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_1069_20231205T105950Z_gen_g4_detsim_hitreco__20240509T222603Z_reco2_truthvtx_2025-08-10T_025033Z_caf.root
=== Start last 100 lines of lar log file ===
DlVertexingAlgorithm::Run() called. m_trainingMode = 0
DlVertexingAlgorithm::Run() calling Infer()
DlVertexingAlgorithm::Infer() called
DlVertexingAlgorithm: Outputting the truth vertex.
Truth vertex position: -9.40248, -442.389, 306.387
%MSG-e ShowerProducedPtrsHolder: LArPandoraModularShowerCreation:pandoraShower@BeginModule 10-Aug-2025 04:49:54 CEST run: 6430442 subRun: 1 event: 106909
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 140, 91, 65
minwire 0: 668
minwire 1: 1845
minwire 2: 465
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary:
Output 0: 0.0395399,
Output 1: 0.679918, 0.00242418, 0.0235329, 0.294125,
Output 2: 0.933566, 0.0139781, 0.0014136, 0.0510421,
Output 3: 0.431141, 0.56288, 0.00584593, 0.00013329,
Output 4: 0.780804, 0.216938, 0.00199418, 0.000263819,
Output 5: 0.99412, 0.00541841, 9.11673e-05, 0.000370313,
Output 6: 0.0568907, 0.312935, 0.172076, 0.458098,
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 128!
Begin processing the 10th record. run: 6430442 subRun: 1 event: 106910 at 10-Aug-2025 04:50:13 CEST
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 20 XUs and 23 XVs -> 12 XUVs
12 XUVs total
8 collection wire objects
12 potential space points
Neighbour search...
54 tests to find 42 neighbours
Iterating with no regularization...
Begin: 116450
0 107771
1 107686
Now with regularization...
Begin: 77335.9
0 77333.9
---MC-PARTICLE-MONITORING-----------------------------------------------------------------------
------------------------------------------------------------------------------------------------
DlVertexingAlgorithm::Run() called. m_trainingMode = 0
DlVertexingAlgorithm::Run() calling Infer()
DlVertexingAlgorithm::Infer() called
DlVertexingAlgorithm: Outputting the truth vertex.
Truth vertex position: -230.178, 605.223, 153.1
DlVertexingAlgorithm::Run() called. m_trainingMode = 0
DlVertexingAlgorithm::Run() calling Infer()
DlVertexingAlgorithm::Infer() called
DlVertexingAlgorithm: Outputting the truth vertex.
Truth vertex position: -230.178, 605.223, 153.1
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2!
10-Aug-2025 04:50:32 CEST Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_1069_20231205T105950Z_gen_g4_detsim_hitreco__20240509T222603Z_reco2_truthvtx_2025-08-09T_125734Z.root"
10-Aug-2025 04:50:32 CEST Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/63/2d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_1069_20231205T105950Z_gen_g4_detsim_hitreco__20240509T222603Z_reco2.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 17.1251 4987.89 49661.3 18.836 14891.1 10
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.0498794 0.0703775 0.103033 0.0513209 0.0244226 10
reco:gaushit:GausHitFinder 0.161361 0.226355 0.651797 0.168549 0.14331 10
reco:spsolve:SpacePointSolver 0.000112245 0.75961 7.31449 0.000865827 2.18606 10
reco:hitfd:DisambigFromSpacePoints 0.000190438 0.0410447 0.394665 0.000506422 0.1179 10
reco:rns:RandomNumberSaver 1.4841e-05 3.33503e-05 0.000173468 1.7005e-05 4.67914e-05 10
reco:pandora:StandardPandora 0.157019 4960.15 49558.7 0.169431 14866.2 10
reco:pandoraTrack:LArPandoraTrackCreation 0.000103435 0.00573838 0.0395379 0.000453029 0.0116331 10
reco:pandoraShower:LArPandoraModularShowerCreation 0.000143026 0.00887049 0.0334335 0.0007037 0.0126416 10
reco:pandoracalo:Calorimetry 8.3633e-05 0.0418388 0.389597 0.000289182 0.116091 10
reco:pandorapid:Chi2ParticleID 3.1531e-05 0.000185565 0.00110945 6.1038e-05 0.000313907 10
reco:cvnmap:CVNMapper 2.067e-05 0.0114274 0.046228 0.00633662 0.0146017 10
reco:cvneva:CVNEvaluator 1.3151e-05 0.541287 2.23453 0.387452 0.677488 10
reco:energyrecnumu:EnergyReco 0.000289563 0.0274727 0.209163 0.0025496 0.0614001 10
reco:energyrecnue:EnergyReco 0.000125256 0.0021241 0.0102089 0.0016637 0.00284053 10
reco:energyrecnc:EnergyReco 0.00188125 0.00349354 0.0130187 0.00225986 0.00323493 10
reco:energyrecnumurange:EnergyReco 0.000280792 0.0251238 0.204435 0.0021016 0.0604642 10
reco:energyrecnumumcs:EnergyReco 0.0018609 0.0267513 0.211975 0.00296437 0.0623984 10
reco:opdec:Deconvolution 0.169676 6.62536 63.797 0.258526 19.0574 10
reco:ophitspe:OpHitFinderDeco 0.445084 0.452239 0.467503 0.448121 0.00745929 10
reco:opflash:OpFlashFinder 0.000123505 0.000428703 0.00214558 0.000199093 0.000588655 10
reco:opslicer:OpSlicer 3.4471e-05 0.00827163 0.050441 0.00133123 0.0148059 10
[art]:TriggerResults:TriggerResultInserter 9.5e-06 1.53075e-05 3.3831e-05 1.18355e-05 7.66941e-06 10
end_path:out1:RootOutput 2.57e-06 4.0531e-06 1.228e-05 3.16e-06 2.75464e-06 10
end_path:out1:RootOutput(write) 16.013 18.8567 34.9357 16.8746 5.51407 10
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 14939.9 MB
Peak resident set size usage (VmHWM): 6559.29 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Generated output files ===
15982.1_dunegpschedd02.fnal.gov.logs.tgz
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_1069_20231205T105950Z_gen_g4_detsim_hitreco__20240509T222603Z_reco2_truthvtx_2025-08-09T_125734Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_1069_20231205T105950Z_gen_g4_detsim_hitreco__20240509T222603Z_reco2_truthvtx_2025-08-09T_125734Z.root
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_1069_20231205T105950Z_gen_g4_detsim_hitreco__20240509T222603Z_reco2_truthvtx_2025-08-10T_025033Z.log
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_1069_20231205T105950Z_gen_g4_detsim_hitreco__20240509T222603Z_reco2_truthvtx_2025-08-10T_025033Z.root
caf_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_1069_20231205T105950Z_gen_g4_detsim_hitreco__20240509T222603Z_reco2_truthvtx_2025-08-10T_025033Z_caf.root
debugprod.log
flatcaf.root
jobscript.log
justin-processed-pfns.txt
reco_hist.root
