Jobsub ID 18660.1@dunegpschedd02.fnal.gov
| Jobsub ID | 18660.1@dunegpschedd02.fnal.gov |
| Workflow ID | 1227 |
| Stage ID | 1 |
| User name | ichong@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-08-17 10:51:15 |
| Site | UK_RAL-Tier1 |
| Entry | LIGO_UK_RAL_arc_ce02 |
| Last heartbeat | 2025-08-18 06:04:03 |
| From worker node | Hostname | dune001-7706150.0-lcg2633.gridpp.rl.ac.uk |
| cpuinfo | AMD EPYC 7763 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4194304000 (4000 MiB) |
| Wall seconds limit | 216000 (60 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | finished |
| Started | 2025-08-17 10:52:35 |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_1069_20231205T105950Z_gen_g4_detsim_hitreco__20240509T222603Z_reco2.root
|
| Jobscript | Exit code | 0 |
| Real time | 19h (69057s) |
| CPU time | 18h (67844s = 98%) |
| Max RSS bytes | 6462025728 (6162 MiB) |
| Outputting started | 2025-08-18 06:03:33 |
| Output files | https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/fnal/01227/1/001/truthvtx_reco_2025-08-17T_105242Z.root
|
| Finished | 2025-08-18 06:04:03 |
| Saved logs | justin-logs:18660.1-dunegpschedd02.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
Input PFN = root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/63/2d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_1069_20231205T105950Z_gen_g4_detsim_hitreco__20240509T222603Z_reco2.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
INPUT_TAR_DIR_LOCAL = /cvmfs/fifeuser3.opensciencegrid.org/sw/dune/1f8f8aaded68cc7c3e873196b083ca84b933f518
Using local products from /cvmfs/fifeuser3.opensciencegrid.org/sw/dune/1f8f8aaded68cc7c3e873196b083ca84b933f518
MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=v10_04_06d00
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/1f8f8aaded68cc7c3e873196b083ca84b933f518/truth_vtx_larsoft
MRB_SOURCE=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/1f8f8aaded68cc7c3e873196b083ca84b933f518/truth_vtx_larsoft/srcs
MRB_BUILDDIR=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/1f8f8aaded68cc7c3e873196b083ca84b933f518/truth_vtx_larsoft/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/1f8f8aaded68cc7c3e873196b083ca84b933f518/truth_vtx_larsoft/localProducts_larsoft_v10_04_06d00_e26_prof
PRODUCTS=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/1f8f8aaded68cc7c3e873196b083ca84b933f518/truth_vtx_larsoft/localProducts_larsoft_v10_04_06d00_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/1f8f8aaded68cc7c3e873196b083ca84b933f518/truth_vtx_larsoft/localProducts_larsoft_v10_04_06d00_e26_prof
local product directory is /cvmfs/fifeuser3.opensciencegrid.org/sw/dune/1f8f8aaded68cc7c3e873196b083ca84b933f518/truth_vtx_larsoft/localProducts_larsoft_v10_04_06d00_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
Running LArSoft with FCL: /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/e3904ec7e283ca65b4e732714ae2b38c8995a716/atm-reco_truth_vtx.fcl
=== Start last 100 lines of lar log file ===
DlVertexingAlgorithm::Run() called. m_trainingMode = 0
DlVertexingAlgorithm::Run() calling Infer()
DlVertexingAlgorithm::Infer() called
DlVertexingAlgorithm: Outputting the truth vertex.
Truth vertex position: -9.40248, -442.389, 306.387
%MSG-e ShowerProducedPtrsHolder: LArPandoraModularShowerCreation:pandoraShower@BeginModule 18-Aug-2025 06:03:07 UTC run: 6430442 subRun: 1 event: 106909
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Boundary wire vector sizes: 140, 91, 65
minwire 0: 668
minwire 1: 1845
minwire 2: 465
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary:
Output 0: 0.0395399,
Output 1: 0.679918, 0.00242418, 0.0235329, 0.294125,
Output 2: 0.933566, 0.0139781, 0.0014136, 0.0510421,
Output 3: 0.431141, 0.56288, 0.00584593, 0.00013329,
Output 4: 0.780804, 0.216938, 0.00199418, 0.000263819,
Output 5: 0.99412, 0.00541841, 9.11673e-05, 0.000370313,
Output 6: 0.0568907, 0.312935, 0.172076, 0.458098,
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 128!
Begin processing the 10th record. run: 6430442 subRun: 1 event: 106910 at 18-Aug-2025 06:03:20 UTC
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 20 XUs and 23 XVs -> 12 XUVs
12 XUVs total
8 collection wire objects
12 potential space points
Neighbour search...
54 tests to find 42 neighbours
Iterating with no regularization...
Begin: 116450
0 107771
1 107686
Now with regularization...
Begin: 77335.9
0 77333.9
---MC-PARTICLE-MONITORING-----------------------------------------------------------------------
------------------------------------------------------------------------------------------------
DlVertexingAlgorithm::Run() called. m_trainingMode = 0
DlVertexingAlgorithm::Run() calling Infer()
DlVertexingAlgorithm::Infer() called
DlVertexingAlgorithm: Outputting the truth vertex.
Truth vertex position: -230.178, 605.223, 153.1
DlVertexingAlgorithm::Run() called. m_trainingMode = 0
DlVertexingAlgorithm::Run() calling Infer()
DlVertexingAlgorithm::Infer() called
DlVertexingAlgorithm: Outputting the truth vertex.
Truth vertex position: -230.178, 605.223, 153.1
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2!
18-Aug-2025 06:03:31 UTC Closed output file "truthvtx_reco_2025-08-17T_105242Z.root"
18-Aug-2025 06:03:31 UTC Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/63/2d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_1069_20231205T105950Z_gen_g4_detsim_hitreco__20240509T222603Z_reco2.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 9.76466 6883.89 68718.4 11.9191 20611.5 10
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.0280827 0.0400768 0.0595452 0.029011 0.0139599 10
reco:gaushit:GausHitFinder 0.104409 0.206157 0.853922 0.120163 0.218123 10
reco:spsolve:SpacePointSolver 0.000169564 1.09324 10.4378 0.00146555 3.11729 10
reco:hitfd:DisambigFromSpacePoints 0.000345977 0.0797741 0.772838 0.00105681 0.231049 10
reco:rns:RandomNumberSaver 2.346e-05 7.32462e-05 0.000379537 3.6126e-05 0.000102774 10
reco:pandora:StandardPandora 0.0942184 6868.09 68675.6 0.118436 20602.5 10
reco:pandoraTrack:LArPandoraTrackCreation 0.000164793 0.00991157 0.0612651 0.000806216 0.0184395 10
reco:pandoraShower:LArPandoraModularShowerCreation 0.000219425 0.0174884 0.0613014 0.00134751 0.0247801 10
reco:pandoracalo:Calorimetry 0.000149932 0.0984225 0.932999 0.000581292 0.278405 10
reco:pandorapid:Chi2ParticleID 5.1031e-05 0.00040952 0.00241734 0.000120597 0.000695341 10
reco:cvnmap:CVNMapper 3.656e-05 0.0250583 0.0998551 0.0148423 0.0308473 10
reco:cvneva:CVNEvaluator 2.297e-05 1.00694 3.50088 0.651526 1.15559 10
reco:energyrecnumu:EnergyReco 0.000413238 0.0544524 0.439243 0.00572503 0.129239 10
reco:energyrecnue:EnergyReco 0.000215294 0.00396771 0.0230408 0.0027368 0.00648469 10
reco:energyrecnc:EnergyReco 0.00307003 0.00651831 0.0264177 0.00333756 0.00681425 10
reco:energyrecnumurange:EnergyReco 0.000295576 0.0514917 0.437183 0.00322683 0.129429 10
reco:energyrecnumumcs:EnergyReco 0.00310258 0.049625 0.40014 0.00475243 0.117781 10
reco:opdec:Deconvolution 0.105687 0.29402 0.653052 0.222905 0.160447 10
reco:ophitspe:OpHitFinderDeco 0.24998 0.256962 0.273321 0.254153 0.00683749 10
reco:opflash:OpFlashFinder 0.000231124 0.000801563 0.00426136 0.000323326 0.00118351 10
reco:opslicer:OpSlicer 5.6391e-05 0.0148221 0.0933928 0.00216783 0.0274105 10
[art]:TriggerResults:TriggerResultInserter 1.446e-05 2.80943e-05 0.000101171 1.709e-05 2.55561e-05 10
end_path:out1:RootOutput 3.44e-06 7.1341e-06 2.6891e-05 4.065e-06 6.87765e-06 10
end_path:out1:RootOutput(write) 9.10553 12.4907 35.1731 9.93982 7.60725 10
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 42928.9 MB
Peak resident set size usage (VmHWM): 6462.03 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Start last 100 lines of lar log file ===
tail: cannot open '' for reading: No such file or directory
=== End last 100 lines of lar log file ===
=== Generated output files ===
18660.1_dunegpschedd02.fnal.gov.logs.tgz
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_1069_20231205T105950Z_gen_g4_detsim_hitreco__20240509T222603Z_reco2_truthvtx_2025-08-17T_105242Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco_hist.root
truthvtx_reco_2025-08-17T_105242Z.root
