Jobsub ID 20669.13@dunegpschedd01.fnal.gov
Jobscript log (last 10,000 characters)
1 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 3
Begin processing the 94th record. run: 6430442 subRun: 1 event: 78994 at 03-Aug-2025 19:15:51 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 6
Begin processing the 95th record. run: 6430442 subRun: 1 event: 78995 at 03-Aug-2025 19:15:53 CEST
Analysing.
Begin processing the 96th record. run: 6430442 subRun: 1 event: 78996 at 03-Aug-2025 19:15:53 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 1
Begin processing the 97th record. run: 6430442 subRun: 1 event: 78997 at 03-Aug-2025 19:15:54 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 1
Begin processing the 98th record. run: 6430442 subRun: 1 event: 78998 at 03-Aug-2025 19:15:55 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 2
Begin processing the 99th record. run: 6430442 subRun: 1 event: 78999 at 03-Aug-2025 19:15:56 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 5
Begin processing the 100th record. run: 6430442 subRun: 1 event: 79000 at 03-Aug-2025 19:15:57 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 1
03-Aug-2025 19:15:58 CEST Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/0d/7c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_789_20231205T074953Z_gen_g4_detsim_hitreco__20240510T021404Z_reco2.root"
========================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================
Full event 0.458819 0.825449 1.63052 0.820102 0.191632 100
------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.0143766 0.0220774 0.0497666 0.0226332 0.00781137 100
end_path:analysistree:AnalysisTree 0.429826 0.803266 1.60163 0.800989 0.192605 100
========================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 7329.25 MB
Peak resident set size usage (VmHWM): 974.864 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of third lar log file ===
=== Start last 100 lines of lar log file ===
* Association Name: pfShowerAssociationsbase * Instance Name: * Type: art::Assns<recob::Shower, recob::PFParticle, void>* *
* Association Name: spShowerAssociationsbase * Instance Name: * Type: art::Assns<recob::Shower, recob::SpacePoint, void>* *
*************************************************************************************************************************************
03-Aug-2025 19:06:53 CEST Initiating request to open input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/0d/7c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_789_20231205T074953Z_gen_g4_detsim_hitreco__20240510T021404Z_reco2.root"
03-Aug-2025 19:06:56 CEST Opened input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/0d/7c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_789_20231205T074953Z_gen_g4_detsim_hitreco__20240510T021404Z_reco2.root"
Begin processing the 1st record. run: 6430442 subRun: 1 event: 78901 at 03-Aug-2025 19:06:57 CEST
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:12 112 XUs and 154 XVs -> 53 XUVs
53 XUVs total
38 collection wire objects
53 potential space points
Neighbour search...
469 tests to find 286 neighbours
Iterating with no regularization...
Begin: 1.77475e+07
0 1.6963e+07
1 1.69607e+07
Now with regularization...
Begin: 1.5057e+07
0 1.50568e+07
---MC-PARTICLE-MONITORING-----------------------------------------------------------------------
BeamNeutrinos:
--Primary 0, MCPDG 13, Energy 0.19708, Dist. 26.5152, nMCHits 76 (42, 18, 16)
MCPDG 13, Energy 0.19708, Dist. 26.5152, nMCHits 74 (41, 17, 16)
\_ MCPDG 22, Energy 1.11042e-05, Dist. 0.0219891, nMCHits 2 (1, 1, 0)
--Primary 1, MCPDG 2212, Energy 1.26609, Dist. 2.91219, nMCHits 57 (20, 25, 12)
MCPDG 2212, Energy 1.26609, Dist. 2.91219, nMCHits 10 (3, 5, 2)
\_ MCPDG 2212, Energy 1.20042, Dist. 12.633, nMCHits 29 (11, 13, 5)
\_ MCPDG 2212, Energy 0.992109, Dist. 2.69906, nMCHits 15 (6, 5, 4)
\_ MCPDG 2212, Energy 0.960754, Dist. 0.559955, nMCHits 2 (0, 1, 1)
\_ MCPDG 2212, Energy 0.95437, Dist. 0.312149, nMCHits 1 (0, 1, 0)
------------------------------------------------------------------------------------------------
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Atmos_1_U_v04_03_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Atmos_1_V_v04_03_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Atmos_1_W_v04_03_00.pt'
Operating in training mode.
The eid is 0
Graph saved to training1_CaloHitListW_graph.data
Size of file training1_CaloHitListW_graph.data is 1260 bytes.
The eid is 0
Graph saved to training1_CaloHitListU_graph.data
Size of file training1_CaloHitListU_graph.data is 2252 bytes.
The eid is 0
Graph saved to training1_CaloHitListV_graph.data
Size of file training1_CaloHitListV_graph.data is 1692 bytes.
Operating in inference mode.
Operating in training mode.
The eid is -1
Graph saved to training2_CaloHitListW_graph.data
Size of file training2_CaloHitListW_graph.data is 1260 bytes.
The eid is -1
Graph saved to training2_CaloHitListU_graph.data
Size of file training2_CaloHitListU_graph.data is 2252 bytes.
The eid is -1
Graph saved to training2_CaloHitListV_graph.data
Size of file training2_CaloHitListV_graph.data is 1692 bytes.
Boundary wire vector sizes: 69, 92, 51
minwire 0: 1821
minwire 1: 1523
minwire 2: 1504
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
03-Aug-2025 19:07:00 CEST Opened output file with pattern "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_789_20231205T074953Z_gen_g4_detsim_hitreco__20240510T021404Z_reco2_graph_2025-08-03T_170617Z.root"
03-Aug-2025 19:13:22 CEST Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/0d/7c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_789_20231205T074953Z_gen_g4_detsim_hitreco__20240510T021404Z_reco2.root"
Malformed TimeTracker database. The TimeEvent table is empty, but
the TimeModule table is not. This can happen if an exception has
been thrown from a module while processing the first event. Any
saved database file is suspect and should not be used.
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 8589.11 MB
Peak resident set size usage (VmHWM): 1687.41 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 03-Aug-2025 19:13:22 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module CVNEvaluator/cvneva run: 6430442 subRun: 1 event: 78901
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path reco
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 0 entries while recob::PCAxisrecob::Showervoidart::Assns_pandoraShower__Reco2. has 100 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
=== Generated output files ===
20669.13_dunegpschedd01.fnal.gov.logs.tgz
RootOutput-91d2-5e72-f315-cb31.root
ana_tree_hd.root
analysiseid.root
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_789_20231205T074953Z_gen_g4_detsim_hitreco__20240510T021404Z_reco2_graph_2025-08-03T_170617Z.log
debugprod.log
jobscript.log
reco_hist.root
secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_789_20231205T074953Z_gen_g4_detsim_hitreco__20240510T021404Z_reco2_graph_2025-08-03T_170617Z.log
third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6430442_789_20231205T074953Z_gen_g4_detsim_hitreco__20240510T021404Z_reco2_graph_2025-08-03T_170617Z.log
training1_CaloHitListU.csv
training1_CaloHitListU_graph.data
training1_CaloHitListV.csv
training1_CaloHitListV_graph.data
training1_CaloHitListW.csv
training1_CaloHitListW_graph.data
training2_CaloHitListU.csv
training2_CaloHitListU_graph.data
training2_CaloHitListV.csv
training2_CaloHitListV_graph.data
training2_CaloHitListW.csv
training2_CaloHitListW_graph.data 