Jobsub ID 14187.1@dunegpschedd02.fnal.gov
Jobscript log (last 10,000 characters)
rack #0 has 3075 hits on calorimetry plane #0, only 2000 stored in tree
%MSG
Begin processing the 94th record. run: 6432575 subRun: 1 event: 73094 at 03-Aug-2025 18:07:55 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 2
Begin processing the 95th record. run: 6432575 subRun: 1 event: 73095 at 03-Aug-2025 18:07:56 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 1
Warning: there was no track found for track-like PFParticle with ID 2
Begin processing the 96th record. run: 6432575 subRun: 1 event: 73096 at 03-Aug-2025 18:07:57 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 1
Warning: there was no track found for track-like PFParticle with ID 2
Begin processing the 97th record. run: 6432575 subRun: 1 event: 73097 at 03-Aug-2025 18:07:58 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 2
Begin processing the 98th record. run: 6432575 subRun: 1 event: 73098 at 03-Aug-2025 18:07:59 CEST
Analysing.
Warning: there was no vertex found for PFParticle with ID 4
Warning: there was no track found for track-like PFParticle with ID 4
Warning: there was no track found for track-like PFParticle with ID 5
Begin processing the 99th record. run: 6432575 subRun: 1 event: 73099 at 03-Aug-2025 18:08:00 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 1
Warning: there was no track found for track-like PFParticle with ID 2
Begin processing the 100th record. run: 6432575 subRun: 1 event: 73100 at 03-Aug-2025 18:08:01 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 2
03-Aug-2025 18:08:02 CEST Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f6/8e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432575_730_20231207T035040Z_gen_g4_detsim_hitreco__20240510T045532Z_reco2.root"
========================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================
Full event 0.437961 0.77852 1.15355 0.796342 0.131638 100
------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.0143197 0.0215912 0.0351787 0.0216409 0.00717132 100
end_path:analysistree:AnalysisTree 0.423378 0.756819 1.12493 0.777363 0.131673 100
========================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 7257.29 MB
Peak resident set size usage (VmHWM): 904.786 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of third lar log file ===
=== Start last 100 lines of lar log file ===
03-Aug-2025 17:59:12 CEST Initiating request to open input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f6/8e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432575_730_20231207T035040Z_gen_g4_detsim_hitreco__20240510T045532Z_reco2.root"
03-Aug-2025 17:59:14 CEST Opened input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f6/8e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432575_730_20231207T035040Z_gen_g4_detsim_hitreco__20240510T045532Z_reco2.root"
Begin processing the 1st record. run: 6432575 subRun: 1 event: 73001 at 03-Aug-2025 17:59:15 CEST
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:3 957 XUs and 322 XVs -> 165 XUVs
165 XUVs total
70 collection wire objects
165 potential space points
Neighbour search...
6179 tests to find 4032 neighbours
Iterating with no regularization...
Begin: 1.08899e+07
0 8.63538e+06
1 8.43738e+06
2 8.37768e+06
3 8.35847e+06
4 8.35067e+06
Now with regularization...
Begin: 6.94468e+06
0 6.93758e+06
1 6.93253e+06
---MC-PARTICLE-MONITORING-----------------------------------------------------------------------
BeamNeutrinos:
--Primary 0, MCPDG 211, Energy 0.427463, Dist. 38.7183, nMCHits 304 (160, 59, 85)
MCPDG 211, Energy 0.427463, Dist. 38.7183, nMCHits 112 (62, 24, 26)
\_ MCPDG 2212, Energy 0.997511, Dist. 3.13933, nMCHits 12 (5, 4, 3)
\_ MCPDG 211, Energy 0.264475, Dist. 35.4139, nMCHits 118 (64, 19, 35)
\_ MCPDG -13, Energy 0.109778, Dist. 0.147511, nMCHits 5 (2, 1, 2)
\_ MCPDG -11, Energy 0.0435383, Dist. 7.25496, nMCHits 57 (27, 11, 19)
--Primary 1, MCPDG 13, Energy 0.379712, Dist. 26.1997, nMCHits 49 (8, 24, 17)
MCPDG 13, Energy 0.379712, Dist. 26.1997, nMCHits 49 (8, 24, 17)
------------------------------------------------------------------------------------------------
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Atmos_1_U_v04_03_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Atmos_1_V_v04_03_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Atmos_1_W_v04_03_00.pt'
Operating in training mode.
The eid is 0
Graph saved to training1_CaloHitListW_graph.data
Size of file training1_CaloHitListW_graph.data is 2764 bytes.
The eid is 0
Graph saved to training1_CaloHitListU_graph.data
Size of file training1_CaloHitListU_graph.data is 4396 bytes.
The eid is 0
Graph saved to training1_CaloHitListV_graph.data
Size of file training1_CaloHitListV_graph.data is 2236 bytes.
Operating in inference mode.
Operating in training mode.
The eid is -1
Graph saved to training2_CaloHitListW_graph.data
Size of file training2_CaloHitListW_graph.data is 2764 bytes.
The eid is -1
Graph saved to training2_CaloHitListU_graph.data
Size of file training2_CaloHitListU_graph.data is 4396 bytes.
The eid is -1
Graph saved to training2_CaloHitListV_graph.data
Size of file training2_CaloHitListV_graph.data is 2236 bytes.
Boundary wire vector sizes: 180, 92, 113
minwire 0: 441
minwire 1: 2709
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
03-Aug-2025 17:59:19 CEST Opened output file with pattern "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432575_730_20231207T035040Z_gen_g4_detsim_hitreco__20240510T045532Z_reco2_graph_2025-08-03T_155836Z.root"
03-Aug-2025 18:05:35 CEST Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f6/8e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432575_730_20231207T035040Z_gen_g4_detsim_hitreco__20240510T045532Z_reco2.root"
Malformed TimeTracker database. The TimeEvent table is empty, but
the TimeModule table is not. This can happen if an exception has
been thrown from a module while processing the first event. Any
saved database file is suspect and should not be used.
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 8587.68 MB
Peak resident set size usage (VmHWM): 1682.85 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 03-Aug-2025 18:05:35 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module CVNEvaluator/cvneva run: 6432575 subRun: 1 event: 73001
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path reco
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 0 entries while recob::PCAxisrecob::Showervoidart::Assns_pandoraShower__Reco2. has 100 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
=== Generated output files ===
14187.1_dunegpschedd02.fnal.gov.logs.tgz
RootOutput-38bb-4f48-b31d-227a.root
ana_tree_hd.root
analysiseid.root
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432575_730_20231207T035040Z_gen_g4_detsim_hitreco__20240510T045532Z_reco2_graph_2025-08-03T_155836Z.log
debugprod.log
jobscript.log
reco_hist.root
secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432575_730_20231207T035040Z_gen_g4_detsim_hitreco__20240510T045532Z_reco2_graph_2025-08-03T_155836Z.log
third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432575_730_20231207T035040Z_gen_g4_detsim_hitreco__20240510T045532Z_reco2_graph_2025-08-03T_155836Z.log
training1_CaloHitListU.csv
training1_CaloHitListU_graph.data
training1_CaloHitListV.csv
training1_CaloHitListV_graph.data
training1_CaloHitListW.csv
training1_CaloHitListW_graph.data
training2_CaloHitListU.csv
training2_CaloHitListU_graph.data
training2_CaloHitListV.csv
training2_CaloHitListV_graph.data
training2_CaloHitListW.csv
training2_CaloHitListW_graph.data 