Jobsub ID 14208.0@dunegpschedd02.fnal.gov
Jobscript log (last 10,000 characters)
1 event: 12895 at 03-Aug-2025 20:05:23 CEST
Analysing.
Warning: there was no vertex found for PFParticle with ID 0
Warning: there was no track found for track-like PFParticle with ID 0
Warning: there was no track found for track-like PFParticle with ID 1
Begin processing the 96th record. run: 6374820 subRun: 1 event: 12896 at 03-Aug-2025 20:05:23 CEST
Analysing.
Warning: there was no vertex found for PFParticle with ID 2
Warning: there was no track found for track-like PFParticle with ID 2
Warning: there was no track found for track-like PFParticle with ID 3
Begin processing the 97th record. run: 6374820 subRun: 1 event: 12897 at 03-Aug-2025 20:05:23 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 1
Begin processing the 98th record. run: 6374820 subRun: 1 event: 12898 at 03-Aug-2025 20:05:23 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 1
Begin processing the 99th record. run: 6374820 subRun: 1 event: 12899 at 03-Aug-2025 20:05:23 CEST
Analysing.
Warning: there was no vertex found for PFParticle with ID 2
Warning: there was no track found for track-like PFParticle with ID 2
Warning: there was no track found for track-like PFParticle with ID 3
Begin processing the 100th record. run: 6374820 subRun: 1 event: 12900 at 03-Aug-2025 20:05:23 CEST
Analysing.
Warning: there was no vertex found for PFParticle with ID 8
Warning: there was no track found for track-like PFParticle with ID 8
Warning: there was no track found for track-like PFParticle with ID 9
03-Aug-2025 20:05:23 CEST Closed input file "root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/45/31/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_128_20231201T050554Z_gen_g4_detsim_hitreco__20240508T065402Z_reco2.root"
========================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================
Full event 0.0098909 0.0327238 0.230691 0.0219746 0.0387676 100
------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000466828 0.000860791 0.00140609 0.000858087 0.000193348 100
end_path:analysistree:AnalysisTree 0.00928129 0.0317666 0.230009 0.0210098 0.0387561 100
========================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 7211.84 MB
Peak resident set size usage (VmHWM): 860.168 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of third lar log file ===
=== Start last 100 lines of lar log file ===
*************************************************************************************************************************************
03-Aug-2025 20:02:17 CEST Initiating request to open input file "root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/45/31/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_128_20231201T050554Z_gen_g4_detsim_hitreco__20240508T065402Z_reco2.root"
03-Aug-2025 20:03:04 CEST Opened input file "root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/45/31/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_128_20231201T050554Z_gen_g4_detsim_hitreco__20240508T065402Z_reco2.root"
Begin processing the 1st record. run: 6374820 subRun: 1 event: 12801 at 03-Aug-2025 20:03:04 CEST
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:8 865 XUs and 827 XVs -> 407 XUVs
C:0 T:11 1 XUs and 0 XVs -> 0 XUVs
C:0 T:12 15 XUs and 15 XVs -> 11 XUVs
418 XUVs total
134 collection wire objects
418 potential space points
Neighbour search...
33306 tests to find 20966 neighbours
Iterating with no regularization...
Begin: 1.36998e+07
0 1.02271e+07
1 9.90644e+06
2 9.85422e+06
3 9.84314e+06
4 9.83679e+06
Now with regularization...
Begin: 6.77762e+06
0 6.76568e+06
1 6.76117e+06
---MC-PARTICLE-MONITORING-----------------------------------------------------------------------
BeamNeutrinos:
--Primary 0, MCPDG -11, Energy 0.46865, Dist. 27.4425, nMCHits 586 (225, 189, 172)
MCPDG -11, Energy 0.46865, Dist. 27.4425, nMCHits 586 (225, 189, 172)
--Primary 1, MCPDG -211, Energy 0.235658, Dist. 25.7907, nMCHits 69 (25, 36, 8)
MCPDG -211, Energy 0.235658, Dist. 25.7907, nMCHits 66 (24, 34, 8)
\_ MCPDG 22, Energy 4.41644e-05, Dist. 0.607415, nMCHits 3 (1, 2, 0)
------------------------------------------------------------------------------------------------
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Atmos_1_U_v04_03_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Atmos_1_V_v04_03_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Atmos_1_W_v04_03_00.pt'
Operating in training mode.
The eid is 0
Graph saved to training1_CaloHitListW_graph.data
Size of file training1_CaloHitListW_graph.data is 5084 bytes.
The eid is 0
Graph saved to training1_CaloHitListU_graph.data
Size of file training1_CaloHitListU_graph.data is 6596 bytes.
The eid is 0
Graph saved to training1_CaloHitListV_graph.data
Size of file training1_CaloHitListV_graph.data is 6004 bytes.
Operating in inference mode.
Operating in training mode.
The eid is -1
Graph saved to training2_CaloHitListW_graph.data
Size of file training2_CaloHitListW_graph.data is 5084 bytes.
The eid is -1
Graph saved to training2_CaloHitListU_graph.data
Size of file training2_CaloHitListU_graph.data is 6596 bytes.
The eid is -1
Graph saved to training2_CaloHitListV_graph.data
Size of file training2_CaloHitListV_graph.data is 6004 bytes.
Boundary wire vector sizes: 246, 268, 208
minwire 0: 1507
minwire 1: 1543
minwire 2: 1253
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
03-Aug-2025 20:03:07 CEST Opened output file with pattern "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_128_20231201T050554Z_gen_g4_detsim_hitreco__20240508T065402Z_reco2_graph_2025-08-03T_180141Z.root"
03-Aug-2025 20:03:15 CEST Closed input file "root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/45/31/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_128_20231201T050554Z_gen_g4_detsim_hitreco__20240508T065402Z_reco2.root"
Malformed TimeTracker database. The TimeEvent table is empty, but
the TimeModule table is not. This can happen if an exception has
been thrown from a module while processing the first event. Any
saved database file is suspect and should not be used.
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 8589.87 MB
Peak resident set size usage (VmHWM): 1687.41 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 03-Aug-2025 20:03:16 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module CVNEvaluator/cvneva run: 6374820 subRun: 1 event: 12801
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path reco
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 0 entries while recob::PCAxisrecob::Showervoidart::Assns_pandoraShower__Reco2. has 100 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
=== Generated output files ===
14208.0_dunegpschedd02.fnal.gov.logs.tgz
RootOutput-ead2-b706-4639-e8a5.root
ana_tree_hd.root
analysiseid.root
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_128_20231201T050554Z_gen_g4_detsim_hitreco__20240508T065402Z_reco2_graph_2025-08-03T_180141Z.log
debugprod.log
jobscript.log
reco_hist.root
secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_128_20231201T050554Z_gen_g4_detsim_hitreco__20240508T065402Z_reco2_graph_2025-08-03T_180141Z.log
third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_128_20231201T050554Z_gen_g4_detsim_hitreco__20240508T065402Z_reco2_graph_2025-08-03T_180141Z.log
training1_CaloHitListU.csv
training1_CaloHitListU_graph.data
training1_CaloHitListV.csv
training1_CaloHitListV_graph.data
training1_CaloHitListW.csv
training1_CaloHitListW_graph.data
training2_CaloHitListU.csv
training2_CaloHitListU_graph.data
training2_CaloHitListV.csv
training2_CaloHitListV_graph.data
training2_CaloHitListW.csv
training2_CaloHitListW_graph.data 