Jobsub ID 20206.9@dunegpschedd01.fnal.gov
Jobscript log (last 10,000 characters)
ocessing the 95th record. run: 74517726 subRun: 1 event: 116795 at 01-Aug-2025 16:03:42 CEST
Analysing.
Warning: there was no vertex found for PFParticle with ID 0
Warning: there was no track found for track-like PFParticle with ID 0
Warning: there was no track found for track-like PFParticle with ID 1
Begin processing the 96th record. run: 74517726 subRun: 1 event: 116796 at 01-Aug-2025 16:03:42 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 6
Begin processing the 97th record. run: 74517726 subRun: 1 event: 116797 at 01-Aug-2025 16:03:42 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 6
Begin processing the 98th record. run: 74517726 subRun: 1 event: 116798 at 01-Aug-2025 16:03:42 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 2
Begin processing the 99th record. run: 74517726 subRun: 1 event: 116799 at 01-Aug-2025 16:03:42 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 2
Begin processing the 100th record. run: 74517726 subRun: 1 event: 116800 at 01-Aug-2025 16:03:42 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 1
01-Aug-2025 16:03:42 CEST Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/45/b5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_1167_20231204T100543Z_gen_g4_detsim_hitreco__20240509T212618Z_reco2.root"
========================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================
Full event 0.00869876 0.0410249 0.348884 0.0187753 0.0605833 100
------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000444133 0.000762036 0.00149912 0.000724974 0.000216806 100
end_path:analysistree:AnalysisTree 0.0079608 0.0401391 0.348079 0.0180047 0.0605867 100
========================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 7294.51 MB
Peak resident set size usage (VmHWM): 941.826 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of third lar log file ===
=== Start last 100 lines of lar log file ===
* Association Name: hitAssociationsbase * Instance Name: * Type: art::Assns<recob::Shower, recob::Hit, void>* *
* Association Name: pfShowerAssociationsbase * Instance Name: * Type: art::Assns<recob::Shower, recob::PFParticle, void>* *
* Association Name: spShowerAssociationsbase * Instance Name: * Type: art::Assns<recob::Shower, recob::SpacePoint, void>* *
*************************************************************************************************************************************
01-Aug-2025 16:02:43 CEST Initiating request to open input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/45/b5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_1167_20231204T100543Z_gen_g4_detsim_hitreco__20240509T212618Z_reco2.root"
01-Aug-2025 16:02:45 CEST Opened input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/45/b5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_1167_20231204T100543Z_gen_g4_detsim_hitreco__20240509T212618Z_reco2.root"
Begin processing the 1st record. run: 74517726 subRun: 1 event: 116701 at 01-Aug-2025 16:02:46 CEST
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:7 743 XUs and 808 XVs -> 271 XUVs
C:0 T:11 64 XUs and 67 XVs -> 33 XUVs
C:0 T:14 2 XUs and 2 XVs -> 2 XUVs
306 XUVs total
170 collection wire objects
306 potential space points
Neighbour search...
7206 tests to find 3434 neighbours
Iterating with no regularization...
Begin: 1.7755e+07
0 1.59257e+07
1 1.57992e+07
2 1.57928e+07
Now with regularization...
Begin: 1.41712e+07
0 1.41685e+07
---MC-PARTICLE-MONITORING-----------------------------------------------------------------------
BeamNeutrinos:
--Primary 0, MCPDG 11, Energy 1.531, Dist. 31.5627, nMCHits 946 (349, 379, 218)
MCPDG 11, Energy 1.531, Dist. 31.5627, nMCHits 902 (336, 362, 204)
\_ MCPDG -13, Energy 0.109778, Dist. 0.149228, nMCHits 2 (0, 2, 0)
\_ MCPDG -11, Energy 0.0300151, Dist. 8.49634, nMCHits 42 (13, 15, 14)
--Primary 1, MCPDG 2212, Energy 1.02696, Dist. 6.28987, nMCHits 16 (7, 4, 5)
MCPDG 2212, Energy 1.02696, Dist. 6.28987, nMCHits 16 (7, 4, 5)
------------------------------------------------------------------------------------------------
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Atmos_1_U_v04_03_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Atmos_1_V_v04_03_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Atmos_1_W_v04_03_00.pt'
Operating in training mode.
The eid is 0
Graph saved to training1_CaloHitListW_graph.data
Size of file training1_CaloHitListW_graph.data is 5732 bytes.
The eid is 0
Graph saved to training1_CaloHitListU_graph.data
Size of file training1_CaloHitListU_graph.data is 9028 bytes.
The eid is 0
Graph saved to training1_CaloHitListV_graph.data
Size of file training1_CaloHitListV_graph.data is 9732 bytes.
Operating in inference mode.
Operating in training mode.
The eid is -1
Graph saved to training2_CaloHitListW_graph.data
Size of file training2_CaloHitListW_graph.data is 5732 bytes.
The eid is -1
Graph saved to training2_CaloHitListU_graph.data
Size of file training2_CaloHitListU_graph.data is 9028 bytes.
The eid is -1
Graph saved to training2_CaloHitListV_graph.data
Size of file training2_CaloHitListV_graph.data is 9732 bytes.
Boundary wire vector sizes: 367, 394, 234
minwire 0: 609
minwire 1: 889
minwire 2: 632
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
01-Aug-2025 16:02:49 CEST Opened output file with pattern "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_1167_20231204T100543Z_gen_g4_detsim_hitreco__20240509T212618Z_reco2_graph_2025-08-01T_140152Z.root"
01-Aug-2025 16:02:56 CEST Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/45/b5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_1167_20231204T100543Z_gen_g4_detsim_hitreco__20240509T212618Z_reco2.root"
Malformed TimeTracker database. The TimeEvent table is empty, but
the TimeModule table is not. This can happen if an exception has
been thrown from a module while processing the first event. Any
saved database file is suspect and should not be used.
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 8582.41 MB
Peak resident set size usage (VmHWM): 1666.49 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 01-Aug-2025 16:02:57 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module CVNEvaluator/cvneva run: 74517726 subRun: 1 event: 116701
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path reco
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 0 entries while recob::PCAxisrecob::Showervoidart::Assns_pandoraShower__Reco2. has 100 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
=== Generated output files ===
20206.9_dunegpschedd01.fnal.gov.logs.tgz
RootOutput-5ebe-7c41-2eb2-e2f8.root
ana_tree_hd.root
analysiseid.root
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_1167_20231204T100543Z_gen_g4_detsim_hitreco__20240509T212618Z_reco2_graph_2025-08-01T_140152Z.log
debugprod.log
jobscript.log
reco_hist.root
secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_1167_20231204T100543Z_gen_g4_detsim_hitreco__20240509T212618Z_reco2_graph_2025-08-01T_140152Z.log
third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_1167_20231204T100543Z_gen_g4_detsim_hitreco__20240509T212618Z_reco2_graph_2025-08-01T_140152Z.log
training1_CaloHitListU.csv
training1_CaloHitListU_graph.data
training1_CaloHitListV.csv
training1_CaloHitListV_graph.data
training1_CaloHitListW.csv
training1_CaloHitListW_graph.data
training2_CaloHitListU.csv
training2_CaloHitListU_graph.data
training2_CaloHitListV.csv
training2_CaloHitListV_graph.data
training2_CaloHitListW.csv
training2_CaloHitListW_graph.data 