Jobsub ID 20669.11@dunegpschedd01.fnal.gov
Jobscript log (last 10,000 characters)
Analysing.
Warning: there was no track found for track-like PFParticle with ID 4
Begin processing the 93rd record. run: 74551443 subRun: 1 event: 28593 at 03-Aug-2025 19:15:41 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 7
Begin processing the 94th record. run: 74551443 subRun: 1 event: 28594 at 03-Aug-2025 19:15:42 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 2
Begin processing the 95th record. run: 74551443 subRun: 1 event: 28595 at 03-Aug-2025 19:15:43 CEST
Analysing.
Warning: there was no vertex found for PFParticle with ID 1
Warning: there was no track found for track-like PFParticle with ID 1
Warning: there was no vertex found for PFParticle with ID 2
Warning: there was no track found for track-like PFParticle with ID 2
Warning: there was no vertex found for PFParticle with ID 3
Warning: there was no track found for track-like PFParticle with ID 3
Warning: there was no vertex found for PFParticle with ID 4
Warning: there was no track found for track-like PFParticle with ID 4
Warning: there was no track found for track-like PFParticle with ID 5
Begin processing the 96th record. run: 74551443 subRun: 1 event: 28596 at 03-Aug-2025 19:15:44 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 3
Begin processing the 97th record. run: 74551443 subRun: 1 event: 28597 at 03-Aug-2025 19:15:45 CEST
Analysing.
Warning: there were 11 reconstructed PFParticle clusters; only the first 10 being stored in tree
Warning: there was no track found for track-like PFParticle with ID 3
Begin processing the 98th record. run: 74551443 subRun: 1 event: 28598 at 03-Aug-2025 19:15:46 CEST
Analysing.
Warning: there was no vertex found for PFParticle with ID 8
Warning: there was no track found for track-like PFParticle with ID 8
Warning: there was no track found for track-like PFParticle with ID 9
%MSG-e AnalysisTree:limits: AnalysisTree:analysistree@BeginModule 03-Aug-2025 19:15:48 CEST run: 74551443 subRun: 1 event: 28598
the pandoraTrack track #0 has 2684 hits on calorimetry plane #0, only 2000 stored in tree
%MSG
Begin processing the 99th record. run: 74551443 subRun: 1 event: 28599 at 03-Aug-2025 19:15:48 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 1
Begin processing the 100th record. run: 74551443 subRun: 1 event: 28600 at 03-Aug-2025 19:15:49 CEST
Analysing.
Warning: there was no vertex found for PFParticle with ID 6
Warning: there was no track found for track-like PFParticle with ID 6
Warning: there was no track found for track-like PFParticle with ID 7
03-Aug-2025 19:15:50 CEST Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/77/63/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_285_20231207T111413Z_gen_g4_detsim_hitreco__20240510T053333Z_reco2.root"
========================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================
Full event 0.452108 0.770077 1.34662 0.796635 0.158423 100
------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.0142066 0.0212319 0.0288203 0.0147797 0.00699678 100
end_path:analysistree:AnalysisTree 0.423898 0.748722 1.31823 0.778893 0.157982 100
========================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 7260.76 MB
Peak resident set size usage (VmHWM): 906.65 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of third lar log file ===
=== Start last 100 lines of lar log file ===
3 4.35756e+06
4 4.34184e+06
5 4.33213e+06
6 4.32544e+06
7 4.32082e+06
8 4.31723e+06
Now with regularization...
Begin: -167106
0 -205462
1 -220096
2 -228053
3 -233079
4 -236555
5 -238812
6 -240177
7 -241163
8 -241870
9 -242352
10 -242763
11 -243131
12 -243459
13 -243756
14 -244031
15 -244288
16 -244533
17 -244769
---MC-PARTICLE-MONITORING-----------------------------------------------------------------------
BeamNeutrinos:
--Primary 0, MCPDG 11, Energy 0.518295, Dist. 29.5285, nMCHits 711 (242, 216, 253)
MCPDG 11, Energy 0.518295, Dist. 29.5285, nMCHits 711 (242, 216, 253)
--Primary 1, MCPDG 2212, Energy 1.16581, Dist. 32.0445, nMCHits 39 (18, 17, 4)
MCPDG 2212, Energy 1.16581, Dist. 32.0445, nMCHits 39 (18, 17, 4)
------------------------------------------------------------------------------------------------
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Atmos_1_U_v04_03_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Atmos_1_V_v04_03_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Atmos_1_W_v04_03_00.pt'
Operating in training mode.
The eid is 0
Graph saved to training1_CaloHitListW_graph.data
Size of file training1_CaloHitListW_graph.data is 6324 bytes.
The eid is 0
Graph saved to training1_CaloHitListU_graph.data
Size of file training1_CaloHitListU_graph.data is 6524 bytes.
The eid is 0
Graph saved to training1_CaloHitListV_graph.data
Size of file training1_CaloHitListV_graph.data is 5772 bytes.
Operating in inference mode.
Operating in training mode.
The eid is -1
Graph saved to training2_CaloHitListW_graph.data
Size of file training2_CaloHitListW_graph.data is 6324 bytes.
The eid is -1
Graph saved to training2_CaloHitListU_graph.data
Size of file training2_CaloHitListU_graph.data is 6524 bytes.
The eid is -1
Graph saved to training2_CaloHitListV_graph.data
Size of file training2_CaloHitListV_graph.data is 5772 bytes.
Boundary wire vector sizes: 266, 234, 259
minwire 0: 2049
minwire 1: 1477
minwire 2: 1648
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
03-Aug-2025 19:07:16 CEST Opened output file with pattern "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_285_20231207T111413Z_gen_g4_detsim_hitreco__20240510T053333Z_reco2_graph_2025-08-03T_170632Z.root"
03-Aug-2025 19:13:28 CEST Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/77/63/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_285_20231207T111413Z_gen_g4_detsim_hitreco__20240510T053333Z_reco2.root"
Malformed TimeTracker database. The TimeEvent table is empty, but
the TimeModule table is not. This can happen if an exception has
been thrown from a module while processing the first event. Any
saved database file is suspect and should not be used.
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 8589.4 MB
Peak resident set size usage (VmHWM): 1686.57 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 03-Aug-2025 19:13:28 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module CVNEvaluator/cvneva run: 74551443 subRun: 1 event: 28501
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path reco
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 0 entries while recob::PCAxisrecob::Showervoidart::Assns_pandoraShower__Reco2. has 100 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
=== Generated output files ===
20669.11_dunegpschedd01.fnal.gov.logs.tgz
RootOutput-05a0-165f-446f-a693.root
ana_tree_hd.root
analysiseid.root
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_285_20231207T111413Z_gen_g4_detsim_hitreco__20240510T053333Z_reco2_graph_2025-08-03T_170632Z.log
debugprod.log
jobscript.log
reco_hist.root
secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_285_20231207T111413Z_gen_g4_detsim_hitreco__20240510T053333Z_reco2_graph_2025-08-03T_170632Z.log
third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_285_20231207T111413Z_gen_g4_detsim_hitreco__20240510T053333Z_reco2_graph_2025-08-03T_170632Z.log
training1_CaloHitListU.csv
training1_CaloHitListU_graph.data
training1_CaloHitListV.csv
training1_CaloHitListV_graph.data
training1_CaloHitListW.csv
training1_CaloHitListW_graph.data
training2_CaloHitListU.csv
training2_CaloHitListU_graph.data
training2_CaloHitListV.csv
training2_CaloHitListV_graph.data
training2_CaloHitListW.csv
training2_CaloHitListW_graph.data 