Jobsub ID 20206.7@dunegpschedd01.fnal.gov
Jobscript log (last 10,000 characters)
like PFParticle with ID 6
Begin processing the 86th record. run: 50557588 subRun: 1 event: 51186 at 01-Aug-2025 16:11:43 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 10
Begin processing the 87th record. run: 50557588 subRun: 1 event: 51187 at 01-Aug-2025 16:11:44 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 5
Begin processing the 88th record. run: 50557588 subRun: 1 event: 51188 at 01-Aug-2025 16:11:45 CEST
Analysing.
Begin processing the 89th record. run: 50557588 subRun: 1 event: 51189 at 01-Aug-2025 16:11:46 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 1
Begin processing the 90th record. run: 50557588 subRun: 1 event: 51190 at 01-Aug-2025 16:11:47 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 5
Begin processing the 91st record. run: 50557588 subRun: 1 event: 51191 at 01-Aug-2025 16:11:48 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 1
Begin processing the 92nd record. run: 50557588 subRun: 1 event: 51192 at 01-Aug-2025 16:11:49 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 4
Begin processing the 93rd record. run: 50557588 subRun: 1 event: 51193 at 01-Aug-2025 16:11:50 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 1
Begin processing the 94th record. run: 50557588 subRun: 1 event: 51194 at 01-Aug-2025 16:11:51 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 3
Begin processing the 95th record. run: 50557588 subRun: 1 event: 51195 at 01-Aug-2025 16:11:52 CEST
Analysing.
Begin processing the 96th record. run: 50557588 subRun: 1 event: 51196 at 01-Aug-2025 16:11:53 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 1
Begin processing the 97th record. run: 50557588 subRun: 1 event: 51197 at 01-Aug-2025 16:11:54 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 4
Begin processing the 98th record. run: 50557588 subRun: 1 event: 51198 at 01-Aug-2025 16:11:55 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 4
Begin processing the 99th record. run: 50557588 subRun: 1 event: 51199 at 01-Aug-2025 16:11:56 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 2
Begin processing the 100th record. run: 50557588 subRun: 1 event: 51200 at 01-Aug-2025 16:11:57 CEST
Analysing.
Warning: there was no track found for track-like PFParticle with ID 2
01-Aug-2025 16:11:58 CEST Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/94/cc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_511_20231202T102837Z_gen_g4_detsim_hitreco__20240507T200237Z_reco2.root"
========================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================
Full event 0.447564 0.823442 2.00535 0.819487 0.186941 100
------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.0144089 0.0216736 0.0291441 0.0178867 0.0070662 100
end_path:analysistree:AnalysisTree 0.427517 0.80164 1.99058 0.79649 0.187782 100
========================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 7598.47 MB
Peak resident set size usage (VmHWM): 1245.88 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of third lar log file ===
=== Start last 100 lines of lar log file ===
16 464586
17 463500
18 462513
19 461623
20 461098
21 460669
Now with regularization...
Begin: -1.65142e+06
0 -1.67306e+06
1 -1.68966e+06
2 -1.70396e+06
3 -1.71767e+06
4 -1.72838e+06
5 -1.73778e+06
6 -1.74511e+06
7 -1.75186e+06
8 -1.75782e+06
9 -1.76286e+06
10 -1.7672e+06
11 -1.77116e+06
12 -1.77434e+06
13 -1.77721e+06
14 -1.77967e+06
15 -1.78206e+06
16 -1.7844e+06
17 -1.78653e+06
18 -1.7884e+06
19 -1.78982e+06
---MC-PARTICLE-MONITORING-----------------------------------------------------------------------
BeamNeutrinos:
--Primary 0, MCPDG -13, Energy 0.306471, Dist. 76.4052, nMCHits 449 (169, 121, 159)
MCPDG -13, Energy 0.306471, Dist. 76.4052, nMCHits 403 (145, 108, 150)
\_ MCPDG -11, Energy 0.0516167, Dist. 13.076, nMCHits 46 (24, 13, 9)
------------------------------------------------------------------------------------------------
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Atmos_1_U_v04_03_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Atmos_1_V_v04_03_00.pt'
Loaded the TorchScript model '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash//PandoraNetworkData/PandoraNet_Vertex_DUNEFD_HD_Atmos_1_W_v04_03_00.pt'
Operating in training mode.
The eid is 0
Graph saved to training1_CaloHitListW_graph.data
Size of file training1_CaloHitListW_graph.data is 4340 bytes.
The eid is 0
Graph saved to training1_CaloHitListU_graph.data
Size of file training1_CaloHitListU_graph.data is 4764 bytes.
The eid is 0
Graph saved to training1_CaloHitListV_graph.data
Size of file training1_CaloHitListV_graph.data is 3580 bytes.
Operating in inference mode.
Operating in training mode.
The eid is -1
Graph saved to training2_CaloHitListW_graph.data
Size of file training2_CaloHitListW_graph.data is 4340 bytes.
The eid is -1
Graph saved to training2_CaloHitListU_graph.data
Size of file training2_CaloHitListU_graph.data is 4764 bytes.
The eid is -1
Graph saved to training2_CaloHitListV_graph.data
Size of file training2_CaloHitListV_graph.data is 3580 bytes.
Boundary wire vector sizes: 148, 197, 180
minwire 0: 2253
minwire 1: 248
minwire 2: 2352
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
01-Aug-2025 16:02:54 CEST Opened output file with pattern "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_511_20231202T102837Z_gen_g4_detsim_hitreco__20240507T200237Z_reco2_graph_2025-08-01T_140155Z.root"
01-Aug-2025 16:09:18 CEST Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/94/cc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_511_20231202T102837Z_gen_g4_detsim_hitreco__20240507T200237Z_reco2.root"
Malformed TimeTracker database. The TimeEvent table is empty, but
the TimeModule table is not. This can happen if an exception has
been thrown from a module while processing the first event. Any
saved database file is suspect and should not be used.
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 8589.01 MB
Peak resident set size usage (VmHWM): 1679.02 MB
====================================================================================================
%MSG-s ArtException: PostEndJob 01-Aug-2025 16:09:18 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- BadAlloc BEGIN
A bad_alloc exception was thrown while processing module CVNEvaluator/cvneva run: 50557588 subRun: 1 event: 51101
The job has probably exhausted the virtual memory available to the process.
---- BadAlloc END
Exception going through path reco
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 0 entries while recob::PCAxisrecob::Showervoidart::Assns_pandoraShower__Reco2. has 100 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
=== Generated output files ===
20206.7_dunegpschedd01.fnal.gov.logs.tgz
RootOutput-8140-dcc4-6676-b592.root
ana_tree_hd.root
analysiseid.root
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_511_20231202T102837Z_gen_g4_detsim_hitreco__20240507T200237Z_reco2_graph_2025-08-01T_140155Z.log
debugprod.log
jobscript.log
reco_hist.root
secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_511_20231202T102837Z_gen_g4_detsim_hitreco__20240507T200237Z_reco2_graph_2025-08-01T_140155Z.log
third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_511_20231202T102837Z_gen_g4_detsim_hitreco__20240507T200237Z_reco2_graph_2025-08-01T_140155Z.log
training1_CaloHitListU.csv
training1_CaloHitListU_graph.data
training1_CaloHitListV.csv
training1_CaloHitListV_graph.data
training1_CaloHitListW.csv
training1_CaloHitListW_graph.data
training2_CaloHitListU.csv
training2_CaloHitListU_graph.data
training2_CaloHitListV.csv
training2_CaloHitListV_graph.data
training2_CaloHitListW.csv
training2_CaloHitListW_graph.data 