Jobsub ID 20675.12@dunegpschedd01.fnal.gov

Jobsub ID	20675.12@dunegpschedd01.fnal.gov
Workflow ID	270
Stage ID	1
User name	ichong@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-03 17:28:47
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_klomp
Last heartbeat	2025-08-03 17:39:43
From worker node	Hostname	wn-pep-013.farm.nikhef.nl
	cpuinfo	Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-08-03 17:29:34
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_458_20231204T120417Z_gen_g4_detsim_hitreco__20240509T220858Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-08-03 17:39:43
Saved logs	justin-logs:20675.12-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

20417Z_gen_g4_detsim_hitreco__20240509T220858Z_reco2.root"

========================================================================================================================
TimeTracker printout (sec)                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================
Full event                             0.457182      0.828608       1.71357      0.823592      0.173625        100    
------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                 0.0143529     0.021643      0.0372507     0.0154056    0.00721491       100    
end_path:analysistree:AnalysisTree     0.428366      0.806861       1.6989       0.800551      0.174853        100    
========================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7453.04 MB
  Peak resident set size usage (VmHWM): 1097.79 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of third lar log file ===
=== Start last 100 lines of lar log file ===
1 -1.42682e+06
2 -1.42883e+06
3 -1.43038e+06
4 -1.43157e+06
---MC-PARTICLE-MONITORING-----------------------------------------------------------------------

BeamNeutrinos: 

--Primary 0, MCPDG 13, Energy 0.345818, Dist. 94.928, nMCHits 499 (152, 166, 181)
MCPDG 13, Energy 0.345818, Dist. 94.928, nMCHits 481 (147, 155, 179)
\_ MCPDG 11, Energy 0.0184747, Dist. 5.35611, nMCHits 17 (4, 11, 2)
\_ MCPDG 11, Energy 0.000515634, Dist. 2.15792e-05, nMCHits 1 (1, 0, 0)
------------------------------------------------------------------------------------------------
Operating in training mode.
The eid is 1
Graph saved to training1_CaloHitListW_graph.data
Size of file training1_CaloHitListW_graph.data is 4916 bytes.
The eid is 1
Graph saved to training1_CaloHitListU_graph.data
Size of file training1_CaloHitListU_graph.data is 4356 bytes.
The eid is 1
Graph saved to training1_CaloHitListV_graph.data
Size of file training1_CaloHitListV_graph.data is 4564 bytes.
Operating in inference mode.
Operating in training mode.
The eid is -1
Graph saved to training2_CaloHitListW_graph.data
Size of file training2_CaloHitListW_graph.data is 4916 bytes.
The eid is -1
Graph saved to training2_CaloHitListU_graph.data
Size of file training2_CaloHitListU_graph.data is 4356 bytes.
The eid is -1
Graph saved to training2_CaloHitListV_graph.data
Size of file training2_CaloHitListV_graph.data is 4564 bytes.
Boundary wire vector sizes: 180, 189, 204
minwire 0: 1118
minwire 1: 1881
minwire 2: 803
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
03-Aug-2025 19:36:48 CEST  Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/19/89/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_458_20231204T120417Z_gen_g4_detsim_hitreco__20240509T220858Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0149231      1.08597       2.15702       1.08597       1.07105         2     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0149231     0.0150142     0.0151053     0.0150142    9.1107e-05        2     
reco:gaushit:GausHitFinder                             0.0583232     0.0760175     0.0937118     0.0760175     0.0176943        2     
reco:spsolve:SpacePointSolver                         0.00557024    0.00567776    0.00578527    0.00567776    0.000107515       2     
reco:hitfd:DisambigFromSpacePoints                    0.00128526    0.00190011    0.00251495    0.00190011    0.000614843       2     
reco:rns:RandomNumberSaver                            2.5864e-05    0.00012943    0.000232995   0.00012943    0.000103565       2     
reco:pandora:StandardPandora                            1.32523       1.48872       1.65221       1.48872       0.16349         2     
reco:pandoraTrack:LArPandoraTrackCreation             0.000131274   0.00100923    0.00188719    0.00100923    0.000877957       2     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000174035   0.00125887    0.00234371    0.00125887    0.00108484        2     
reco:pandoracalo:Calorimetry                          0.000109378   0.000609057   0.00110874    0.000609057   0.000499678       2     
reco:pandorapid:Chi2ParticleID                        4.7876e-05     0.0003135    0.000579124    0.0003135    0.000265624       2     
reco:cvnmap:CVNMapper                                 0.00015057     0.0262793     0.0524081     0.0262793     0.0261288        2     
reco:cvneva:CVNEvaluator                              0.000112029   0.000112029   0.000112029   0.000112029        0            1     
reco:energyrecnumu:EnergyReco                         0.00359582    0.00359582    0.00359582    0.00359582         0            1     
reco:energyrecnue:EnergyReco                          0.00185816    0.00185816    0.00185816    0.00185816         0            1     
reco:energyrecnc:EnergyReco                           0.00186672    0.00186672    0.00186672    0.00186672         0            1     
reco:energyrecnumurange:EnergyReco                    0.00178346    0.00178346    0.00178346    0.00178346         0            1     
reco:energyrecnumumcs:EnergyReco                      0.00174244    0.00174244    0.00174244    0.00174244         0            1     
reco:opdec:Deconvolution                                0.20738       0.20738       0.20738       0.20738          0            1     
reco:ophitspe:OpHitFinderDeco                          0.135317      0.135317      0.135317      0.135317          0            1     
reco:opflash:OpFlashFinder                            0.00276779    0.00276779    0.00276779    0.00276779         0            1     
reco:opslicer:OpSlicer                                0.00292186    0.00292186    0.00292186    0.00292186         0            1     
[art]:TriggerResults:TriggerResultInserter            4.3359e-05    4.3359e-05    4.3359e-05    4.3359e-05         0            1     
end_path:out1:RootOutput                              1.4082e-05    1.4082e-05    1.4082e-05    1.4082e-05         0            1     
end_path:out1:RootOutput(write)                        0.0563919     0.0563919     0.0563919     0.0563919         0            1     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8581.45 MB
  Peak resident set size usage (VmHWM): 1676.71 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 03-Aug-2025 19:36:48 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module CVNEvaluator/cvneva run: 6422149 subRun: 1 event: 45802
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while recob::PCAxisrecob::Showervoidart::Assns_pandoraShower__Reco2. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
=== Generated output files ===
20675.12_dunegpschedd01.fnal.gov.logs.tgz
RootOutput-921d-0018-bba7-110d.root
ana_tree_hd.root
analysiseid.root
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_458_20231204T120417Z_gen_g4_detsim_hitreco__20240509T220858Z_reco2_graph_2025-08-03T_172938Z.log
debugprod.log
jobscript.log
reco_hist.root
secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_458_20231204T120417Z_gen_g4_detsim_hitreco__20240509T220858Z_reco2_graph_2025-08-03T_172938Z.log
third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6422149_458_20231204T120417Z_gen_g4_detsim_hitreco__20240509T220858Z_reco2_graph_2025-08-03T_172938Z.log
training1_CaloHitListU.csv
training1_CaloHitListU_graph.data
training1_CaloHitListV.csv
training1_CaloHitListV_graph.data
training1_CaloHitListW.csv
training1_CaloHitListW_graph.data
training2_CaloHitListU.csv
training2_CaloHitListU_graph.data
training2_CaloHitListV.csv
training2_CaloHitListV_graph.data
training2_CaloHitListW.csv
training2_CaloHitListW_graph.data