Jobsub ID 20663.0@dunegpschedd01.fnal.gov

Jobsub ID	20663.0@dunegpschedd01.fnal.gov
Workflow ID	270
Stage ID	1
User name	ichong@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-03 16:38:45
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_brug
Last heartbeat	2025-08-03 16:49:21
From worker node	Hostname	wn-pep-010.farm.nikhef.nl
	cpuinfo	Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-08-03 16:39:17
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_22_20231202T195714Z_gen_g4_detsim_hitreco__20240508T061127Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-08-03 16:49:21
Saved logs	justin-logs:20663.0-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

n_g4_detsim_hitreco__20240508T061127Z_reco2.root"

========================================================================================================================
TimeTracker printout (sec)                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================
Full event                             0.444367      0.813672       1.70645      0.824964      0.165172        100    
------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                 0.0144583     0.0216422     0.029663      0.0157326    0.00710992       100    
end_path:analysistree:AnalysisTree     0.429662      0.791919       1.67739      0.799362      0.163989        100    
========================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7389.12 MB
  Peak resident set size usage (VmHWM): 1036.88 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of third lar log file ===
=== Start last 100 lines of lar log file ===
2 1.43858e+08
Now with regularization...
Begin: 1.36445e+08
0 1.36343e+08
---MC-PARTICLE-MONITORING-----------------------------------------------------------------------

BeamNeutrinos: 

--Primary 0, MCPDG 211, Energy 1.1182, Dist. 45.6002, nMCHits 617 (201, 210, 206)
MCPDG 211, Energy 1.1182, Dist. 45.6002, nMCHits 215 (80, 64, 71)
\_ MCPDG 2212, Energy 1.13704, Dist. 25.7603, nMCHits 48 (24, 19, 5)
   \_ MCPDG 22, Energy 0.210359, Dist. 0.869694, nMCHits 354 (97, 127, 130)
------------------------------------------------------------------------------------------------
Operating in training mode.
The eid is 2
Graph saved to training1_CaloHitListW_graph.data
Size of file training1_CaloHitListW_graph.data is 7996 bytes.
The eid is 2
Graph saved to training1_CaloHitListU_graph.data
Size of file training1_CaloHitListU_graph.data is 8388 bytes.
The eid is 2
Graph saved to training1_CaloHitListV_graph.data
Size of file training1_CaloHitListV_graph.data is 8252 bytes.
Operating in inference mode.
Operating in training mode.
The eid is -1
Graph saved to training2_CaloHitListW_graph.data
Size of file training2_CaloHitListW_graph.data is 7996 bytes.
The eid is -1
Graph saved to training2_CaloHitListU_graph.data
Size of file training2_CaloHitListU_graph.data is 8388 bytes.
The eid is -1
Graph saved to training2_CaloHitListV_graph.data
Size of file training2_CaloHitListV_graph.data is 8252 bytes.
Boundary wire vector sizes: 344, 339, 328
minwire 0: 891
minwire 1: 1276
minwire 2: 949
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2317, 2816
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
03-Aug-2025 18:46:28 CEST  Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/cd/f7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_22_20231202T195714Z_gen_g4_detsim_hitreco__20240508T061127Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0289367      1.24717       2.05419       1.65838      0.876444         3     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0148044     0.0195296     0.0289367     0.0148476    0.00665185        3     
reco:gaushit:GausHitFinder                             0.0556045     0.0633257     0.0762087     0.0581639    0.00916937        3     
reco:spsolve:SpacePointSolver                         9.9837e-05     0.0101796     0.0256576    0.00478136     0.0111102        3     
reco:hitfd:DisambigFromSpacePoints                    0.000451528   0.00189443     0.003045     0.00218678    0.00107877        3     
reco:rns:RandomNumberSaver                            1.7243e-05    7.8783e-05    0.00019923    1.9876e-05    8.51757e-05       3     
reco:pandora:StandardPandora                            1.22803       1.40673       1.65338       1.33877      0.180175         3     
reco:pandoraTrack:LArPandoraTrackCreation             0.00011736    0.000649589   0.00165804    0.000173362   0.000713452       3     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000153207   0.000793805   0.00205056    0.000177645   0.000888718       3     
reco:pandoracalo:Calorimetry                          9.7254e-05    0.000423115   0.00104597    0.000126125   0.000440579       3     
reco:pandorapid:Chi2ParticleID                        3.4995e-05    0.000210765   0.000557641   3.9659e-05    0.000245286       3     
reco:cvnmap:CVNMapper                                 5.5249e-05     0.0168465     0.0503335    0.000150922    0.0236789        3     
reco:cvneva:CVNEvaluator                               1.643e-05    6.47235e-05   0.000113017   6.47235e-05   4.82935e-05       2     
reco:energyrecnumu:EnergyReco                          0.0018265    0.00274648    0.00366646    0.00274648    0.00091998        2     
reco:energyrecnue:EnergyReco                          0.00189056    0.00195418     0.0020178    0.00195418    6.36195e-05       2     
reco:energyrecnc:EnergyReco                           0.00178288    0.00183027    0.00187766    0.00183027    4.7387e-05        2     
reco:energyrecnumurange:EnergyReco                     0.0017724    0.00178002    0.00178763    0.00178002     7.616e-06        2     
reco:energyrecnumumcs:EnergyReco                      0.00179134     0.0018024    0.00181346     0.0018024    1.1059e-05        2     
reco:opdec:Deconvolution                               0.0930389     0.113721      0.134403      0.113721      0.0206818        2     
reco:ophitspe:OpHitFinderDeco                          0.129592      0.132001       0.13441      0.132001     0.00240909        2     
reco:opflash:OpFlashFinder                            0.000113894   0.00117716    0.00224042    0.00117716    0.00106326        2     
reco:opslicer:OpSlicer                                 3.259e-05    0.000652791   0.00127299    0.000652791    0.0006202        2     
[art]:TriggerResults:TriggerResultInserter            1.1042e-05    2.51545e-05   3.9267e-05    2.51545e-05   1.41125e-05       2     
end_path:out1:RootOutput                               2.486e-06    8.7485e-06    1.5011e-05    8.7485e-06    6.2625e-06        2     
end_path:out1:RootOutput(write)                        0.0155413     0.0230994     0.0306575     0.0230994    0.00755811        2     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8582.46 MB
  Peak resident set size usage (VmHWM): 1680.78 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 03-Aug-2025 18:46:28 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module CVNEvaluator/cvneva run: 6413290 subRun: 1 event: 2203
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 2 entries while recob::PCAxisrecob::Showervoidart::Assns_pandoraShower__Reco2. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
=== Generated output files ===
20663.0_dunegpschedd01.fnal.gov.logs.tgz
RootOutput-a4dd-4025-ed04-fc11.root
ana_tree_hd.root
analysiseid.root
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_22_20231202T195714Z_gen_g4_detsim_hitreco__20240508T061127Z_reco2_graph_2025-08-03T_163921Z.log
debugprod.log
jobscript.log
reco_hist.root
secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_22_20231202T195714Z_gen_g4_detsim_hitreco__20240508T061127Z_reco2_graph_2025-08-03T_163921Z.log
third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_22_20231202T195714Z_gen_g4_detsim_hitreco__20240508T061127Z_reco2_graph_2025-08-03T_163921Z.log
training1_CaloHitListU.csv
training1_CaloHitListU_graph.data
training1_CaloHitListV.csv
training1_CaloHitListV_graph.data
training1_CaloHitListW.csv
training1_CaloHitListW_graph.data
training2_CaloHitListU.csv
training2_CaloHitListU_graph.data
training2_CaloHitListV.csv
training2_CaloHitListV_graph.data
training2_CaloHitListW.csv
training2_CaloHitListW_graph.data