Jobsub ID 14155.0@dunegpschedd02.fnal.gov

Jobsub ID	14155.0@dunegpschedd02.fnal.gov
Workflow ID	270
Stage ID	1
User name	ichong@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-03 15:36:41
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_brug
Last heartbeat	2025-08-03 15:46:14
From worker node	Hostname	wn-pep-010.farm.nikhef.nl
	cpuinfo	Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-08-03 15:38:10
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6405486_187_20231202T112025Z_gen_g4_detsim_hitreco__20240507T214622Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-08-03 15:46:14
Saved logs	justin-logs:14155.0-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6405486_187_20231202T112025Z_gen_g4_detsim_hitreco__20240507T214622Z_reco2.root"

========================================================================================================================
TimeTracker printout (sec)                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================
Full event                             0.385594      0.746514       1.19304      0.760668       0.15246        100    
------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                0.00872411     0.0646695     0.211404       0.06483      0.0371747       100    
end_path:analysistree:AnalysisTree     0.360174       0.68174       1.18419      0.686646      0.157886        100    
========================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7277.01 MB
  Peak resident set size usage (VmHWM): 922.931 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of third lar log file ===
=== Start last 100 lines of lar log file ===
9 30404.5
10 30358.3
11 30318.6
12 30284.7
13 30255.6
---MC-PARTICLE-MONITORING-----------------------------------------------------------------------

BeamNeutrinos: 

--Primary 0, MCPDG -13, Energy 0.196469, Dist. 25.9972, nMCHits 153 (65, 42, 46)
MCPDG -13, Energy 0.196469, Dist. 25.9972, nMCHits 83 (39, 21, 23)
\_ MCPDG -11, Energy 0.0317477, Dist. 9.92825, nMCHits 70 (26, 21, 23)
------------------------------------------------------------------------------------------------
Operating in training mode.
The eid is 1
Graph saved to training1_CaloHitListW_graph.data
Size of file training1_CaloHitListW_graph.data is 1724 bytes.
The eid is 1
Graph saved to training1_CaloHitListU_graph.data
Size of file training1_CaloHitListU_graph.data is 2204 bytes.
The eid is 1
Graph saved to training1_CaloHitListV_graph.data
Size of file training1_CaloHitListV_graph.data is 1700 bytes.
Operating in inference mode.
Operating in training mode.
The eid is -1
Graph saved to training2_CaloHitListW_graph.data
Size of file training2_CaloHitListW_graph.data is 1724 bytes.
The eid is -1
Graph saved to training2_CaloHitListU_graph.data
Size of file training2_CaloHitListU_graph.data is 2204 bytes.
The eid is -1
Graph saved to training2_CaloHitListV_graph.data
Size of file training2_CaloHitListV_graph.data is 1700 bytes.
Boundary wire vector sizes: 90, 69, 70
minwire 0: 841
minwire 1: 1578
minwire 2: 900
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
03-Aug-2025 17:42:49 CEST  Closed input file "root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/19/66/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6405486_187_20231202T112025Z_gen_g4_detsim_hitreco__20240507T214622Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0598328      1.31956       2.57929       1.31956       1.25973         2     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0464849     0.0531589     0.0598328     0.0531589    0.00667393        2     
reco:gaushit:GausHitFinder                             0.125858      0.138943      0.152028      0.138943      0.0130854        2     
reco:spsolve:SpacePointSolver                         0.000651587   0.00394106    0.00723052    0.00394106    0.00328947        2     
reco:hitfd:DisambigFromSpacePoints                    0.00117817    0.00169107    0.00220396    0.00169107    0.000512896       2     
reco:rns:RandomNumberSaver                            2.3509e-05    0.000137209   0.00025091    0.000137209   0.000113701       2     
reco:pandora:StandardPandora                            1.44369       1.65103       1.85837       1.65103      0.207341         2     
reco:pandoraTrack:LArPandoraTrackCreation             0.000163582   0.000883775   0.00160397    0.000883775   0.000720193       2     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000164041   0.00116876    0.00217348    0.00116876    0.00100472        2     
reco:pandoracalo:Calorimetry                          0.000105502   0.000571977   0.00103845    0.000571977   0.000466474       2     
reco:pandorapid:Chi2ParticleID                        3.7028e-05    0.00028066    0.000524292   0.00028066    0.000243632       2     
reco:cvnmap:CVNMapper                                 0.00014439     0.0248995     0.0496547     0.0248995     0.0247552        2     
reco:cvneva:CVNEvaluator                              0.000105832   0.000105832   0.000105832   0.000105832        0            1     
reco:energyrecnumu:EnergyReco                         0.00337407    0.00337407    0.00337407    0.00337407         0            1     
reco:energyrecnue:EnergyReco                           0.0018202     0.0018202     0.0018202     0.0018202         0            1     
reco:energyrecnc:EnergyReco                           0.00177568    0.00177568    0.00177568    0.00177568         0            1     
reco:energyrecnumurange:EnergyReco                    0.00172756    0.00172756    0.00172756    0.00172756         0            1     
reco:energyrecnumumcs:EnergyReco                      0.00177828    0.00177828    0.00177828    0.00177828         0            1     
reco:opdec:Deconvolution                               0.176286      0.176286      0.176286      0.176286          0            1     
reco:ophitspe:OpHitFinderDeco                          0.307752      0.307752      0.307752      0.307752          0            1     
reco:opflash:OpFlashFinder                            0.00104501    0.00104501    0.00104501    0.00104501         0            1     
reco:opslicer:OpSlicer                                0.00028124    0.00028124    0.00028124    0.00028124         0            1     
[art]:TriggerResults:TriggerResultInserter            4.0088e-05    4.0088e-05    4.0088e-05    4.0088e-05         0            1     
end_path:out1:RootOutput                              1.4468e-05    1.4468e-05    1.4468e-05    1.4468e-05         0            1     
end_path:out1:RootOutput(write)                        0.0168758     0.0168758     0.0168758     0.0168758         0            1     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8581.33 MB
  Peak resident set size usage (VmHWM): 1677.25 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 03-Aug-2025 17:42:49 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module CVNEvaluator/cvneva run: 6405486 subRun: 1 event: 18702
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while recob::PCAxisrecob::Showervoidart::Assns_pandoraShower__Reco2. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
=== Generated output files ===
14155.0_dunegpschedd02.fnal.gov.logs.tgz
RootOutput-3c4a-1bec-9827-23f7.root
ana_tree_hd.root
analysiseid.root
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6405486_187_20231202T112025Z_gen_g4_detsim_hitreco__20240507T214622Z_reco2_graph_2025-08-03T_153814Z.log
debugprod.log
jobscript.log
reco_hist.root
secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6405486_187_20231202T112025Z_gen_g4_detsim_hitreco__20240507T214622Z_reco2_graph_2025-08-03T_153814Z.log
third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6405486_187_20231202T112025Z_gen_g4_detsim_hitreco__20240507T214622Z_reco2_graph_2025-08-03T_153814Z.log
training1_CaloHitListU.csv
training1_CaloHitListU_graph.data
training1_CaloHitListV.csv
training1_CaloHitListV_graph.data
training1_CaloHitListW.csv
training1_CaloHitListW_graph.data
training2_CaloHitListU.csv
training2_CaloHitListU_graph.data
training2_CaloHitListV.csv
training2_CaloHitListV_graph.data
training2_CaloHitListW.csv
training2_CaloHitListW_graph.data