Jobsub ID 14154.3@dunegpschedd02.fnal.gov

Jobsub ID	14154.3@dunegpschedd02.fnal.gov
Workflow ID	270
Stage ID	1
User name	ichong@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-03 15:34:41
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_brug
Last heartbeat	2025-08-03 15:36:50
From worker node	Hostname	wn-pep-010.farm.nikhef.nl
	cpuinfo	Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-08-03 15:34:56
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_718_20231203T110354Z_gen_g4_detsim_hitreco__20240509T200627Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-08-03 15:36:50
Saved logs	justin-logs:14154.3-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

20240509T200627Z_reco2.root"

========================================================================================================================
TimeTracker printout (sec)                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================
Full event                            0.00847713     0.0569828      1.38774      0.0195652     0.165525        100    
------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                0.000414175   0.000715726   0.00432208    0.000636187   0.000409052      100    
end_path:analysistree:AnalysisTree    0.00791099     0.0561666      1.38696      0.0188462     0.165532        100    
========================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7496.03 MB
  Peak resident set size usage (VmHWM): 1144.09 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of third lar log file ===
=== Start last 100 lines of lar log file ===
5 5.19216e+06
6 5.18686e+06
7 5.18246e+06
---MC-PARTICLE-MONITORING-----------------------------------------------------------------------

BeamNeutrinos: 

--Primary 0, MCPDG 11, Energy 1.62435, Dist. 20.3178, nMCHits 1623 (584, 481, 558)
MCPDG 11, Energy 1.62435, Dist. 20.3178, nMCHits 1623 (584, 481, 558)

--Primary 1, MCPDG 2212, Energy 1.24699, Dist. 54.1132, nMCHits 67 (42, 18, 7)
MCPDG 2212, Energy 1.24699, Dist. 54.1132, nMCHits 67 (42, 18, 7)
------------------------------------------------------------------------------------------------
Operating in training mode.
The eid is 2
Graph saved to training1_CaloHitListW_graph.data
Size of file training1_CaloHitListW_graph.data is 14584 bytes.
The eid is 2
Graph saved to training1_CaloHitListU_graph.data
Size of file training1_CaloHitListU_graph.data is 16424 bytes.
The eid is 2
Graph saved to training1_CaloHitListV_graph.data
Size of file training1_CaloHitListV_graph.data is 12760 bytes.
Operating in inference mode.
Operating in training mode.
The eid is -1
Graph saved to training2_CaloHitListW_graph.data
Size of file training2_CaloHitListW_graph.data is 14584 bytes.
The eid is -1
Graph saved to training2_CaloHitListU_graph.data
Size of file training2_CaloHitListU_graph.data is 16424 bytes.
The eid is -1
Graph saved to training2_CaloHitListV_graph.data
Size of file training2_CaloHitListV_graph.data is 12760 bytes.
Boundary wire vector sizes: 633, 506, 570
minwire 0: 2430
minwire 1: 210
minwire 2: 2585
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
03-Aug-2025 17:35:50 CEST  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/95/66/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_718_20231203T110354Z_gen_g4_detsim_hitreco__20240509T200627Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                            0.00111092      1.09274       1.82479       1.45231      0.786734         3     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000687073   0.00108287    0.00145061    0.00111092    0.000312343       3     
reco:gaushit:GausHitFinder                             0.0140723     0.0286458     0.0564091     0.015456      0.0196397        3     
reco:spsolve:SpacePointSolver                         0.000985174    0.0126941     0.0269056     0.0101915     0.0107289        3     
reco:hitfd:DisambigFromSpacePoints                    0.00115474    0.00303735    0.00549693    0.00246039    0.00181903        3     
reco:rns:RandomNumberSaver                            1.6404e-05    8.48013e-05   0.000218356   1.9644e-05    9.44467e-05       3     
reco:pandora:StandardPandora                            1.17328       1.38177       1.71013       1.26192      0.234984         3     
reco:pandoraTrack:LArPandoraTrackCreation             0.000117432   0.000618206   0.00159727    0.000139913   0.000692365       3     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000156678   0.000883896    0.0023367    0.000158315   0.00102728        3     
reco:pandoracalo:Calorimetry                          9.9014e-05    0.000413598   0.00103612    0.000105657    0.0004402        3     
reco:pandorapid:Chi2ParticleID                        3.2864e-05    0.000197922   0.00052655    3.4352e-05    0.000232376       3     
reco:cvnmap:CVNMapper                                 7.2137e-05     0.0201183     0.0601392    0.000143684    0.028299         3     
reco:cvneva:CVNEvaluator                              1.5333e-05    6.0792e-05    0.000106251   6.0792e-05    4.5459e-05        2     
reco:energyrecnumu:EnergyReco                         0.00184759    0.00262394    0.00340029    0.00262394    0.00077635        2     
reco:energyrecnue:EnergyReco                          0.00187478    0.00188177    0.00188877    0.00188177    6.9935e-06        2     
reco:energyrecnc:EnergyReco                           0.00179026     0.0018051    0.00181994     0.0018051    1.48395e-05       2     
reco:energyrecnumurange:EnergyReco                     0.0017802    0.00178338    0.00178656    0.00178338     3.179e-06        2     
reco:energyrecnumumcs:EnergyReco                      0.00173931     0.0017553    0.00177128     0.0017553    1.5984e-05        2     
reco:opdec:Deconvolution                               0.0436067     0.0707328     0.0978589     0.0707328     0.0271261        2     
reco:ophitspe:OpHitFinderDeco                         0.00488993    0.00640149    0.00791304    0.00640149    0.00151156        2     
reco:opflash:OpFlashFinder                            0.000198688   0.00121977    0.00224086    0.00121977    0.00102108        2     
reco:opslicer:OpSlicer                                0.00128882    0.00473396    0.00817911    0.00473396    0.00344514        2     
[art]:TriggerResults:TriggerResultInserter            1.0804e-05    2.59365e-05   4.1069e-05    2.59365e-05   1.51325e-05       2     
end_path:out1:RootOutput                               2.952e-06    8.5335e-06    1.4115e-05    8.5335e-06    5.5815e-06        2     
end_path:out1:RootOutput(write)                        0.0216835     0.0322927     0.042902      0.0322927     0.0106093        2     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8582.97 MB
  Peak resident set size usage (VmHWM): 1679.61 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 03-Aug-2025 17:35:51 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module CVNEvaluator/cvneva run: 50574782 subRun: 1 event: 71803
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 2 entries while recob::PCAxisrecob::Showervoidart::Assns_pandoraShower__Reco2. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
=== Generated output files ===
14154.3_dunegpschedd02.fnal.gov.logs.tgz
RootOutput-2c8c-0e0a-6288-9a0d.root
ana_tree_hd.root
analysiseid.root
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_718_20231203T110354Z_gen_g4_detsim_hitreco__20240509T200627Z_reco2_graph_2025-08-03T_153501Z.log
debugprod.log
jobscript.log
reco_hist.root
secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_718_20231203T110354Z_gen_g4_detsim_hitreco__20240509T200627Z_reco2_graph_2025-08-03T_153501Z.log
third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_718_20231203T110354Z_gen_g4_detsim_hitreco__20240509T200627Z_reco2_graph_2025-08-03T_153501Z.log
training1_CaloHitListU.csv
training1_CaloHitListU_graph.data
training1_CaloHitListV.csv
training1_CaloHitListV_graph.data
training1_CaloHitListW.csv
training1_CaloHitListW_graph.data
training2_CaloHitListU.csv
training2_CaloHitListU_graph.data
training2_CaloHitListV.csv
training2_CaloHitListV_graph.data
training2_CaloHitListW.csv
training2_CaloHitListW_graph.data