Jobsub ID 14068.8@dunegpschedd02.fnal.gov

Jobsub ID	14068.8@dunegpschedd02.fnal.gov
Workflow ID	253
Stage ID	1
User name	ichong@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-08-02 23:05:47
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_klomp
Last heartbeat	2025-08-02 23:17:12
From worker node	Hostname	wn-pep-004.farm.nikhef.nl
	cpuinfo	Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-08-02 23:07:10
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_470_20231205T100809Z_gen_g4_detsim_hitreco__20240510T035419Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-08-02 23:17:11
Saved logs	justin-logs:14068.8-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

=========================================================================================
Full event                             0.439668      0.798272       1.36947      0.806584      0.154622        100    
------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                 0.0143241     0.0213257     0.0288313     0.0148647    0.00700822       100    
end_path:analysistree:AnalysisTree     0.425119      0.776819       1.35496      0.786443      0.155375        100    
========================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7335.55 MB
  Peak resident set size usage (VmHWM): 983.429 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of third lar log file ===
=== Start last 100 lines of lar log file ===
--Primary 10, MCPDG -211, Energy 1.15071, Dist. 181.764, nMCHits 337 (205, 51, 81)
MCPDG -211, Energy 1.15071, Dist. 181.764, nMCHits 294 (183, 43, 68)
\_ MCPDG 2212, Energy 1.04781, Dist. 9.26548, nMCHits 25 (14, 5, 6)
\_ MCPDG 2212, Energy 0.996239, Dist. 3.00507, nMCHits 12 (5, 2, 5)
\_ MCPDG 2212, Energy 0.964765, Dist. 0.730624, nMCHits 4 (2, 1, 1)
\_ MCPDG 2212, Energy 0.960276, Dist. 0.533248, nMCHits 2 (1, 0, 1)

--Primary 11, MCPDG 211, Energy 0.617275, Dist. 42.8907, nMCHits 96 (56, 28, 12)
MCPDG 211, Energy 0.617275, Dist. 42.8907, nMCHits 42 (28, 9, 5)
\_ MCPDG 2212, Energy 1.27321, Dist. 6.36039, nMCHits 4 (4, 0, 0)
   \_ MCPDG 2212, Energy 1.07572, Dist. 13.4519, nMCHits 26 (14, 6, 6)
   \_ MCPDG 2212, Energy 1.07411, Dist. 13.5995, nMCHits 24 (10, 13, 1)
------------------------------------------------------------------------------------------------
Operating in training mode.
The eid is 1
Graph saved to training1_CaloHitListW_graph.data
Size of file training1_CaloHitListW_graph.data is 85664 bytes.
The eid is 1
Graph saved to training1_CaloHitListU_graph.data
Size of file training1_CaloHitListU_graph.data is 176712 bytes.
The eid is 1
Graph saved to training1_CaloHitListV_graph.data
Size of file training1_CaloHitListV_graph.data is 103880 bytes.
Operating in inference mode.
Operating in training mode.
The eid is -1
Graph saved to training2_CaloHitListW_graph.data
Size of file training2_CaloHitListW_graph.data is 85664 bytes.
The eid is -1
Graph saved to training2_CaloHitListU_graph.data
Size of file training2_CaloHitListU_graph.data is 176712 bytes.
The eid is -1
Graph saved to training2_CaloHitListV_graph.data
Size of file training2_CaloHitListV_graph.data is 103880 bytes.
Boundary wire vector sizes: 7209, 4217, 3489
minwire 0: 1549
minwire 1: 138
minwire 2: 1265
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2371, 2870
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
03-Aug-2025 01:14:19 CEST  Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/5a/47/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_470_20231205T100809Z_gen_g4_detsim_hitreco__20240510T035419Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0331176      1.14784       2.26256       1.14784       1.11472         2     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0154223     0.0242699     0.0331176     0.0242699    0.00884766        2     
reco:gaushit:GausHitFinder                             0.0588517     0.238464      0.418077      0.238464      0.179613         2     
reco:spsolve:SpacePointSolver                          0.0129525     0.521965       1.03098      0.521965      0.509013         2     
reco:hitfd:DisambigFromSpacePoints                    0.00293283     0.162661      0.322388      0.162661      0.159728         2     
reco:rns:RandomNumberSaver                            2.8846e-05    0.000140351   0.000251855   0.000140351   0.000111505       2     
reco:pandora:StandardPandora                            1.69752       3.34306       4.98859       3.34306       1.64553         2     
reco:pandoraTrack:LArPandoraTrackCreation             0.00151085    0.00185811    0.00220536    0.00185811    0.000347251       2     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00022455    0.00153935    0.00285416    0.00153935     0.0013148        2     
reco:pandoracalo:Calorimetry                          0.000147088   0.000823006   0.00149892    0.000823006   0.000675918       2     
reco:pandorapid:Chi2ParticleID                        4.2045e-05    0.000388751   0.000735457   0.000388751   0.000346706       2     
reco:cvnmap:CVNMapper                                 0.000189921    0.0467413     0.0932928     0.0467413     0.0465514        2     
reco:cvneva:CVNEvaluator                              0.000141647   0.000141647   0.000141647   0.000141647        0            1     
reco:energyrecnumu:EnergyReco                         0.00415833    0.00415833    0.00415833    0.00415833         0            1     
reco:energyrecnue:EnergyReco                          0.00193642    0.00193642    0.00193642    0.00193642         0            1     
reco:energyrecnc:EnergyReco                           0.00184249    0.00184249    0.00184249    0.00184249         0            1     
reco:energyrecnumurange:EnergyReco                    0.00177663    0.00177663    0.00177663    0.00177663         0            1     
reco:energyrecnumumcs:EnergyReco                      0.00177067    0.00177067    0.00177067    0.00177067         0            1     
reco:opdec:Deconvolution                               0.246678      0.246678      0.246678      0.246678          0            1     
reco:ophitspe:OpHitFinderDeco                          0.136529      0.136529      0.136529      0.136529          0            1     
reco:opflash:OpFlashFinder                            0.00302366    0.00302366    0.00302366    0.00302366         0            1     
reco:opslicer:OpSlicer                                 0.0029372     0.0029372     0.0029372     0.0029372         0            1     
[art]:TriggerResults:TriggerResultInserter            4.9682e-05    4.9682e-05    4.9682e-05    4.9682e-05         0            1     
end_path:out1:RootOutput                              2.9661e-05    2.9661e-05    2.9661e-05    2.9661e-05         0            1     
end_path:out1:RootOutput(write)                        0.0649917     0.0649917     0.0649917     0.0649917         0            1     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8589.81 MB
  Peak resident set size usage (VmHWM): 1681.87 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 03-Aug-2025 01:14:20 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module CVNEvaluator/cvneva run: 50602933 subRun: 1 event: 47002
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while recob::PCAxisrecob::Showervoidart::Assns_pandoraShower__Reco2. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
=== Generated output files ===
14068.8_dunegpschedd02.fnal.gov.logs.tgz
RootOutput-f14d-1ff1-17f3-9a34.root
ana_tree_hd.root
analysiseid.root
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_470_20231205T100809Z_gen_g4_detsim_hitreco__20240510T035419Z_reco2_graph_2025-08-02T_230714Z.log
debugprod.log
jobscript.log
reco_hist.root
secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_470_20231205T100809Z_gen_g4_detsim_hitreco__20240510T035419Z_reco2_graph_2025-08-02T_230714Z.log
third_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_470_20231205T100809Z_gen_g4_detsim_hitreco__20240510T035419Z_reco2_graph_2025-08-02T_230714Z.log
training1_CaloHitListU.csv
training1_CaloHitListU_graph.data
training1_CaloHitListV.csv
training1_CaloHitListV_graph.data
training1_CaloHitListW.csv
training1_CaloHitListW_graph.data
training2_CaloHitListU.csv
training2_CaloHitListU_graph.data
training2_CaloHitListV.csv
training2_CaloHitListV_graph.data
training2_CaloHitListW.csv
training2_CaloHitListW_graph.data