Jobsub ID 19735.85@dunegpschedd01.fnal.gov

Jobsub ID	19735.85@dunegpschedd01.fnal.gov
Workflow ID	166
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-31 08:42:39
Site	IT_CNAF
Entry	CMSHTPC_T1_IT_CNAF_condor_ce02
Last heartbeat	2025-07-31 14:15:35
From worker node	Hostname	wn-f01-14-06-02.cr.cnaf.infn.it
	cpuinfo	Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-07-31 10:35:19
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250205T054854Z_gen_000282_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-07-31 14:15:35
Saved logs	justin-logs:19735.85-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

ireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:10:34 CEST run: 5058 subRun: 0 event: 2815
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:10:34 CEST run: 5058 subRun: 0 event: 2815
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:10:34 CEST run: 5058 subRun: 0 event: 2815
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:10:34 CEST run: 5058 subRun: 0 event: 2815
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:10:34 CEST run: 5058 subRun: 0 event: 2815
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:10:34 CEST run: 5058 subRun: 0 event: 2815
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:10:34 CEST run: 5058 subRun: 0 event: 2815
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:10:34 CEST run: 5058 subRun: 0 event: 2815
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:10:34 CEST run: 5058 subRun: 0 event: 2815
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:10:34 CEST run: 5058 subRun: 0 event: 2815
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:10:34 CEST run: 5058 subRun: 0 event: 2815
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:10:34 CEST run: 5058 subRun: 0 event: 2815
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:10:34 CEST run: 5058 subRun: 0 event: 2815
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:10:34 CEST run: 5058 subRun: 0 event: 2815
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:10:34 CEST run: 5058 subRun: 0 event: 2815
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:10:34 CEST run: 5058 subRun: 0 event: 2815
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:10:34 CEST run: 5058 subRun: 0 event: 2815
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:10:34 CEST run: 5058 subRun: 0 event: 2815
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:10:34 CEST run: 5058 subRun: 0 event: 2815
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:10:34 CEST run: 5058 subRun: 0 event: 2815
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:10:34 CEST run: 5058 subRun: 0 event: 2815
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:10:34 CEST run: 5058 subRun: 0 event: 2815
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:10:35 CEST run: 5058 subRun: 0 event: 2815
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:10:35 CEST run: 5058 subRun: 0 event: 2815
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:10:35 CEST run: 5058 subRun: 0 event: 2815
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:10:35 CEST run: 5058 subRun: 0 event: 2815
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:10:35 CEST run: 5058 subRun: 0 event: 2815
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:10:35 CEST run: 5058 subRun: 0 event: 2815
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 16:10:35 CEST run: 5058 subRun: 0 event: 2815
Comparing two wires in the same plane: return failure
%MSG
Begin processing the 6th record. run: 5058 subRun: 0 event: 2816 at 31-Jul-2025 16:15:18 CEST
31-Jul-2025 16:15:33 CEST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/58/75/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250205T054854Z_gen_000282_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.125896       2144.56       2875.24       2451.2        976.102         6     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00526465     0.0484091     0.131226      0.0108663     0.0567707        6     
reco:pandora:StandardPandora                            2052.14       2264.41       2479.1        2192.23       167.77          5     
reco:pandoraTrack:LArPandoraTrackCreation             0.000463592    0.0155744     0.0671292    0.00285535     0.0258855        5     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000380213   0.00585284     0.0250746    0.00069362    0.00965002        5     
reco:pandoracalo:Calorimetry                          0.000335173   0.00458114    0.00946188    0.00464363    0.00355963        5     
reco:pandorapid:Chi2ParticleID                        8.7439e-05    0.000812566   0.00356016    0.000159503   0.00137417        5     
reco:linecluster:LineCluster                            1.23693       1.4134        1.70351       1.28321      0.191009         5     
reco:trajcluster:TrajCluster                           0.528605      0.576762      0.616803      0.588814      0.0301782        5     
reco:pmtracktc:PMAlgTrackMaker                          6.13499       9.62569       10.7291       10.581        1.75869         5     
reco:emtrkmichelid:EmTrackMichelId                      261.581       295.031       381.357       278.726       43.8958         5     
reco:solarflash:SolarOpFlash                            1.75871       1.86412       1.99925       1.87086      0.0853887        5     
[art]:TriggerResults:TriggerResultInserter            3.2875e-05    5.93742e-05   0.000111225   4.4709e-05    2.86471e-05       5     
end_path:out1:RootOutput                               8.034e-06    1.76178e-05   4.3781e-05     8.52e-06     1.38047e-05       5     
end_path:out1:RootOutput(write)                        0.359801       0.45731      0.653224      0.423441      0.101343         5     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4024.66 MB
  Peak resident set size usage (VmHWM): 2552.68 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 31-Jul-2025 16:15:34 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- ProductNotFound BEGIN
      A request to resolve an Ptr to a product containing items of type: simb::MCTruth with ProductID 3501390858
      cannot be satisfied because the product cannot be found.
      Probably the branch containing the product is not stored in the input file.
      The above exception was thrown while processing module StandardPandora/pandora run: 5058 subRun: 0 event: 2816
    ---- ProductNotFound END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 5 entries while sim::OpDetDivRecs_sipmAr10ppmExt__detsim. has 10 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.