Jobsub ID 18891.140@dunegpschedd01.fnal.gov

Jobsub ID	18891.140@dunegpschedd01.fnal.gov
Workflow ID	102
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-29 12:44:19
Site	US_UCSD
Entry	CMSHTPC_T2_US_UCSD_gw7
Last heartbeat	2025-07-29 15:52:05
From worker node	Hostname	mh-7763-14.t2.ucsd.edu
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-07-29 13:00:37
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250205T112831Z_gen_003034_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-07-29 15:52:05
Saved logs	justin-logs:18891.140-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

ersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 08:47:27 PDT run: 5059 subRun: 0 event: 30339
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 08:47:27 PDT run: 5059 subRun: 0 event: 30339
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 08:47:27 PDT run: 5059 subRun: 0 event: 30339
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 08:47:27 PDT run: 5059 subRun: 0 event: 30339
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 08:47:27 PDT run: 5059 subRun: 0 event: 30339
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 08:47:27 PDT run: 5059 subRun: 0 event: 30339
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 08:47:27 PDT run: 5059 subRun: 0 event: 30339
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 08:47:27 PDT run: 5059 subRun: 0 event: 30339
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 08:47:27 PDT run: 5059 subRun: 0 event: 30339
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 08:47:27 PDT run: 5059 subRun: 0 event: 30339
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 08:47:27 PDT run: 5059 subRun: 0 event: 30339
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 08:47:27 PDT run: 5059 subRun: 0 event: 30339
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 08:47:27 PDT run: 5059 subRun: 0 event: 30339
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 08:47:27 PDT run: 5059 subRun: 0 event: 30339
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 08:47:27 PDT run: 5059 subRun: 0 event: 30339
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 08:47:27 PDT run: 5059 subRun: 0 event: 30339
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 08:47:27 PDT run: 5059 subRun: 0 event: 30339
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 08:47:27 PDT run: 5059 subRun: 0 event: 30339
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 08:47:27 PDT run: 5059 subRun: 0 event: 30339
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 08:47:27 PDT run: 5059 subRun: 0 event: 30339
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 08:47:27 PDT run: 5059 subRun: 0 event: 30339
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 08:47:27 PDT run: 5059 subRun: 0 event: 30339
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 08:47:27 PDT run: 5059 subRun: 0 event: 30339
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 08:47:27 PDT run: 5059 subRun: 0 event: 30339
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 08:47:27 PDT run: 5059 subRun: 0 event: 30339
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 08:47:27 PDT run: 5059 subRun: 0 event: 30339
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 08:47:27 PDT run: 5059 subRun: 0 event: 30339
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 08:47:27 PDT run: 5059 subRun: 0 event: 30339
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  29-Jul-2025 08:47:27 PDT run: 5059 subRun: 0 event: 30339
Comparing two wires in the same plane: return failure
%MSG
Begin processing the 10th record. run: 5059 subRun: 0 event: 30340 at 29-Jul-2025 08:49:07 PDT
29-Jul-2025 08:51:37 PDT  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/e2/ee/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250205T112831Z_gen_003034_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0076616      746.621       852.453       823.25        249.187        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00170748     0.0179891     0.0727215    0.00646191     0.0253725       10     
reco:pandora:StandardPandora                            709.123       722.463       747.919       717.024       12.5061         9     
reco:pandoraTrack:LArPandoraTrackCreation             0.000180098   0.00603055     0.0443627    0.00153259     0.0135727        9     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000189577   0.00137091    0.00526396    0.00074428    0.00153042        9     
reco:pandoracalo:Calorimetry                          0.000223307   0.000740129   0.00185539    0.000329076   0.000670243       9     
reco:pandorapid:Chi2ParticleID                        3.3829e-05    0.000133089   0.00068828    6.8139e-05    0.000196942       9     
reco:linecluster:LineCluster                           0.846656       1.86637       3.98458       1.62971       0.9755          9     
reco:trajcluster:TrajCluster                           0.200836       0.21777      0.251144      0.216251      0.0138826        9     
reco:pmtracktc:PMAlgTrackMaker                          3.99993       4.40541       4.76269       4.38529      0.241512         9     
reco:emtrkmichelid:EmTrackMichelId                      95.4595       98.4005       104.223       98.6425       2.37379         9     
reco:solarflash:SolarOpFlash                           0.886465       2.0277        4.07044       1.74945      0.997105         9     
[art]:TriggerResults:TriggerResultInserter             1.722e-05    2.61888e-05    4.695e-05     2.211e-05    9.77065e-06       9     
end_path:out1:RootOutput                               5.21e-06     7.47989e-06    1.871e-05     6.23e-06     4.00838e-06       9     
end_path:out1:RootOutput(write)                         0.15841       0.1691       0.189043      0.164223     0.00967643        9     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4040.43 MB
  Peak resident set size usage (VmHWM): 2530.05 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 29-Jul-2025 08:51:38 PDT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- ProductNotFound BEGIN
      A request to resolve an Ptr to a product containing items of type: simb::MCTruth with ProductID 1452518184
      cannot be satisfied because the product cannot be found.
      The productGetter was not set -- are you trying to dereference a Ptr during mixing?
      The above exception was thrown while processing module StandardPandora/pandora run: 5059 subRun: 0 event: 30340
    ---- ProductNotFound END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 9 entries while sim::OpDetDivRecs_sipmAr10ppmExt__detsim. has 10 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.