Jobsub ID 18029.76@dunegpschedd01.fnal.gov

Jobsub ID	18029.76@dunegpschedd01.fnal.gov
Workflow ID	69
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-26 03:11:05
Site	IT_CNAF
Entry	CMSHTPC_T1_IT_CNAF_condor_ce02
Last heartbeat	2025-07-26 05:34:46
From worker node	Hostname	wn-s04-05-09-01.cr.cnaf.infn.it
	cpuinfo	AMD EPYC 7313 16-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-07-26 03:50:15
Input files	fardet-vd:prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250214T190857Z_gen_000939_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-07-26 05:34:46
Saved logs	justin-logs:18029.76-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 07:27:48 CEST run: 5178 subRun: 0 event: 9384
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 07:27:48 CEST run: 5178 subRun: 0 event: 9384
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 07:27:48 CEST run: 5178 subRun: 0 event: 9384
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 07:27:48 CEST run: 5178 subRun: 0 event: 9384
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 07:27:48 CEST run: 5178 subRun: 0 event: 9384
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 07:27:48 CEST run: 5178 subRun: 0 event: 9384
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 07:27:48 CEST run: 5178 subRun: 0 event: 9384
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 07:27:48 CEST run: 5178 subRun: 0 event: 9384
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 07:27:48 CEST run: 5178 subRun: 0 event: 9384
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 07:27:48 CEST run: 5178 subRun: 0 event: 9384
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 07:27:48 CEST run: 5178 subRun: 0 event: 9384
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 07:27:48 CEST run: 5178 subRun: 0 event: 9384
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 07:27:48 CEST run: 5178 subRun: 0 event: 9384
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 07:27:48 CEST run: 5178 subRun: 0 event: 9384
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 07:27:48 CEST run: 5178 subRun: 0 event: 9384
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 07:27:48 CEST run: 5178 subRun: 0 event: 9384
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 07:27:48 CEST run: 5178 subRun: 0 event: 9384
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 07:27:48 CEST run: 5178 subRun: 0 event: 9384
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 07:27:48 CEST run: 5178 subRun: 0 event: 9384
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 07:27:48 CEST run: 5178 subRun: 0 event: 9384
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 07:27:48 CEST run: 5178 subRun: 0 event: 9384
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 07:27:48 CEST run: 5178 subRun: 0 event: 9384
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 07:27:48 CEST run: 5178 subRun: 0 event: 9384
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 07:27:48 CEST run: 5178 subRun: 0 event: 9384
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 07:27:48 CEST run: 5178 subRun: 0 event: 9384
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 07:27:48 CEST run: 5178 subRun: 0 event: 9384
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 07:27:48 CEST run: 5178 subRun: 0 event: 9384
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 07:27:48 CEST run: 5178 subRun: 0 event: 9384
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  26-Jul-2025 07:27:48 CEST run: 5178 subRun: 0 event: 9384
Comparing two wires in the same plane: return failure
%MSG
Begin processing the 5th record. run: 5178 subRun: 0 event: 9385 at 26-Jul-2025 07:32:01 CEST
26-Jul-2025 07:32:08 CEST  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/fardet-vd/05/82/prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250214T190857Z_gen_000939_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0237174      1152.72       1978.38       1315.98       650.048         5     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00429383     0.033443      0.116815      0.0114665     0.0421554        5     
reco:pandora:StandardPandora                            897.569       1246.58       1718.33       1185.22       298.476         4     
reco:pandoraTrack:LArPandoraTrackCreation             0.000398438    0.0170425     0.0589061    0.00443283     0.0242264        4     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000240461   0.00384573     0.0122381    0.00145216    0.00487353        4     
reco:pandoracalo:Calorimetry                          0.000192301   0.00128496    0.00377519    0.000586181   0.00145004        4     
reco:pandorapid:Chi2ParticleID                        4.8581e-05    0.000454811    0.0014879    0.000141381   0.000597793       4     
reco:linecluster:LineCluster                            0.57033      0.695566      0.853023      0.679456      0.106374         4     
reco:trajcluster:TrajCluster                           0.191996      0.290368      0.353812      0.307831      0.0614127        4     
reco:pmtracktc:PMAlgTrackMaker                          7.04225       7.42954       8.20501       7.23546      0.468209         4     
reco:emtrkmichelid:EmTrackMichelId                      151.42        184.703       249.564       168.913       38.1231         4     
reco:solarflash:SolarOpFlash                           0.685939      0.894387       1.00712      0.942244      0.123568         4     
[art]:TriggerResults:TriggerResultInserter            2.9846e-05    6.3649e-05    8.3556e-05    7.0597e-05    2.16233e-05       4     
end_path:out1:RootOutput                               9.538e-06    2.40003e-05   5.7368e-05    1.45475e-05   1.93764e-05       4     
end_path:out1:RootOutput(write)                        0.170941       0.23812      0.281237      0.250151      0.0413235        4     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4042.01 MB
  Peak resident set size usage (VmHWM): 2567.59 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 26-Jul-2025 07:32:09 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- OtherArt BEGIN
      ---- LArPandora BEGIN
        LArPandora::CreatePandoraInput - Can't build MCParticle to Hit map.
        No SimChannels found with label "tpcrawdecoder:simpleSC", and BackTrackerModuleLabel isn't set in FHiCL.
        The above exception was thrown while processing module StandardPandora/pandora run: 5178 subRun: 0 event: 9385
      ---- LArPandora END
    ---- OtherArt END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 4 entries while sim::OpDetDivRecs_sipmAr10ppmExt__detsim. has 10 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.