justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 22633.0@dunegpschedd01.fnal.gov

Jobsub ID22633.0@dunegpschedd01.fnal.gov
Workflow ID685
Stage ID1
User nameepennacc@fnal.gov
HTCondor Groupgroup_dune.prod.mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-08-09 21:36:32
SiteUS_UChicago
EntryEngage_US_MWT2_uct2_condce_mcore
Last heartbeat2025-08-10 03:15:23
From worker nodeHostnameiut2-c312.iu.edu
cpuinfoIntel(R) Xeon(R) Gold 6242 CPU @ 2.80GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-08-09 21:37:58
Input filesfardet-vd:prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250731T214830Z_gen_007400_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-08-10 03:15:23
Saved logsjustin-logs:22633.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0 EDT run: 167 subRun: 0 event: 73998
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  09-Aug-2025 23:09:30 EDT run: 167 subRun: 0 event: 73998
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  09-Aug-2025 23:09:30 EDT run: 167 subRun: 0 event: 73998
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  09-Aug-2025 23:09:30 EDT run: 167 subRun: 0 event: 73998
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  09-Aug-2025 23:09:31 EDT run: 167 subRun: 0 event: 73998
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  09-Aug-2025 23:09:31 EDT run: 167 subRun: 0 event: 73998
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  09-Aug-2025 23:09:31 EDT run: 167 subRun: 0 event: 73998
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  09-Aug-2025 23:09:31 EDT run: 167 subRun: 0 event: 73998
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  09-Aug-2025 23:09:31 EDT run: 167 subRun: 0 event: 73998
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  09-Aug-2025 23:09:31 EDT run: 167 subRun: 0 event: 73998
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  09-Aug-2025 23:09:31 EDT run: 167 subRun: 0 event: 73998
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  09-Aug-2025 23:09:31 EDT run: 167 subRun: 0 event: 73998
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  09-Aug-2025 23:09:31 EDT run: 167 subRun: 0 event: 73998
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  09-Aug-2025 23:09:31 EDT run: 167 subRun: 0 event: 73998
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  09-Aug-2025 23:09:31 EDT run: 167 subRun: 0 event: 73998
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  09-Aug-2025 23:09:31 EDT run: 167 subRun: 0 event: 73998
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  09-Aug-2025 23:09:31 EDT run: 167 subRun: 0 event: 73998
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  09-Aug-2025 23:09:31 EDT run: 167 subRun: 0 event: 73998
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  09-Aug-2025 23:09:31 EDT run: 167 subRun: 0 event: 73998
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  09-Aug-2025 23:09:31 EDT run: 167 subRun: 0 event: 73998
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  09-Aug-2025 23:09:31 EDT run: 167 subRun: 0 event: 73998
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  09-Aug-2025 23:09:31 EDT run: 167 subRun: 0 event: 73998
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  09-Aug-2025 23:09:32 EDT run: 167 subRun: 0 event: 73998
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  09-Aug-2025 23:09:32 EDT run: 167 subRun: 0 event: 73998
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  09-Aug-2025 23:09:32 EDT run: 167 subRun: 0 event: 73998
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  09-Aug-2025 23:09:32 EDT run: 167 subRun: 0 event: 73998
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  09-Aug-2025 23:09:32 EDT run: 167 subRun: 0 event: 73998
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  09-Aug-2025 23:09:32 EDT run: 167 subRun: 0 event: 73998
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  09-Aug-2025 23:09:32 EDT run: 167 subRun: 0 event: 73998
Comparing two wires in the same plane: return failure
%MSG
Begin processing the 9th record. run: 167 subRun: 0 event: 73999 at 09-Aug-2025 23:14:48 EDT
09-Aug-2025 23:14:54 EDT  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/c4/56/prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250731T214830Z_gen_007400_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0109055      2227.63       2721.34       2493.55       799.359         9     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00537227     0.0125243     0.0336982     0.0109055    0.00826625        9     
reco:pandora:StandardPandora                            1985.56       2186.87       2374.18       2159.7        120.649         8     
reco:pandoraTrack:LArPandoraTrackCreation             0.000353272    0.0119384     0.0917858    0.000495374    0.0301802        8     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000264997   0.000945209   0.00472338    0.000414873   0.00143148        8     
reco:pandoracalo:Calorimetry                          0.000203853   0.00220687    0.00709084    0.000407868   0.00272889        8     
reco:pandorapid:Chi2ParticleID                        5.2246e-05    0.000211164   0.00109501    8.26385e-05   0.000334521       8     
reco:linecluster:LineCluster                           0.329061      0.396126      0.464582      0.387567      0.0463317        8     
reco:trajcluster:TrajCluster                           0.422708      0.505479      0.606469       0.50571      0.0587544        8     
reco:pmtracktc:PMAlgTrackMaker                          8.78115       10.9579       13.2668       11.189        1.53275         8     
reco:emtrkmichelid:EmTrackMichelId                      226.183       306.088       332.35        317.668       33.1761         8     
reco:solarflash:SolarOpFlash                            0.55007      0.838508       1.06325      0.861458      0.147563         8     
[art]:TriggerResults:TriggerResultInserter            2.4381e-05    3.99331e-05   5.9957e-05    3.86715e-05   1.17649e-05       8     
end_path:out1:RootOutput                               8.116e-06    1.53382e-05   2.9795e-05    1.4568e-05    6.25244e-06       8     
end_path:out1:RootOutput(write)                        0.305939       0.40296      0.508774      0.397132      0.0597035        8     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3959.56 MB
  Peak resident set size usage (VmHWM): 2493.08 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 09-Aug-2025 23:14:55 EDT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- LogicError BEGIN
      Invalid FindOneP
      ---- ProductNotFound BEGIN
        Found zero products matching all selection criteria
          C++ type: art::Assns<simb::MCParticle,simb::MCTruth,void>
          Module label: 'largeant'
          Product instance name: ''
          Process name: (empty)
      ---- ProductNotFound END
      Attempt to use a FindOneP where the underlying art::Assns product was not found.The above exception was thrown while processing module StandardPandora/pandora run: 167 subRun: 0 event: 73999
    ---- LogicError END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 8 entries while art::TriggerResults_TriggerResults__RadBkgGen. has 10 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
justIN time: 2025-08-15 05:57:22 UTC       justIN version: 01.04.00