justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297414.46@dunegpschedd02.fnal.gov

Jobsub ID	297414.46@dunegpschedd02.fnal.gov
Workflow ID	12654
Stage ID	4
User name	usher@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	28800 (8 hours)
Submitted time	2026-02-02 20:46:11
Site	UK_Oxford
Entry	DUNE_UK_SGrid_Oxford_arc01
Last heartbeat	2026-02-02 20:54:00
From worker node	Hostname	t2wn062.physics.ox.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2026-02-02 20:48:08
Input files	usertests:prodgenie_monte-carlo-012654-000060_gen_g4_detsim.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2026-02-02 20:54:00
Saved logs	justin-logs:297414.46-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

nvolution: 1
StandardRawDigitPrepService::ctor:  DoPedestalAdjustment: 0
StandardRawDigitPrepService::ctor:                 DoROI: 1
StandardRawDigitPrepService::ctor:               DoWires: 1
StandardRawDigitPrepService::ctor:                DoDump: 0
StandardRawDigitPrepService::ctor:  DoIntermediateStates: 0
StandardRawDigitPrepService::ctor: No display tools.
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Deconvolution::BuildExtraFilter sigma is 4.60087
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=0 Type: SuperaMCTruth w/ Name: MultiPartVrtx
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=1 Type: SuperaBBoxInteraction w/ Name: SuperaBBoxInteraction
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=2 Type: SuperaMCParticleCluster w/ Name: SuperaMCParticleCluster
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=3 Type: SuperaSimEnergyDeposit w/ Name: SuperaSimEnergyDeposit
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=4 Type: SuperaSpacePoint w/ Name: SuperaSpacePoint
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=5 Type: Tensor3DFromCluster3D w/ Name: Tensor3DFromCluster3D
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=6 Type: CombineTensor3D w/ Name: CombineTensor3DGhost
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=7 Type: ThresholdTensor3D w/ Name: ThresholdTensor3D
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=8 Type: CombineTensor3D w/ Name: CombineTensor3D
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=9 Type: ParticleCorrector w/ Name: ParticleCorrector
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=10 Type: EmptyTensorFilter w/ Name: EmptyTensorFilter
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=11 Type: RescaleChargeTensor3D w/ Name: RescaleChargeTensor3D
02-Feb-2026 20:49:27 GMT  Initiating request to open input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/usertests/5c/22/prodgenie_monte-carlo-012654-000060_gen_g4_detsim.root"
02-Feb-2026 20:49:30 GMT  Opened input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/usertests/5c/22/prodgenie_monte-carlo-012654-000060_gen_g4_detsim.root"
Begin processing the 1st record. run: 100060 subRun: 0 event: 1 at 02-Feb-2026 20:49:31 GMT
++++>>>> total num hits: 23183, num free: 23183

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2011!
    [NORMAL]  <IOManager::get_data> Created TTree particle_mpv_tree (id=0) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree neutrino_mpv_tree (id=1) w/ 0 entries...
bad particle: track id = 0 PDG = 14 StatusCode = 0
bad particle: track id = 1 PDG = 1000180400 StatusCode = 0
bad particle: track id = 2 PDG = 2212 StatusCode = 11
bad particle: track id = 3 PDG = 1000170390 StatusCode = 2
bad particle: track id = 6 PDG = 2224 StatusCode = 3
bad particle: track id = 7 PDG = 2212 StatusCode = 14
bad particle: track id = 8 PDG = 211 StatusCode = 14
bad particle: track id = 12 PDG = 2000000002 StatusCode = 15
    [NORMAL]  <IOManager::get_data> Created TTree cluster3d_mcst_tree (id=2) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree cluster3d_pcluster_tree (id=3) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree cluster3d_sed_tree (id=4) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree cluster3d_masked_true2reco_tree (id=5) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree sparse3d_reco_tree (id=6) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree sparse3d_pcluster_index_tree (id=7) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree sparse3d_masked_true_tree (id=8) w/ 0 entries...
Filling voxel information...
   [WARNING]  <SuperaMCParticleCluster::AnalyzeSimChannel> Total IDE ctr 40047
 ... due to missing track ID 0 from 0 track IDs
 ... due to invalid particle 0
 ... due to invalid 2d index 0
 ... due to invalid 3d index 7501

    [NORMAL]  <SuperaMCParticleCluster::process> Saved interaction count: 1
    [NORMAL]  <SuperaMCParticleCluster::process> 0 particles have no ancestor, summed deposit energy = 0
    [NORMAL]  <IOManager::get_data> Created TTree cluster3d_pcluster_highE_tree (id=9) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree cluster3d_pcluster_lowE_tree (id=10) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree sparse3d_pcluster_leftover_tree (id=11) w/ 0 entries...
  [CRITICAL]  <SuperaMCParticleCluster::process> SuperaMCParticleCluster.cxx::L2402 Particle index 12 holds particle w/ an invalid parent ID 65535
      Particle  (PdgCode,TrackID) = (22,1) ... with Parent (0,1)
      Vertex   (x, y, z, t) = (151.175,131.107,996.373,0)
      Momentum (px, py, pz) = (-1.4889,1.74464,3.33448)
      Final Momentum (end_px, end_py, end_pz) = (1.79769e+308,1.79769e+308,1.79769e+308)
      Initial Energy  = 4.04714
      Deposit  Energy  = 0
      Creation Process = primary
      Instance ID = 12
      Group ID    = 12
      Interaction ID = 0

      Particle  (PdgCode,TrackID) = (22,1) ... with Parent (0,1)
      Vertex   (x, y, z, t) = (151.175,131.107,996.373,0)
      Momentum (px, py, pz) = (-1.4889,1.74464,3.33448)
      Final Momentum (end_px, end_py, end_pz) = (1.79769e+308,1.79769e+308,1.79769e+308)
      Initial Energy  = 4.04714
      Deposit  Energy  = 0
      Creation Process = primary
      Instance ID = 12
      Group ID    = 12
      Interaction ID = 0

02-Feb-2026 20:49:36 GMT  Opened output file with pattern "prodgenie_monte-carlo-012654-000060_gen_g4_detsim_reco.root"
    [NORMAL]  <IOManager::finalize> Writing particle_mpv_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing neutrino_mpv_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_mcst_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_pcluster_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_sed_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_masked_true2reco_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_pcluster_index_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_masked_true_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_pcluster_highE_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_pcluster_lowE_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_pcluster_leftover_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Closing output file
02-Feb-2026 20:53:24 GMT  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/usertests/5c/22/prodgenie_monte-carlo-012654-000060_gen_g4_detsim.root"
Malformed TimeTracker database.  The TimeEvent table is empty, but
the TimeModule table is not.  This can happen if an exception has
been thrown from a module while processing the first event.  Any
saved database file is suspect and should not be used.

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 35886.9 MB
  Peak resident set size usage (VmHWM): 891.99 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 02-Feb-2026 20:53:24 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- EventProcessorFailure BEGIN
    EndPathExecutor: an exception occurred during current event processing
    ---- ScheduleExecutionFailure BEGIN
      Path: ProcessingStopped.
      ---- StdException BEGIN
        An exception was thrown while processing module LArSoftSuperaDriver/superaMC run: 100060 subRun: 0 event: 1
        std::exception
      ---- StdException END
      Exception going through path end_path
    ---- ScheduleExecutionFailure END
  ---- EventProcessorFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 0 entries while sim::OpDetDivRecs_opdigi__detsim. has 50 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
'larcv.root' -> 'prodgenie_monte-carlo-012654-000060_gen_g4_detsim_reco_larcv.root'
ERROR: Output file prodgenie_monte-carlo-012654-000060_gen_g4_detsim_reco.root not produced!

justIN time: 2026-02-04 04:46:07 UTC       justIN version: 01.06.00