justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297402.97@dunegpschedd02.fnal.gov

Jobsub ID	297402.97@dunegpschedd02.fnal.gov
Workflow ID	12654
Stage ID	4
User name	usher@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	28800 (8 hours)
Submitted time	2026-02-02 20:28:07
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod03
Last heartbeat	2026-02-02 20:32:59
From worker node	Hostname	wl07.grid.hep.ph.ic.ac.uk
	cpuinfo	AMD EPYC 7742 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2026-02-02 20:30:53
Input files	usertests:prodgenie_monte-carlo-012654-000656_gen_g4_detsim.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2026-02-02 20:32:59
Saved logs	justin-logs:297402.97-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ation: 0
StandardRawDigitPrepService::ctor:  DoEarlySignalFinding: 0
StandardRawDigitPrepService::ctor:        DoNoiseRemoval: 0
StandardRawDigitPrepService::ctor:       DoDeconvolution: 1
StandardRawDigitPrepService::ctor:  DoPedestalAdjustment: 0
StandardRawDigitPrepService::ctor:                 DoROI: 1
StandardRawDigitPrepService::ctor:               DoWires: 1
StandardRawDigitPrepService::ctor:                DoDump: 0
StandardRawDigitPrepService::ctor:  DoIntermediateStates: 0
StandardRawDigitPrepService::ctor: No display tools.
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Deconvolution::BuildExtraFilter sigma is 4.60087
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=0 Type: SuperaMCTruth w/ Name: MultiPartVrtx
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=1 Type: SuperaBBoxInteraction w/ Name: SuperaBBoxInteraction
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=2 Type: SuperaMCParticleCluster w/ Name: SuperaMCParticleCluster
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=3 Type: SuperaSimEnergyDeposit w/ Name: SuperaSimEnergyDeposit
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=4 Type: SuperaSpacePoint w/ Name: SuperaSpacePoint
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=5 Type: Tensor3DFromCluster3D w/ Name: Tensor3DFromCluster3D
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=6 Type: CombineTensor3D w/ Name: CombineTensor3DGhost
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=7 Type: ThresholdTensor3D w/ Name: ThresholdTensor3D
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=8 Type: CombineTensor3D w/ Name: CombineTensor3D
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=9 Type: ParticleCorrector w/ Name: ParticleCorrector
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=10 Type: EmptyTensorFilter w/ Name: EmptyTensorFilter
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=11 Type: RescaleChargeTensor3D w/ Name: RescaleChargeTensor3D
02-Feb-2026 20:31:39 GMT  Initiating request to open input file "root://xrootd.pic.es:1094/pnfs/pic.es/data/dune/RSE/usertests/ac/5e/prodgenie_monte-carlo-012654-000656_gen_g4_detsim.root"
02-Feb-2026 20:31:42 GMT  Opened input file "root://xrootd.pic.es:1094/pnfs/pic.es/data/dune/RSE/usertests/ac/5e/prodgenie_monte-carlo-012654-000656_gen_g4_detsim.root"
Begin processing the 1st record. run: 100656 subRun: 0 event: 1 at 02-Feb-2026 20:31:43 GMT
++++>>>> total num hits: 53668, num free: 53668

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2283!
    [NORMAL]  <IOManager::get_data> Created TTree particle_mpv_tree (id=0) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree neutrino_mpv_tree (id=1) w/ 0 entries...
bad particle: track id = 0 PDG = 14 StatusCode = 0
bad particle: track id = 1 PDG = 1000180400 StatusCode = 0
bad particle: track id = 2 PDG = 2112 StatusCode = 11
bad particle: track id = 3 PDG = 1000180390 StatusCode = 2
bad particle: track id = 6 PDG = 2212 StatusCode = 14
bad particle: track id = 9 PDG = 2000000002 StatusCode = 15
    [NORMAL]  <IOManager::get_data> Created TTree cluster3d_mcst_tree (id=2) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree cluster3d_pcluster_tree (id=3) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree cluster3d_sed_tree (id=4) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree cluster3d_masked_true2reco_tree (id=5) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree sparse3d_reco_tree (id=6) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree sparse3d_pcluster_index_tree (id=7) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree sparse3d_masked_true_tree (id=8) w/ 0 entries...
Filling voxel information...
   [WARNING]  <SuperaMCParticleCluster::AnalyzeSimChannel> Total IDE ctr 75291
 ... due to missing track ID 0 from 0 track IDs
 ... due to invalid particle 600
 ... due to invalid 2d index 0
 ... due to invalid 3d index 12250

    [NORMAL]  <SuperaMCParticleCluster::process> Saved interaction count: 1
    [NORMAL]  <SuperaMCParticleCluster::process> 0 particles have no ancestor, summed deposit energy = 0
    [NORMAL]  <IOManager::get_data> Created TTree cluster3d_pcluster_highE_tree (id=9) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree cluster3d_pcluster_lowE_tree (id=10) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree sparse3d_pcluster_leftover_tree (id=11) w/ 0 entries...
  [CRITICAL]  <SuperaMCParticleCluster::process> SuperaMCParticleCluster.cxx::L2402 Particle index 9 holds particle w/ an invalid parent ID 65535
      Particle  (PdgCode,TrackID) = (22,1) ... with Parent (0,1)
      Vertex   (x, y, z, t) = (-220.988,485.668,255.392,0)
      Momentum (px, py, pz) = (3.50774,2.03655,-3.32815)
      Final Momentum (end_px, end_py, end_pz) = (1.79769e+308,1.79769e+308,1.79769e+308)
      Initial Energy  = 5.24675
      Deposit  Energy  = 0
      Creation Process = primary
      Instance ID = 9
      Group ID    = 9
      Interaction ID = 0

      Particle  (PdgCode,TrackID) = (22,1) ... with Parent (0,1)
      Vertex   (x, y, z, t) = (-220.988,485.668,255.392,0)
      Momentum (px, py, pz) = (3.50774,2.03655,-3.32815)
      Final Momentum (end_px, end_py, end_pz) = (1.79769e+308,1.79769e+308,1.79769e+308)
      Initial Energy  = 5.24675
      Deposit  Energy  = 0
      Creation Process = primary
      Instance ID = 9
      Group ID    = 9
      Interaction ID = 0

02-Feb-2026 20:31:48 GMT  Opened output file with pattern "prodgenie_monte-carlo-012654-000656_gen_g4_detsim_reco.root"
    [NORMAL]  <IOManager::finalize> Writing particle_mpv_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing neutrino_mpv_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_mcst_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_pcluster_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_sed_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_masked_true2reco_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_pcluster_index_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_masked_true_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_pcluster_highE_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_pcluster_lowE_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_pcluster_leftover_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Closing output file
02-Feb-2026 20:32:34 GMT  Closed input file "root://xrootd.pic.es:1094/pnfs/pic.es/data/dune/RSE/usertests/ac/5e/prodgenie_monte-carlo-012654-000656_gen_g4_detsim.root"
Malformed TimeTracker database.  The TimeEvent table is empty, but
the TimeModule table is not.  This can happen if an exception has
been thrown from a module while processing the first event.  Any
saved database file is suspect and should not be used.

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 35895.5 MB
  Peak resident set size usage (VmHWM): 932.274 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 02-Feb-2026 20:32:34 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- EventProcessorFailure BEGIN
    EndPathExecutor: an exception occurred during current event processing
    ---- ScheduleExecutionFailure BEGIN
      Path: ProcessingStopped.
      ---- StdException BEGIN
        An exception was thrown while processing module LArSoftSuperaDriver/superaMC run: 100656 subRun: 0 event: 1
        std::exception
      ---- StdException END
      Exception going through path end_path
    ---- ScheduleExecutionFailure END
  ---- EventProcessorFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 0 entries while sim::OpDetDivRecs_opdigi__detsim. has 50 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
'larcv.root' -> 'prodgenie_monte-carlo-012654-000656_gen_g4_detsim_reco_larcv.root'
ERROR: Output file prodgenie_monte-carlo-012654-000656_gen_g4_detsim_reco.root not produced!

justIN time: 2026-02-04 04:53:06 UTC       justIN version: 01.06.00