justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 297429.41@dunegpschedd02.fnal.gov

Jobsub ID	297429.41@dunegpschedd02.fnal.gov
Workflow ID	12654
Stage ID	4
User name	usher@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	28800 (8 hours)
Submitted time	2026-02-02 21:28:12
Site	UK_QMUL
Entry	DUNE_UK_London_QMUL_arcce03
Last heartbeat	2026-02-02 21:32:18
From worker node	Hostname	cn109.htc.esc.qmul
	cpuinfo	Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2026-02-02 21:28:40
Input files	usertests:prodgenie_monte-carlo-012654-000601_gen_g4_detsim.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2026-02-02 21:32:18
Saved logs	justin-logs:297429.41-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

iseRemoval: 0
StandardRawDigitPrepService::ctor:       DoDeconvolution: 1
StandardRawDigitPrepService::ctor:  DoPedestalAdjustment: 0
StandardRawDigitPrepService::ctor:                 DoROI: 1
StandardRawDigitPrepService::ctor:               DoWires: 1
StandardRawDigitPrepService::ctor:                DoDump: 0
StandardRawDigitPrepService::ctor:  DoIntermediateStates: 0
StandardRawDigitPrepService::ctor: No display tools.
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Deconvolution::BuildExtraFilter sigma is 4.60087
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=0 Type: SuperaMCTruth w/ Name: MultiPartVrtx
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=1 Type: SuperaBBoxInteraction w/ Name: SuperaBBoxInteraction
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=2 Type: SuperaMCParticleCluster w/ Name: SuperaMCParticleCluster
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=3 Type: SuperaSimEnergyDeposit w/ Name: SuperaSimEnergyDeposit
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=4 Type: SuperaSpacePoint w/ Name: SuperaSpacePoint
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=5 Type: Tensor3DFromCluster3D w/ Name: Tensor3DFromCluster3D
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=6 Type: CombineTensor3D w/ Name: CombineTensor3DGhost
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=7 Type: ThresholdTensor3D w/ Name: ThresholdTensor3D
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=8 Type: CombineTensor3D w/ Name: CombineTensor3D
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=9 Type: ParticleCorrector w/ Name: ParticleCorrector
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=10 Type: EmptyTensorFilter w/ Name: EmptyTensorFilter
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=11 Type: RescaleChargeTensor3D w/ Name: RescaleChargeTensor3D
02-Feb-2026 21:29:31 GMT  Initiating request to open input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/usertests/63/95/prodgenie_monte-carlo-012654-000601_gen_g4_detsim.root"
02-Feb-2026 21:29:34 GMT  Opened input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/usertests/63/95/prodgenie_monte-carlo-012654-000601_gen_g4_detsim.root"
Begin processing the 1st record. run: 100601 subRun: 0 event: 1 at 02-Feb-2026 21:29:35 GMT
++++>>>> total num hits: 500, num free: 500

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1942!
    [NORMAL]  <IOManager::get_data> Created TTree particle_mpv_tree (id=0) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree neutrino_mpv_tree (id=1) w/ 0 entries...
bad particle: track id = 0 PDG = 14 StatusCode = 0
bad particle: track id = 1 PDG = 1000180400 StatusCode = 0
bad particle: track id = 2 PDG = 2212 StatusCode = 11
bad particle: track id = 3 PDG = 1000170390 StatusCode = 2
bad particle: track id = 6 PDG = 2214 StatusCode = 3
bad particle: track id = 7 PDG = 2112 StatusCode = 14
bad particle: track id = 8 PDG = 211 StatusCode = 14
bad particle: track id = 12 PDG = 2000000002 StatusCode = 15
    [NORMAL]  <IOManager::get_data> Created TTree cluster3d_mcst_tree (id=2) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree cluster3d_pcluster_tree (id=3) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree cluster3d_sed_tree (id=4) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree cluster3d_masked_true2reco_tree (id=5) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree sparse3d_reco_tree (id=6) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree sparse3d_pcluster_index_tree (id=7) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree sparse3d_masked_true_tree (id=8) w/ 0 entries...
Filling voxel information...
   [WARNING]  <SuperaMCParticleCluster::AnalyzeSimChannel> Total IDE ctr 25618
 ... due to missing track ID 0 from 0 track IDs
 ... due to invalid particle 1907
 ... due to invalid 2d index 0
 ... due to invalid 3d index 15993

    [NORMAL]  <SuperaMCParticleCluster::process> Saved interaction count: 1
    [NORMAL]  <SuperaMCParticleCluster::process> 0 particles have no ancestor, summed deposit energy = 0
    [NORMAL]  <IOManager::get_data> Created TTree cluster3d_pcluster_highE_tree (id=9) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree cluster3d_pcluster_lowE_tree (id=10) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree sparse3d_pcluster_leftover_tree (id=11) w/ 0 entries...
  [CRITICAL]  <SuperaMCParticleCluster::process> SuperaMCParticleCluster.cxx::L2402 Particle index 8 holds particle w/ an invalid parent ID 65535
      Particle  (PdgCode,TrackID) = (22,1) ... with Parent (0,1)
      Vertex   (x, y, z, t) = (-150.929,-255.962,554.944,0)
      Momentum (px, py, pz) = (1.66526,4.29527,7.7546)
      Final Momentum (end_px, end_py, end_pz) = (1.79769e+308,1.79769e+308,1.79769e+308)
      Initial Energy  = 9.01977
      Deposit  Energy  = 0
      Creation Process = primary
      Instance ID = 8
      Group ID    = 8
      Interaction ID = 0

      Particle  (PdgCode,TrackID) = (22,1) ... with Parent (0,1)
      Vertex   (x, y, z, t) = (-150.929,-255.962,554.944,0)
      Momentum (px, py, pz) = (1.66526,4.29527,7.7546)
      Final Momentum (end_px, end_py, end_pz) = (1.79769e+308,1.79769e+308,1.79769e+308)
      Initial Energy  = 9.01977
      Deposit  Energy  = 0
      Creation Process = primary
      Instance ID = 8
      Group ID    = 8
      Interaction ID = 0

02-Feb-2026 21:29:36 GMT  Opened output file with pattern "prodgenie_monte-carlo-012654-000601_gen_g4_detsim_reco.root"
    [NORMAL]  <IOManager::finalize> Writing particle_mpv_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing neutrino_mpv_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_mcst_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_pcluster_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_sed_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_masked_true2reco_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_pcluster_index_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_masked_true_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_pcluster_highE_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_pcluster_lowE_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_pcluster_leftover_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Closing output file
02-Feb-2026 21:29:41 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/usertests/63/95/prodgenie_monte-carlo-012654-000601_gen_g4_detsim.root"
Malformed TimeTracker database.  The TimeEvent table is empty, but
the TimeModule table is not.  This can happen if an exception has
been thrown from a module while processing the first event.  Any
saved database file is suspect and should not be used.

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 14400 MB
  Peak resident set size usage (VmHWM): 937.378 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 02-Feb-2026 21:29:48 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- EventProcessorFailure BEGIN
    EndPathExecutor: an exception occurred during current event processing
    ---- ScheduleExecutionFailure BEGIN
      Path: ProcessingStopped.
      ---- StdException BEGIN
        An exception was thrown while processing module LArSoftSuperaDriver/superaMC run: 100601 subRun: 0 event: 1
        std::exception
      ---- StdException END
      Exception going through path end_path
    ---- ScheduleExecutionFailure END
  ---- EventProcessorFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 0 entries while sim::OpDetDivRecs_opdigi__detsim. has 50 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
'larcv.root' -> 'prodgenie_monte-carlo-012654-000601_gen_g4_detsim_reco_larcv.root'
ERROR: Output file prodgenie_monte-carlo-012654-000601_gen_g4_detsim_reco.root not produced!

justIN time: 2026-02-04 07:23:54 UTC       justIN version: 01.06.00