justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 296855.175@dunegpschedd02.fnal.gov

Jobsub ID	296855.175@dunegpschedd02.fnal.gov
Workflow ID	12578
Stage ID	4
User name	usher@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	28800 (8 hours)
Submitted time	2026-01-30 15:55:18
Site	CZ_FZU
Entry	Nova_CZ_FZU_osgce1
Last heartbeat	2026-01-30 15:58:31
From worker node	Hostname	milan17.farm.particle.cz
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2026-01-30 15:57:23
Input files	usertests:prodgenie_monte-carlo-012578-000392_gen_g4_detsim.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2026-01-30 15:58:31
Saved logs	justin-logs:296855.175-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ctor:          DoMitigation: 0
StandardRawDigitPrepService::ctor:  DoEarlySignalFinding: 0
StandardRawDigitPrepService::ctor:        DoNoiseRemoval: 0
StandardRawDigitPrepService::ctor:       DoDeconvolution: 1
StandardRawDigitPrepService::ctor:  DoPedestalAdjustment: 0
StandardRawDigitPrepService::ctor:                 DoROI: 1
StandardRawDigitPrepService::ctor:               DoWires: 1
StandardRawDigitPrepService::ctor:                DoDump: 0
StandardRawDigitPrepService::ctor:  DoIntermediateStates: 0
StandardRawDigitPrepService::ctor: No display tools.
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Deconvolution::BuildExtraFilter sigma is 4.60087
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=0 Type: SuperaMCTruth w/ Name: MultiPartVrtx
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=1 Type: SuperaBBoxInteraction w/ Name: SuperaBBoxInteraction
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=2 Type: SuperaMCParticleCluster w/ Name: SuperaMCParticleCluster
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=3 Type: SuperaSimEnergyDeposit w/ Name: SuperaSimEnergyDeposit
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=4 Type: SuperaSpacePoint w/ Name: SuperaSpacePoint
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=5 Type: Tensor3DFromCluster3D w/ Name: Tensor3DFromCluster3D
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=6 Type: CombineTensor3D w/ Name: CombineTensor3DGhost
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=7 Type: ThresholdTensor3D w/ Name: ThresholdTensor3D
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=8 Type: CombineTensor3D w/ Name: CombineTensor3D
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=9 Type: ParticleCorrector w/ Name: ParticleCorrector
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=10 Type: EmptyTensorFilter w/ Name: EmptyTensorFilter
    [NORMAL]  <ProcessDriver::configure> Instantiating Process ID=11 Type: RescaleChargeTensor3D w/ Name: RescaleChargeTensor3D
30-Jan-2026 16:58:07 CET  Initiating request to open input file "root://se1.farm.particle.cz:1094//dune/RSE/usertests/82/b4/prodgenie_monte-carlo-012578-000392_gen_g4_detsim.root"
30-Jan-2026 16:58:08 CET  Opened input file "root://se1.farm.particle.cz:1094//dune/RSE/usertests/82/b4/prodgenie_monte-carlo-012578-000392_gen_g4_detsim.root"
Begin processing the 1st record. run: 400392 subRun: 0 event: 1 at 30-Jan-2026 16:58:08 CET
++++>>>> total num hits: 6022, num free: 6022

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2020!
    [NORMAL]  <IOManager::get_data> Created TTree particle_mpv_tree (id=0) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree neutrino_mpv_tree (id=1) w/ 0 entries...
bad particle: track id = 0 PDG = 12 StatusCode = 0
bad particle: track id = 1 PDG = 1000180400 StatusCode = 0
bad particle: track id = 2 PDG = 2112 StatusCode = 11
bad particle: track id = 3 PDG = 1000180390 StatusCode = 2
bad particle: track id = 6 PDG = 2212 StatusCode = 14
bad particle: track id = 8 PDG = 2000000002 StatusCode = 15
    [NORMAL]  <IOManager::get_data> Created TTree cluster3d_mcst_tree (id=2) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree cluster3d_pcluster_tree (id=3) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree cluster3d_sed_tree (id=4) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree cluster3d_masked_true2reco_tree (id=5) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree sparse3d_reco_tree (id=6) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree sparse3d_pcluster_index_tree (id=7) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree sparse3d_masked_true_tree (id=8) w/ 0 entries...
Filling voxel information...
   [WARNING]  <SuperaMCParticleCluster::AnalyzeSimChannel> Total IDE ctr 59932
 ... due to missing track ID 0 from 0 track IDs
 ... due to invalid particle 679
 ... due to invalid 2d index 0
 ... due to invalid 3d index 30356

    [NORMAL]  <SuperaMCParticleCluster::process> Saved interaction count: 1
    [NORMAL]  <SuperaMCParticleCluster::process> 0 particles have no ancestor, summed deposit energy = 0
    [NORMAL]  <IOManager::get_data> Created TTree cluster3d_pcluster_highE_tree (id=9) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree cluster3d_pcluster_lowE_tree (id=10) w/ 0 entries...
    [NORMAL]  <IOManager::get_data> Created TTree sparse3d_pcluster_leftover_tree (id=11) w/ 0 entries...
  [CRITICAL]  <SuperaMCParticleCluster::process> SuperaMCParticleCluster.cxx::L2402 Particle index 21 holds particle w/ an invalid parent ID 65535
      Particle  (PdgCode,TrackID) = (22,1) ... with Parent (0,1)
      Vertex   (x, y, z, t) = (172.443,336.345,775.362,0)
      Momentum (px, py, pz) = (-7.12118,-3.07734,-2.49541)
      Final Momentum (end_px, end_py, end_pz) = (1.79769e+308,1.79769e+308,1.79769e+308)
      Initial Energy  = 8.14913
      Deposit  Energy  = 0
      Creation Process = primary
      Instance ID = 21
      Group ID    = 21
      Interaction ID = 0

      Particle  (PdgCode,TrackID) = (22,1) ... with Parent (0,1)
      Vertex   (x, y, z, t) = (172.443,336.345,775.362,0)
      Momentum (px, py, pz) = (-7.12118,-3.07734,-2.49541)
      Final Momentum (end_px, end_py, end_pz) = (1.79769e+308,1.79769e+308,1.79769e+308)
      Initial Energy  = 8.14913
      Deposit  Energy  = 0
      Creation Process = primary
      Instance ID = 21
      Group ID    = 21
      Interaction ID = 0

30-Jan-2026 16:58:10 CET  Opened output file with pattern "prodgenie_monte-carlo-012578-000392_gen_g4_detsim_reco.root"
    [NORMAL]  <IOManager::finalize> Writing particle_mpv_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing neutrino_mpv_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_mcst_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_pcluster_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_sed_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_masked_true2reco_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_reco_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_pcluster_index_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_masked_true_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_pcluster_highE_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing cluster3d_pcluster_lowE_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Writing sparse3d_pcluster_leftover_tree with 0 entries
    [NORMAL]  <IOManager::finalize> Closing output file
30-Jan-2026 16:58:14 CET  Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/usertests/82/b4/prodgenie_monte-carlo-012578-000392_gen_g4_detsim.root"
Malformed TimeTracker database.  The TimeEvent table is empty, but
the TimeModule table is not.  This can happen if an exception has
been thrown from a module while processing the first event.  Any
saved database file is suspect and should not be used.

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 18690.7 MB
  Peak resident set size usage (VmHWM): 930.902 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 30-Jan-2026 16:58:14 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- EventProcessorFailure BEGIN
    EndPathExecutor: an exception occurred during current event processing
    ---- ScheduleExecutionFailure BEGIN
      Path: ProcessingStopped.
      ---- StdException BEGIN
        An exception was thrown while processing module LArSoftSuperaDriver/superaMC run: 400392 subRun: 0 event: 1
        std::exception
      ---- StdException END
      Exception going through path end_path
    ---- ScheduleExecutionFailure END
  ---- EventProcessorFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 0 entries while sim::OpDetDivRecs_opdigi__detsim. has 50 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
'larcv.root' -> 'prodgenie_monte-carlo-012578-000392_gen_g4_detsim_reco_larcv.root'
ERROR: Output file prodgenie_monte-carlo-012578-000392_gen_g4_detsim_reco.root not produced!

justIN time: 2026-02-04 04:29:04 UTC       justIN version: 01.06.00