justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 255308.115@dunegpschedd02.fnal.gov

Jobsub ID255308.115@dunegpschedd02.fnal.gov
Workflow ID10840
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-29 18:43:29
SiteCA_SFU
EntryDUNE_CA_SFU_lcg-ce3
Last heartbeat2025-11-29 19:46:32
From worker nodeHostnamefc30608.int.fir.alliancecan.ca
cpuinfoAMD EPYC 9655 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit84598 (23 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-11-29 19:42:28
Input fileshd-protodune:pdhd_prod_beam__248380_29_1_20251113T032620Z_gen_g4_IonScintPDExt.root_253519_40_1_20251118T015451Z_PDInt.root
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-11-29 19:46:32
Saved logsjustin-logs:255308.115-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1148
Info in <TGeoManager::CloseGeometry>: 25610 nodes/ 5537 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
%MSG-i AuxDetGeometryCore:  Early 29-Nov-2025 11:42:34 PST JobSetup
New detector geometry loaded from
	/cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_10_04d00/gdml/protodunehd_v6_refactored.gdml
%MSG
%MSG-i GeometryCore:  Early 29-Nov-2025 11:42:34 PST JobSetup
Sorting volumes...
%MSG
%MSG-i GeometryCore:  Early 29-Nov-2025 11:42:34 PST JobSetup
New detector geometry loaded from
	/cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_10_04d00/gdml/protodunehd_v6_refactored.gdml
%MSG
%MSG-i DuneApaWireReadoutGeom:  Early 29-Nov-2025 11:42:34 PST JobSetup
Initializing wire readout...
%MSG
Cryostat 0:
  10240 total channels
  4 APAs
  For all identical APA:
    Number of channels per APA = 2560
    U channels per APA = 800
    V channels per APA = 800
    Z channels per APA = 960
    Pitch in U Plane = 0.4669
    Pitch in V Plane = 0.4669
    Pitch in Z Plane = 0.4792
%MSG-i SimpleChannelStatusService:  Early 29-Nov-2025 11:42:34 PST JobSetup
Loaded from configuration:
  - 0 bad channels
  - 0 noisy channels
  - largest channel ID: 10239, largest present: 10239
%MSG
%MSG-i setupProvider<DetectorPropertiesStandard>:  Early  29-Nov-2025 11:42:34 PST JobSetup
Asked to ignore 1 keys: 'InheritNumberTimeSamples'
%MSG
%MSG-i LArG4DetectorService::Ctr:  Early 29-Nov-2025 11:42:34 PST JobSetup
Reading stepLimit(s) from the configuration file, for volume(s):
%MSG
%MSG-i LArG4DetectorService::Ctr:  Early 29-Nov-2025 11:42:34 PST JobSetup
Volume: volTPCActiveInner, stepLimit: 0.3
%MSG
%MSG-i LArG4DetectorService::Ctr:  Early 29-Nov-2025 11:42:34 PST JobSetup
Volume: volTPCActiveOuter, stepLimit: 0.3
%MSG
registering to  primaryGeneratorActionsMap_
registering to   eventActionsMap_
registering to   trackingActionsMap_
registering to  steppingActionsMap_
%MSG-i ParticleListActionService:  Early 29-Nov-2025 11:42:34 PST JobSetup
The full tracking information will not be stored for particles resulting from the following processes: 
{ "conv" "LowEnConversion" "Pair" "compt" "Compt" "Brem" "phot" "Photo" "Ion" "annihil" }

%MSG
%MSG-i ParticleListActionService:  Early 29-Nov-2025 11:42:34 PST JobSetup
Trajectory sparsification enabled with SparsifyMargin : 0.015

%MSG
%MSG-i PhotonVisibilityService:  Early 29-Nov-2025 11:42:34 PST JobSetup
PhotonVisbilityService initializing
%MSG
%MSG-i SignalShapingServiceDUNE:  Early 29-Nov-2025 11:42:34 PST JobSetup
Getting Filter from .fcl file
%MSG
%MSG-i SignalShapingServiceDUNE:  Early 29-Nov-2025 11:42:34 PST JobSetup
 using the field response provided from a .root file 
%MSG
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Using custom SPE response taken from SPE_NP04_FBK_2024_without_pretrigger.dat
Using custom SPE response taken from SPE_NP04_HPK_2024_without_pretrigger.dat
%MSG-i RANDOM:  OpDetDigitizerProtoDUNEHD:opdigi@Construction  29-Nov-2025 11:42:34 PST ModuleConstruction
Instantiated HepJamesRandom engine "opdigi:0:" with seed 98691351.
%MSG
Generating waveforms of 5255.74us = 328483 Samples
	TimeBegin: -2255.74 
	fTimeEnd: 3000 
	SampleFreq: 62.5 MHz
	ReadoutWindow: 328483 ticks
	PreTrigger: 0 ticks
	SampleSize: 16 ns
Full streaming channels: 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 
SPE Templates: 
SPE_NP04_FBK_2024_without_pretrigger.dat 4 14 24 34 40 42 45 46 47 49 50 52 55 56 57 59 60 62 65 66 67 69 70 72 75 76 77 79 84 85 86 87 94 95 96 97 104 105 106 107 114 115 116 117 120 121 124 125 127 129 130 131 134 135 137 139 140 141 144 145 147 149 150 151 154 155 157 159 
SPE_NP04_HPK_2024_without_pretrigger.dat 0 1 2 3 5 6 7 8 9 10 11 12 13 15 16 17 18 19 20 21 22 23 25 26 27 28 29 30 31 32 33 35 36 37 38 39 41 43 44 48 51 53 54 58 61 63 64 68 71 73 74 78 80 81 82 83 88 89 90 91 92 93 98 99 100 101 102 103 108 109 110 111 112 113 118 119 122 123 126 128 132 133 136 138 142 143 146 148 152 153 156 158 
Inputer: "wclsSimDepoSource:"
Outputer: "wclsSimChannelSink:postdrift"
Outputer: "wclsFrameSaver:simdigits"
wclsFrameSaver: promising to produce raw::RawDigits named "daq"
29-Nov-2025 11:42:41 PST  Initiating request to open input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/hd-protodune/8c/09/pdhd_prod_beam__248380_29_1_20251113T032620Z_gen_g4_IonScintPDExt.root_253519_40_1_20251118T015451Z_PDInt.root"
%MSG-i Root_Information:  TriggerResultInserter:TriggerResults@Construction  TClass::Init() 29-Nov-2025 11:42:42 PST ModuleConstruction
no dictionary for class sim::IDE is available
ROOT severity: 2000
%MSG
29-Nov-2025 11:42:43 PST  Opened input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/hd-protodune/8c/09/pdhd_prod_beam__248380_29_1_20251113T032620Z_gen_g4_IonScintPDExt.root_253519_40_1_20251118T015451Z_PDInt.root"
Begin processing the 1st record. run: 20250627 subRun: 1 event: 570 at 29-Nov-2025 11:42:48 PST
R__unzip: error -3 in inflate (zlib)
%MSG-w ParticleInventory:  ProcessEvent 29-Nov-2025 11:42:50 PST  run: 20250627 subRun: 1 event: 570
Rebuild failed to get the MCParticles. This is expected when running on a generation or simulation step.
%MSG
R__unzip: error -3 in inflate (zlib)
%MSG-w ParticleInventory:  ProcessEvent 29-Nov-2025 11:42:51 PST  run: 20250627 subRun: 1 event: 570
Rebuild failed to get the MCParticles. This is expected when running on a generation or simulation step.
%MSG
R__unzip: error -3 in inflate (zlib)
%MSG-w ParticleInventory:  ProcessEvent 29-Nov-2025 11:42:52 PST  run: 20250627 subRun: 1 event: 570
Rebuild failed to get the MCParticles. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService:  ProcessEvent 29-Nov-2025 11:42:52 PST  run: 20250627 subRun: 1 event: 570
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
SimDepoSource got 1009826 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
Larwirecell::SimDepoSource got 1009826 associated depos from InputTag: label = 'IonAndScint', instance = 'priorSCE'
SimDepoSource: ready with 1009826 depos spanning: [-3223.46, 3029.34]us
Retagger: tagging trace set: daq with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "daq"
R__unzip: error -3 in inflate (zlib)
29-Nov-2025 11:45:00 PST  Opened output file with pattern "temp_detsim.root"
%MSG-i FastCloning:  CRTSimRefac:crt@BeginModule 29-Nov-2025 11:45:00 PST  run: 20250627 subRun: 1 event: 570
Fast cloning event data products from input file.
%MSG
29-Nov-2025 11:46:20 PST  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/hd-protodune/8c/09/pdhd_prod_beam__248380_29_1_20251113T032620Z_gen_g4_IonScintPDExt.root_253519_40_1_20251118T015451Z_PDInt.root"
Malformed TimeTracker database.  The TimeEvent table is empty, but
the TimeModule table is not.  This can happen if an exception has
been thrown from a module while processing the first event.  Any
saved database file is suspect and should not be used.
%MSG-i NuRandomService:  RootOutput:out1@EndJob 29-Nov-2025 11:46:20 PST  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'random'
  master seed: 503066724
  seed within: [ 1 ; 900000000 ]

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7059.92 MB
  Peak resident set size usage (VmHWM): 5367.23 MB
  Details saved in: 'mem.db'
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 1 passed = 0 failed = 1

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport          0          0          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 122.683682 Real = 215.848813

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 7059.92 VmHWM = 5367.23

%MSG-s ArtException:  PostEndJob 29-Nov-2025 11:46:20 PST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- FileReadError BEGIN
      ---- FatalRootError BEGIN
        Fatal Root Error: TBasket::ReadBasketBuffers
        fNbytes = 71143755, fKeylen = 109, fObjlen = 299624207, noutot = 117440505, nout=0, nin=5616957, nbuf=16777215
        ROOT severity: 3000
      ---- FatalRootError END
      
      The above exception was thrown while processing module CRTSimRefac/crt run: 20250627 subRun: 1 event: 570
    ---- FileReadError END
    Exception going through path simulate
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 0 entries while art::RNGsnapshots_rns__IonScintPDExt. has 10 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Detsim returns 1
justIN time: 2025-12-19 04:35:44 UTC       justIN version: 01.05.03