justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 248701.29@dunegpschedd02.fnal.gov

Jobsub ID248701.29@dunegpschedd02.fnal.gov
Workflow ID10388
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-20 02:46:36
SiteUK_Lancaster
EntryUBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat2025-11-20 03:25:49
From worker nodeHostnamecomp02-07
cpuinfoIntel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-11-20 02:48:04
Input fileshd-protodune:pdhd_prod_beam__242517_96_1_20251113T002904Z_gen_g4_IonScintPDExt.root_247897_47_1_20251118T032224Z_PDInt.root
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-11-20 03:25:49
Saved logsjustin-logs:248701.29-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

rviceDUNE:  Early 20-Nov-2025 02:48:27 GMT JobSetup
 using the field response provided from a .root file 
%MSG
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Using custom SPE response taken from SPE_NP04_FBK_2024_without_pretrigger.dat
Using custom SPE response taken from SPE_NP04_HPK_2024_without_pretrigger.dat
%MSG-i RANDOM:  OpDetDigitizerProtoDUNEHD:opdigi@Construction  20-Nov-2025 02:48:28 GMT ModuleConstruction
Instantiated HepJamesRandom engine "opdigi:0:" with seed 449142682.
%MSG
Generating waveforms of 5255.74us = 328483 Samples
	TimeBegin: -2255.74 
	fTimeEnd: 3000 
	SampleFreq: 62.5 MHz
	ReadoutWindow: 328483 ticks
	PreTrigger: 0 ticks
	SampleSize: 16 ns
Full streaming channels: 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 
SPE Templates: 
SPE_NP04_FBK_2024_without_pretrigger.dat 4 14 24 34 40 42 45 46 47 49 50 52 55 56 57 59 60 62 65 66 67 69 70 72 75 76 77 79 84 85 86 87 94 95 96 97 104 105 106 107 114 115 116 117 120 121 124 125 127 129 130 131 134 135 137 139 140 141 144 145 147 149 150 151 154 155 157 159 
SPE_NP04_HPK_2024_without_pretrigger.dat 0 1 2 3 5 6 7 8 9 10 11 12 13 15 16 17 18 19 20 21 22 23 25 26 27 28 29 30 31 32 33 35 36 37 38 39 41 43 44 48 51 53 54 58 61 63 64 68 71 73 74 78 80 81 82 83 88 89 90 91 92 93 98 99 100 101 102 103 108 109 110 111 112 113 118 119 122 123 126 128 132 133 136 138 142 143 146 148 152 153 156 158 
Inputer: "wclsSimDepoSource:"
Outputer: "wclsSimChannelSink:postdrift"
Outputer: "wclsFrameSaver:simdigits"
wclsFrameSaver: promising to produce raw::RawDigits named "daq"
20-Nov-2025 02:48:43 GMT  Initiating request to open input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/hd-protodune/c9/a3/pdhd_prod_beam__242517_96_1_20251113T002904Z_gen_g4_IonScintPDExt.root_247897_47_1_20251118T032224Z_PDInt.root"
%MSG-i Root_Information:  TriggerResultInserter:TriggerResults@Construction  TClass::Init() 20-Nov-2025 02:48:48 GMT ModuleConstruction
no dictionary for class sim::IDE is available
ROOT severity: 2000
%MSG
20-Nov-2025 02:48:48 GMT  Opened input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/hd-protodune/c9/a3/pdhd_prod_beam__242517_96_1_20251113T002904Z_gen_g4_IonScintPDExt.root_247897_47_1_20251118T032224Z_PDInt.root"
Begin processing the 1st record. run: 20250627 subRun: 1 event: 260 at 20-Nov-2025 02:48:59 GMT
%MSG-w PhotonBackTrackerService:  ProcessEvent 20-Nov-2025 02:49:09 GMT  run: 20250627 subRun: 1 event: 260
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
SimDepoSource got 1348512 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
Larwirecell::SimDepoSource got 1348512 associated depos from InputTag: label = 'IonAndScint', instance = 'priorSCE'
SimDepoSource: ready with 1348512 depos spanning: [-3315.05, 2998.21]us
Retagger: tagging trace set: daq with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "daq"
20-Nov-2025 03:06:58 GMT  Opened output file with pattern "temp_detsim.root"
%MSG-i FastCloning:  PostProcessEvent 20-Nov-2025 03:06:59 GMT  run: 20250627 subRun: 1 event: 260
Fast cloning event data products from input file.
%MSG
Begin processing the 2nd record. run: 20250627 subRun: 1 event: 261 at 20-Nov-2025 03:10:39 GMT
%MSG-w PhotonBackTrackerService:  ProcessEvent 20-Nov-2025 03:10:48 GMT  run: 20250627 subRun: 1 event: 261
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
SimDepoSource got 1012802 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
Larwirecell::SimDepoSource got 1012802 associated depos from InputTag: label = 'IonAndScint', instance = 'priorSCE'
SimDepoSource: ready with 1012802 depos spanning: [-3299.42, 2998.91]us
Retagger: tagging trace set: daq with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "daq"
Begin processing the 3rd record. run: 20250627 subRun: 1 event: 262 at 20-Nov-2025 03:16:40 GMT
R__unzip: error -3 in inflate (zlib)
%MSG-w ParticleInventory:  ProcessEvent 20-Nov-2025 03:16:45 GMT  run: 20250627 subRun: 1 event: 262
Rebuild failed to get the MCParticles. This is expected when running on a generation or simulation step.
%MSG
R__unzip: error -3 in inflate (zlib)
%MSG-w ParticleInventory:  ProcessEvent 20-Nov-2025 03:16:49 GMT  run: 20250627 subRun: 1 event: 262
Rebuild failed to get the MCParticles. This is expected when running on a generation or simulation step.
%MSG
R__unzip: error -3 in inflate (zlib)
%MSG-w ParticleInventory:  ProcessEvent 20-Nov-2025 03:16:53 GMT  run: 20250627 subRun: 1 event: 262
Rebuild failed to get the MCParticles. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService:  ProcessEvent 20-Nov-2025 03:16:53 GMT  run: 20250627 subRun: 1 event: 262
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
SimDepoSource got 1410195 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
Larwirecell::SimDepoSource got 1410195 associated depos from InputTag: label = 'IonAndScint', instance = 'priorSCE'
SimDepoSource: ready with 1410195 depos spanning: [-3321.54, 2995.24]us
Retagger: tagging trace set: daq with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "daq"
R__unzip: error -3 in inflate (zlib)
20-Nov-2025 03:25:20 GMT  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/hd-protodune/c9/a3/pdhd_prod_beam__242517_96_1_20251113T002904Z_gen_g4_IonScintPDExt.root_247897_47_1_20251118T032224Z_PDInt.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                    0.00617529      475.011       1073.84       351.185       447.05          3     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00617529     0.059111      0.120974      0.0501838     0.0472896        3     
simulate:rns:RandomNumberSaver                5.0142e-05    0.000127485   0.000275643   5.6669e-05    0.000104798       3     
simulate:tpcrawdecoder:WireCellToolkit          316.945       599.065       1024.25         456         305.964         3     
simulate:opdigi:OpDetDigitizerProtoDUNEHD       30.2951       40.2439       45.7989       44.6377       7.05081         3     
simulate:crt:CRTSimRefac                       0.123086      0.141338      0.159591      0.141338      0.018252         2     
[art]:TriggerResults:TriggerResultInserter    3.1683e-05    6.3635e-05    9.5587e-05    6.3635e-05    3.1952e-05        2     
end_path:out1:RootOutput                       8.34e-06     1.64425e-05   2.4545e-05    1.64425e-05   8.1025e-06        2     
end_path:out1:RootOutput(write)                 3.6899         4.132        4.57409        4.132       0.442099         2     
================================================================================================================================
%MSG-i NuRandomService:  RootOutput:out1@EndJob 20-Nov-2025 03:25:20 GMT  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'random'
  master seed: 508632448
  seed within: [ 1 ; 900000000 ]

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9019.7 MB
  Peak resident set size usage (VmHWM): 7292.96 MB
  Details saved in: 'mem.db'
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 3 passed = 2 failed = 1

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport          2          2          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 1291.125193 Real = 2190.269212

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 9019.7 VmHWM = 7292.96

%MSG-s ArtException:  PostEndJob 20-Nov-2025 03:25:22 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- FileReadError BEGIN
      ---- FatalRootError BEGIN
        Fatal Root Error: TBasket::ReadBasketBuffers
        fNbytes = 91782878, fKeylen = 109, fObjlen = 386893433, noutot = 150994935, nout=0, nin=5617843, nbuf=16777215
        ROOT severity: 3000
      ---- FatalRootError END
      
      The above exception was thrown while processing module CRTSimRefac/crt run: 20250627 subRun: 1 event: 262
    ---- FileReadError END
    Exception going through path simulate
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 2 entries while art::RNGsnapshots_rns__IonScintPDExt. has 10 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Detsim returns 1
justIN time: 2025-12-19 02:04:23 UTC       justIN version: 01.05.03