Jobsub ID 261077.8@dunegpschedd02.fnal.gov

Jobsub ID	261077.8@dunegpschedd02.fnal.gov
Workflow ID	11149
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod_mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	9437184000 (9000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-12-11 16:04:20
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce4
Last heartbeat	2025-12-11 18:13:18
From worker node	Hostname	n153.dur.scotgrid.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 5220 CPU @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	10485760000 (10000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-12-11 17:12:37
Input files	hd-protodune:pdhd_prod_beam__247960_8_1_20251112T152010Z_gen_g4_IonScintPDExt.root_247891_138_1_20251118T025925Z_PDInt.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-12-11 18:13:18
Saved logs	justin-logs:261077.8-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

612.
%MSG
Generating waveforms of 5255.74us = 328483 Samples
	TimeBegin: -2255.74 
	fTimeEnd: 3000 
	SampleFreq: 62.5 MHz
	ReadoutWindow: 328483 ticks
	PreTrigger: 0 ticks
	SampleSize: 16 ns
Full streaming channels: 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 
SPE Templates: 
SPE_NP04_FBK_2024_without_pretrigger.dat 4 14 24 34 40 42 45 46 47 49 50 52 55 56 57 59 60 62 65 66 67 69 70 72 75 76 77 79 84 85 86 87 94 95 96 97 104 105 106 107 114 115 116 117 120 121 124 125 127 129 130 131 134 135 137 139 140 141 144 145 147 149 150 151 154 155 157 159 
SPE_NP04_HPK_2024_without_pretrigger.dat 0 1 2 3 5 6 7 8 9 10 11 12 13 15 16 17 18 19 20 21 22 23 25 26 27 28 29 30 31 32 33 35 36 37 38 39 41 43 44 48 51 53 54 58 61 63 64 68 71 73 74 78 80 81 82 83 88 89 90 91 92 93 98 99 100 101 102 103 108 109 110 111 112 113 118 119 122 123 126 128 132 133 136 138 142 143 146 148 152 153 156 158 
Inputer: "wclsSimDepoSource:"
Outputer: "wclsSimChannelSink:postdrift"
Outputer: "wclsFrameSaver:simdigits"
wclsFrameSaver: promising to produce raw::RawDigits named "daq"
11-Dec-2025 17:13:12 GMT  Initiating request to open input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/hd-protodune/bc/9c/pdhd_prod_beam__247960_8_1_20251112T152010Z_gen_g4_IonScintPDExt.root_247891_138_1_20251118T025925Z_PDInt.root"
%MSG-i Root_Information:  TriggerResultInserter:TriggerResults@Construction  TClass::Init() 11-Dec-2025 17:13:15 GMT ModuleConstruction
no dictionary for class sim::IDE is available
ROOT severity: 2000
%MSG
11-Dec-2025 17:13:15 GMT  Opened input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/hd-protodune/bc/9c/pdhd_prod_beam__247960_8_1_20251112T152010Z_gen_g4_IonScintPDExt.root_247891_138_1_20251118T025925Z_PDInt.root"
Begin processing the 1st record. run: 20250627 subRun: 1 event: 500 at 11-Dec-2025 17:13:25 GMT
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Dec-2025 17:14:36 GMT  run: 20250627 subRun: 1 event: 500
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
SimDepoSource got 1040351 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
Larwirecell::SimDepoSource got 1040351 associated depos from InputTag: label = 'IonAndScint', instance = 'priorSCE'
SimDepoSource: ready with 1040351 depos spanning: [-3271.7, 3110.9]us
Retagger: tagging trace set: daq with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "daq"
11-Dec-2025 17:28:16 GMT  Opened output file with pattern "temp_detsim.root"
%MSG-i FastCloning:  PostProcessEvent 11-Dec-2025 17:28:16 GMT  run: 20250627 subRun: 1 event: 500
Fast cloning event data products from input file.
%MSG
Begin processing the 2nd record. run: 20250627 subRun: 1 event: 501 at 11-Dec-2025 17:49:12 GMT
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Dec-2025 17:49:40 GMT  run: 20250627 subRun: 1 event: 501
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
SimDepoSource got 1229503 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
Larwirecell::SimDepoSource got 1229503 associated depos from InputTag: label = 'IonAndScint', instance = 'priorSCE'
SimDepoSource: ready with 1229503 depos spanning: [-3319.93, 2998.63]us
Retagger: tagging trace set: daq with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "daq"
Begin processing the 3rd record. run: 20250627 subRun: 1 event: 502 at 11-Dec-2025 17:57:46 GMT
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Dec-2025 17:58:08 GMT  run: 20250627 subRun: 1 event: 502
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
SimDepoSource got 1087832 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
Larwirecell::SimDepoSource got 1087832 associated depos from InputTag: label = 'IonAndScint', instance = 'priorSCE'
SimDepoSource: ready with 1087832 depos spanning: [-3287.1, 3025.67]us
Retagger: tagging trace set: daq with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "daq"
Begin processing the 4th record. run: 20250627 subRun: 1 event: 503 at 11-Dec-2025 18:05:08 GMT
R__unzip: error -3 in inflate (zlib)
%MSG-w ParticleInventory:  ProcessEvent 11-Dec-2025 18:05:26 GMT  run: 20250627 subRun: 1 event: 503
Rebuild failed to get the MCParticles. This is expected when running on a generation or simulation step.
%MSG
R__unzip: error -3 in inflate (zlib)
%MSG-w ParticleInventory:  ProcessEvent 11-Dec-2025 18:05:41 GMT  run: 20250627 subRun: 1 event: 503
Rebuild failed to get the MCParticles. This is expected when running on a generation or simulation step.
%MSG
R__unzip: error -3 in inflate (zlib)
%MSG-w ParticleInventory:  ProcessEvent 11-Dec-2025 18:06:02 GMT  run: 20250627 subRun: 1 event: 503
Rebuild failed to get the MCParticles. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Dec-2025 18:06:02 GMT  run: 20250627 subRun: 1 event: 503
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
SimDepoSource got 997466 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
Larwirecell::SimDepoSource got 997466 associated depos from InputTag: label = 'IonAndScint', instance = 'priorSCE'
SimDepoSource: ready with 997466 depos spanning: [-3315.13, 3084.13]us
Retagger: tagging trace set: daq with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "daq"
R__unzip: error -3 in inflate (zlib)
11-Dec-2025 18:12:45 GMT  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/hd-protodune/bc/9c/pdhd_prod_beam__247960_8_1_20251112T152010Z_gen_g4_IonScintPDExt.root_247891_138_1_20251118T025925Z_PDInt.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                    0.00510822      431.87        824.135       451.67        293.004         4     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00510822     0.0321383     0.109789     0.00682778     0.0448429        4     
simulate:rns:RandomNumberSaver                4.8738e-05    0.000131988   0.000240721   0.000119247   8.43754e-05       4     
simulate:tpcrawdecoder:WireCellToolkit          294.122       356.782       450.73        341.137       60.1693         4     
simulate:opdigi:OpDetDigitizerProtoDUNEHD       92.1146       168.127       368.558       105.917       116.012         4     
simulate:crt:CRTSimRefac                       0.112026      0.151421      0.195275      0.146963      0.0341324        3     
[art]:TriggerResults:TriggerResultInserter    2.6942e-05    4.4139e-05    6.9455e-05     3.602e-05    1.82807e-05       3     
end_path:out1:RootOutput                       6.154e-06    6.58873e-05   0.000172723   1.8785e-05     7.572e-05        3     
end_path:out1:RootOutput(write)                 4.28229       4.40755       4.5344        4.40595      0.102929         3     
================================================================================================================================
%MSG-i NuRandomService:  RootOutput:out1@EndJob 11-Dec-2025 18:12:45 GMT  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'random'
  master seed: 200456657
  seed within: [ 1 ; 900000000 ]

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8781.14 MB
  Peak resident set size usage (VmHWM): 6794.35 MB
  Details saved in: 'mem.db'
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 4 passed = 3 failed = 1

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport          3          3          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 1325.056925 Real = 3567.777037

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 8781.14 VmHWM = 6794.35

%MSG-s ArtException:  PostEndJob 11-Dec-2025 18:12:46 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- FileReadError BEGIN
      ---- FatalRootError BEGIN
        Fatal Root Error: TBasket::ReadBasketBuffers
        fNbytes = 81787375, fKeylen = 109, fObjlen = 347947222, noutot = 50331645, nout=0, nin=5642235, nbuf=16777215
        ROOT severity: 3000
      ---- FatalRootError END
      
      The above exception was thrown while processing module CRTSimRefac/crt run: 20250627 subRun: 1 event: 503
    ---- FileReadError END
    Exception going through path simulate
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 3 entries while art::RNGsnapshots_rns__IonScintPDExt. has 10 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Detsim returns 1