justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 266996.47@dunegpschedd01.fnal.gov

Jobsub ID266996.47@dunegpschedd01.fnal.gov
Workflow ID11160
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-11 20:42:39
SiteCA_SFU
EntryDUNE_CA_SFU_lcg-ce3
Last heartbeat2025-12-12 06:38:10
From worker nodeHostnamefc30570.int.fir.alliancecan.ca
cpuinfoAMD EPYC 9655 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit84598 (23 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-12-12 06:25:17
Input fileshd-protodune:pdhd_prod_beam__248370_37_1_20251113T032209Z_gen_g4_IonScintPDExt.root_253504_107_1_20251118T004123Z_PDInt.root
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-12-12 06:38:10
Saved logsjustin-logs:266996.47-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

NDOM:  OpDetDigitizerProtoDUNEHD:opdigi@Construction  11-Dec-2025 22:25:27 PST ModuleConstruction
Instantiated HepJamesRandom engine "opdigi:0:" with seed 549004594.
%MSG
Generating waveforms of 5255.74us = 328483 Samples
	TimeBegin: -2255.74 
	fTimeEnd: 3000 
	SampleFreq: 62.5 MHz
	ReadoutWindow: 328483 ticks
	PreTrigger: 0 ticks
	SampleSize: 16 ns
Full streaming channels: 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 
SPE Templates: 
SPE_NP04_FBK_2024_without_pretrigger.dat 4 14 24 34 40 42 45 46 47 49 50 52 55 56 57 59 60 62 65 66 67 69 70 72 75 76 77 79 84 85 86 87 94 95 96 97 104 105 106 107 114 115 116 117 120 121 124 125 127 129 130 131 134 135 137 139 140 141 144 145 147 149 150 151 154 155 157 159 
SPE_NP04_HPK_2024_without_pretrigger.dat 0 1 2 3 5 6 7 8 9 10 11 12 13 15 16 17 18 19 20 21 22 23 25 26 27 28 29 30 31 32 33 35 36 37 38 39 41 43 44 48 51 53 54 58 61 63 64 68 71 73 74 78 80 81 82 83 88 89 90 91 92 93 98 99 100 101 102 103 108 109 110 111 112 113 118 119 122 123 126 128 132 133 136 138 142 143 146 148 152 153 156 158 
Inputer: "wclsSimDepoSource:"
Outputer: "wclsSimChannelSink:postdrift"
Outputer: "wclsFrameSaver:simdigits"
wclsFrameSaver: promising to produce raw::RawDigits named "daq"
11-Dec-2025 22:25:36 PST  Initiating request to open input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/hd-protodune/43/60/pdhd_prod_beam__248370_37_1_20251113T032209Z_gen_g4_IonScintPDExt.root_253504_107_1_20251118T004123Z_PDInt.root"
%MSG-i Root_Information:  TriggerResultInserter:TriggerResults@Construction  TClass::Init() 11-Dec-2025 22:25:37 PST ModuleConstruction
no dictionary for class sim::IDE is available
ROOT severity: 2000
%MSG
11-Dec-2025 22:25:37 PST  Opened input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/hd-protodune/43/60/pdhd_prod_beam__248370_37_1_20251113T032209Z_gen_g4_IonScintPDExt.root_253504_107_1_20251118T004123Z_PDInt.root"
Begin processing the 1st record. run: 20250627 subRun: 1 event: 470 at 11-Dec-2025 22:25:43 PST
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Dec-2025 22:25:46 PST  run: 20250627 subRun: 1 event: 470
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
SimDepoSource got 1368499 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
Larwirecell::SimDepoSource got 1368499 associated depos from InputTag: label = 'IonAndScint', instance = 'priorSCE'
SimDepoSource: ready with 1368499 depos spanning: [-3110.25, 3074.92]us
Retagger: tagging trace set: daq with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "daq"
11-Dec-2025 22:28:42 PST  Opened output file with pattern "temp_detsim.root"
%MSG-i FastCloning:  PostProcessEvent 11-Dec-2025 22:28:42 PST  run: 20250627 subRun: 1 event: 470
Fast cloning event data products from input file.
%MSG
Begin processing the 2nd record. run: 20250627 subRun: 1 event: 471 at 11-Dec-2025 22:30:04 PST
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Dec-2025 22:30:08 PST  run: 20250627 subRun: 1 event: 471
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
SimDepoSource got 1082437 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
Larwirecell::SimDepoSource got 1082437 associated depos from InputTag: label = 'IonAndScint', instance = 'priorSCE'
SimDepoSource: ready with 1082437 depos spanning: [-3276.09, 3100.4]us
Retagger: tagging trace set: daq with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "daq"
Begin processing the 3rd record. run: 20250627 subRun: 1 event: 472 at 11-Dec-2025 22:33:10 PST
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Dec-2025 22:33:14 PST  run: 20250627 subRun: 1 event: 472
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
SimDepoSource got 730497 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
Larwirecell::SimDepoSource got 730497 associated depos from InputTag: label = 'IonAndScint', instance = 'priorSCE'
SimDepoSource: ready with 730497 depos spanning: [-3300.51, 2998]us
Retagger: tagging trace set: daq with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "daq"
Begin processing the 4th record. run: 20250627 subRun: 1 event: 473 at 11-Dec-2025 22:35:08 PST
%MSG-w ParticleInventory:  ProcessEvent 11-Dec-2025 22:35:10 PST  run: 20250627 subRun: 1 event: 473
Rebuild failed to get the MCParticles. This is expected when running on a generation or simulation step.
%MSG
%MSG-w ParticleInventory:  ProcessEvent 11-Dec-2025 22:35:11 PST  run: 20250627 subRun: 1 event: 473
Rebuild failed to get the MCParticles. This is expected when running on a generation or simulation step.
%MSG
%MSG-w ParticleInventory:  ProcessEvent 11-Dec-2025 22:35:12 PST  run: 20250627 subRun: 1 event: 473
Rebuild failed to get the MCParticles. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService:  ProcessEvent 11-Dec-2025 22:35:12 PST  run: 20250627 subRun: 1 event: 473
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
SimDepoSource got 1226558 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
Larwirecell::SimDepoSource got 1226558 associated depos from InputTag: label = 'IonAndScint', instance = 'priorSCE'
SimDepoSource: ready with 1226558 depos spanning: [-3271.98, 2994.06]us
Retagger: tagging trace set: daq with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "daq"
11-Dec-2025 22:37:55 PST  Closed input file "root://fndcadoor.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/hd-protodune/43/60/pdhd_prod_beam__248370_37_1_20251113T032209Z_gen_g4_IonScintPDExt.root_253504_107_1_20251118T004123Z_PDInt.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                    0.00462187      118.115       181.989       145.234       73.4379         4     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00462187     0.0178326     0.0556777    0.00551544     0.0218531        4     
simulate:rns:RandomNumberSaver                 4.209e-05    0.00010565    0.000269229    5.564e-05    9.46402e-05       4     
simulate:tpcrawdecoder:WireCellToolkit          103.075       144.01        167.21        152.877       24.5172         4     
simulate:opdigi:OpDetDigitizerProtoDUNEHD       8.19524       13.1339       18.2182       13.0611       3.55356         4     
simulate:crt:CRTSimRefac                       0.028569      0.0607057     0.0777151     0.0758331     0.0227371        3     
[art]:TriggerResults:TriggerResultInserter     1.782e-05    3.18867e-05    5.998e-05     1.786e-05    1.9865e-05        3     
end_path:out1:RootOutput                       5.43e-06      9.62e-06      1.771e-05     5.72e-06     5.72172e-06       3     
end_path:out1:RootOutput(write)                 1.61831       1.84051       2.00994       1.89326      0.164176         3     
================================================================================================================================
%MSG-i NuRandomService:  RootOutput:out1@EndJob 11-Dec-2025 22:37:55 PST  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'random'
  master seed: 706161065
  seed within: [ 1 ; 900000000 ]

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8259.78 MB
  Peak resident set size usage (VmHWM): 6552.71 MB
  Details saved in: 'mem.db'
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 4 passed = 3 failed = 1

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport          3          3          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 616.487369 Real = 736.214072

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 8259.78 VmHWM = 6552.71

%MSG-s ArtException:  PostEndJob 11-Dec-2025 22:37:55 PST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- FileReadError BEGIN
      ---- FatalRootError BEGIN
        Fatal Root Error: TBufferFile::ReadClassBuffer
        class: TVector3, attempting to access a wrong version: 259, object skipped at offset 331020445
        ROOT severity: 3000
      ---- FatalRootError END
      
      The above exception was thrown while processing module CRTSimRefac/crt run: 20250627 subRun: 1 event: 473
    ---- FileReadError END
    Exception going through path simulate
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 3 entries while art::RNGsnapshots_rns__IonScintPDExt. has 10 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Detsim returns 1
justIN time: 2025-12-19 04:37:37 UTC       justIN version: 01.05.03