Jobsub ID 231623.123@dunegpschedd01.fnal.gov
| Jobsub ID | 231623.123@dunegpschedd01.fnal.gov |
| Workflow ID | 8686 |
| Stage ID | 1 |
| User name | gabrielb@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 25165824000 (24000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-10-08 11:10:24 |
| Site | US_UCSD |
| Entry | CMSHTPC_T2_US_UCSD_gw6 |
| Last heartbeat | 2025-10-08 13:07:15 |
| From worker node | Hostname | mh-7763-1.t2.ucsd.edu |
| cpuinfo | AMD EPYC 7763 64-Core Processor |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 26214400000 (25000 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Started | 2025-10-08 12:36:27 |
| Input files | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2025-10-08 13:07:15 |
| Saved logs | justin-logs:231623.123-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
======================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
================================================================================================================================
Full event 208.269 249.104 280.644 249.383 25.9946 5
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000960154 0.00399914 0.00908547 0.00251676 0.00289031 5
simulate:rns:RandomNumberSaver 0.000370418 0.00770584 0.0108017 0.00925441 0.00375293 5
simulate:largeant:larg4Main 10.5718 14.7801 19.6132 15.0584 3.01335 5
simulate:IonAndScint:IonAndScint 0.640804 0.851831 1.30294 0.741959 0.233976 5
simulate:IonAndScintExternal:IonAndScint 0.0185629 0.0378747 0.0523304 0.0410934 0.0111051 5
simulate:PDFastSim:PDFastSimPAR 153.077 185.081 214.924 188.249 19.7834 5
simulate:PDFastSimExternal:PDFastSimPVS 4.34346 7.03253 14.4042 5.67799 3.7414 5
[art]:TriggerResults:TriggerResultInserter 1.842e-05 3.86158e-05 0.00010024 2.18e-05 3.12826e-05 5
end_path:out1:RootOutput 1.013e-05 1.6476e-05 3.33e-05 1.414e-05 8.61046e-06 5
end_path:out1:RootOutput(write) 39.2692 51.6139 61.1995 52.9936 8.28971 4
================================================================================================================================
%MSG-i NuRandomService: RootOutput:out1@EndJob 08-Oct-2025 06:05:20 PDT ModuleEndJob
Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
algorithm version: EventTimestamp_v1
Configured value Last value ModuleLabel.InstanceName
(per event) 685284422 IonAndScint.ISCalcAlg
(per event) 627377276 IonAndScintExternal.ISCalcAlg
(per event) 340949249 PDFastSim.photon
(per event) 100339568 PDFastSim.scinttime
(per event) 4490513 PDFastSimExternal.photon
(per event) 862182557 PDFastSimExternal.scinttime
(per event) 871311475 largeant
%MSG
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 13533.7 MB
Peak resident set size usage (VmHWM): 12413.6 MB
====================================================================================================
TrigReport ---------- Event summary -------------
TrigReport Events total = 5 passed = 5 failed = 0
TrigReport ---------- Modules in End-path ----------
TrigReport Run Success Error Name
TrigReport 5 5 0 out1
TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 1251.153240 Real = 1296.174280
MemReport ---------- Memory summary [base-10 MB] ------
MemReport VmPeak = 13533.7 VmHWM = 12413.6
%MSG-s ArtException: PostEndJob 08-Oct-2025 06:06:08 PDT ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 4 entries while simb::MCTruths_cosmicgenerator__SinglesGen. has 10 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
%MSG-i MF_INIT_OK: Early 08-Oct-2025 06:06:27 PDT JobSetup
Messagelogger initialization complete.
%MSG
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_08_00d00/gdml/protodunehd_v6_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1148
Info in <TGeoManager::CloseGeometry>: 25610 nodes/ 5537 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
%MSG-i AuxDetGeometryCore: Early 08-Oct-2025 06:06:32 PDT JobSetup
New detector geometry loaded from
/cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_08_00d00/gdml/protodunehd_v6_refactored.gdml
%MSG
%MSG-i GeometryCore: Early 08-Oct-2025 06:06:32 PDT JobSetup
Sorting volumes...
%MSG
%MSG-i GeometryCore: Early 08-Oct-2025 06:06:32 PDT JobSetup
New detector geometry loaded from
/cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_08_00d00/gdml/protodunehd_v6_refactored.gdml
%MSG
%MSG-i DuneApaWireReadoutGeom: Early 08-Oct-2025 06:06:32 PDT JobSetup
Initializing wire readout...
%MSG
Cryostat 0:
10240 total channels
4 APAs
For all identical APA:
Number of channels per APA = 2560
U channels per APA = 800
V channels per APA = 800
Z channels per APA = 960
Pitch in U Plane = 0.4669
Pitch in V Plane = 0.4669
Pitch in Z Plane = 0.4792
%MSG-i SimpleChannelStatusService: Early 08-Oct-2025 06:06:32 PDT JobSetup
Loaded from configuration:
- 0 bad channels
- 0 noisy channels
- largest channel ID: 10239, largest present: 10239
%MSG
%MSG-i setupProvider<DetectorPropertiesStandard>: Early 08-Oct-2025 06:06:32 PDT JobSetup
Asked to ignore 1 keys: 'InheritNumberTimeSamples'
%MSG
%MSG-i LArG4DetectorService::Ctr: Early 08-Oct-2025 06:06:32 PDT JobSetup
Reading stepLimit(s) from the configuration file, for volume(s):
%MSG
%MSG-i LArG4DetectorService::Ctr: Early 08-Oct-2025 06:06:32 PDT JobSetup
Volume: volTPCActiveInner, stepLimit: 0.3
%MSG
%MSG-i LArG4DetectorService::Ctr: Early 08-Oct-2025 06:06:32 PDT JobSetup
Volume: volTPCActiveOuter, stepLimit: 0.3
%MSG
registering to primaryGeneratorActionsMap_
registering to eventActionsMap_
registering to trackingActionsMap_
registering to steppingActionsMap_
%MSG-i ParticleListActionService: Early 08-Oct-2025 06:06:32 PDT JobSetup
The full tracking information will not be stored for particles resulting from the following processes:
{ "conv" "LowEnConversion" "Pair" "compt" "Compt" "Brem" "phot" "Photo" "Ion" "annihil" }
%MSG
%MSG-i ParticleListActionService: Early 08-Oct-2025 06:06:32 PDT JobSetup
Trajectory sparsification enabled with SparsifyMargin : 0.015
%MSG
%MSG-i PhotonVisibilityService: Early 08-Oct-2025 06:06:32 PDT JobSetup
PhotonVisbilityService initializing
%MSG
%MSG-i SignalShapingServiceDUNE: Early 08-Oct-2025 06:06:32 PDT JobSetup
Getting Filter from .fcl file
%MSG
%MSG-i SignalShapingServiceDUNE: Early 08-Oct-2025 06:06:32 PDT JobSetup
using the field response provided from a .root file
%MSG
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Using custom SPE response taken from SPE_NP04_FBK_2024_without_pretrigger.dat
Using custom SPE response taken from SPE_NP04_HPK_2024_without_pretrigger.dat
%MSG-i RANDOM: OpDetDigitizerProtoDUNEHD:opdigi@Construction 08-Oct-2025 06:06:32 PDT ModuleConstruction
Instantiated HepJamesRandom engine "opdigi:0:" with seed 183397508.
%MSG
Generating waveforms of 5327.74us = 332983 Samples
TimeBegin: -2255.74
fTimeEnd: 3072
SampleFreq: 62.5 MHz
ReadoutWindow: 332983 ticks
PreTrigger: 0 ticks
SampleSize: 16 ns
Full streaming channels: 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159
SPE Templates:
SPE_NP04_FBK_2024_without_pretrigger.dat 4 14 24 34 40 42 45 46 47 49 50 52 55 56 57 59 60 62 65 66 67 69 70 72 75 76 77 79 84 85 86 87 94 95 96 97 104 105 106 107 114 115 116 117 120 121 124 125 127 129 130 131 134 135 137 139 140 141 144 145 147 149 150 151 154 155 157 159
SPE_NP04_HPK_2024_without_pretrigger.dat 0 1 2 3 5 6 7 8 9 10 11 12 13 15 16 17 18 19 20 21 22 23 25 26 27 28 29 30 31 32 33 35 36 37 38 39 41 43 44 48 51 53 54 58 61 63 64 68 71 73 74 78 80 81 82 83 88 89 90 91 92 93 98 99 100 101 102 103 108 109 110 111 112 113 118 119 122 123 126 128 132 133 136 138 142 143 146 148 152 153 156 158
Inputer: "wclsSimDepoSource:"
Outputer: "wclsSimChannelSink:postdrift"
Outputer: "wclsFrameSaver:simdigits"
wclsFrameSaver: promising to produce raw::RawDigits named "daq"
%MSG-s ArtException: TriggerResultInserter:TriggerResults@Construction 08-Oct-2025 06:06:53 PDT ModuleConstruction
cet::exception caught in art
---- CatalogServiceError BEGIN
Input file not found: cosmics_g4_.root.
---- CatalogServiceError END
%MSG
Art has completed and will exit with status 24.
lar exit code 24
pfn
outfile cosmics_detsim_.root 