Jobsub ID 298637.0@dunegpschedd01.fnal.gov
| Jobsub ID | 298637.0@dunegpschedd01.fnal.gov |
| Workflow ID | 12079 |
| Stage ID | 1 |
| User name | nivedpm@fnal.gov |
| Requested | Processors | 4 |
| GPU | No |
| RSS bytes | 6815744000 (6500 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2026-01-21 15:54:54 |
| Site | UK_Imperial |
| Entry | DUNE_T2_UK_London_IC_ceprod00 |
| Last heartbeat | 2026-01-21 16:00:08 |
| From worker node | Hostname | wi07.grid.hep.ph.ic.ac.uk |
| cpuinfo | Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 4 |
| RSS bytes | 7864320000 (7500 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | finished |
| Started | 2026-01-21 15:56:40 |
| Input files | monte-carlo-012079-000296
|
| Jobscript | Exit code | 0 |
| Real time | 2m (164s) |
| CPU time | 0m (48s = 29%) |
| Max RSS bytes | 565174272 (538 MiB) |
| Outputting started | 2026-01-21 15:59:25 |
| Output files | |
| Finished | 2026-01-21 16:00:08 |
| Saved logs | justin-logs:298637.0-dunegpschedd01.fnal.gov.logs.tgz |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
CParticles. This is expected when running on a generation or simulation step.
%MSG
%MSG-w ParticleInventory: ProcessEvent 21-Jan-2026 15:58:12 GMT run: 1 subRun: 0 event: 1
Rebuild failed to get the MCParticles. This is expected when running on a generation or simulation step.
%MSG
%MSG-w ParticleInventory: ProcessEvent 21-Jan-2026 15:58:12 GMT run: 1 subRun: 0 event: 1
Rebuild failed to get the MCParticles. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService: ProcessEvent 21-Jan-2026 15:58:12 GMT run: 1 subRun: 0 event: 1
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-i generatePrimaries: larg4Main:largeant@BeginModule 21-Jan-2026 15:58:12 GMT run: 1 subRun: 0 event: 1 MCTruthEventAction.cc:112
Generating 3 particles
%MSG
%MSG-i generatePrimaries: larg4Main:largeant@BeginModule 21-Jan-2026 15:58:12 GMT run: 1 subRun: 0 event: 1 MCTruthEventAction.cc:112
Generating 2 particles
%MSG
%MSG-i generatePrimaries: larg4Main:largeant@BeginModule 21-Jan-2026 15:58:12 GMT run: 1 subRun: 0 event: 1 MCTruthEventAction.cc:112
Generating 4 particles
%MSG
%MSG-i ParticleListActionService: larg4Main:largeant@BeginModule 21-Jan-2026 15:58:12 GMT run: 1 subRun: 0 event: 1
Not Stored Process summary:
conv : 44
annihil : 50
Brem : 396
Ion : 1237
compt : 2375
phot : 697
%MSG
%MSG-i endOfEventAction: larg4Main:largeant@BeginModule 21-Jan-2026 15:58:12 GMT run: 1 subRun: 0 event: 1 ParticleListAction.cc:817
MCTruth Handles Size: 3
%MSG
%MSG-i endOfEventAction: larg4Main:largeant@BeginModule 21-Jan-2026 15:58:12 GMT run: 1 subRun: 0 event: 1 ParticleListAction.cc:827
mclistHandle Size: 1
%MSG
%MSG-i endOfEventAction: larg4Main:largeant@BeginModule 21-Jan-2026 15:58:12 GMT run: 1 subRun: 0 event: 1 ParticleListAction.cc:830
Found 3 particles
%MSG
%MSG-i endOfEventAction: larg4Main:largeant@BeginModule 21-Jan-2026 15:58:12 GMT run: 1 subRun: 0 event: 1 ParticleListAction.cc:827
mclistHandle Size: 1
%MSG
%MSG-i endOfEventAction: larg4Main:largeant@BeginModule 21-Jan-2026 15:58:12 GMT run: 1 subRun: 0 event: 1 ParticleListAction.cc:830
Found 2 particles
%MSG
%MSG-i endOfEventAction: larg4Main:largeant@BeginModule 21-Jan-2026 15:58:12 GMT run: 1 subRun: 0 event: 1 ParticleListAction.cc:827
mclistHandle Size: 1
%MSG
%MSG-i endOfEventAction: larg4Main:largeant@BeginModule 21-Jan-2026 15:58:12 GMT run: 1 subRun: 0 event: 1 ParticleListAction.cc:830
Found 4 particles
%MSG
IonAndScint Module Producer
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActive
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveOuter
%MSG-i PhotonVisibilityService: PDFastSimPVS:PDFastSim@BeginModule 21-Jan-2026 15:58:12 GMT run: 1 subRun: 0 event: 1
PhotonVisibilityService Loading photon library from file /cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash/PhotonPropagation/LibraryData/lib_Protodunev7_merged_avg.root for 2352000 voxels and 90 optical detectors.
%MSG
%MSG-i PhotonLibrary: PDFastSimPVS:PDFastSim@BeginModule 21-Jan-2026 15:58:12 GMT run: 1 subRun: 0 event: 1
Reading photon library from input file: /cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash/PhotonPropagation/LibraryData/lib_Protodunev7_merged_avg.root
%MSG
21-Jan-2026 15:59:20 GMT Opened output file with pattern "prod_mpvmpr_protodunesp_1_298637_0_1769011025_gen_g4_stage1.root"
%MSG-i Root_Information: PDFastSimPVS:PDFastSim@BeginModule TClass::Init() 21-Jan-2026 15:59:20 GMT run: 1 subRun: 0 event: 1
no dictionary for class sim::IDE is available
ROOT severity: 2000
%MSG
%MSG-i FastCloning: PDFastSimPVS:PDFastSim@BeginModule 21-Jan-2026 15:59:21 GMT run: 1 subRun: 0 event: 1
Fast cloning event data products from input file.
%MSG
IonAndScint endJob.
21-Jan-2026 15:59:21 GMT Closed input file "prod_mpvmpr_protodunesp_1_298637_0_1769011025_gen.root"
Malformed TimeTracker database. The TimeEvent table is empty, but
the TimeModule table is not. This can happen if an exception has
been thrown from a module while processing the first event. Any
saved database file is suspect and should not be used.
%MSG-i NuRandomService: RootOutput:out1@EndJob 21-Jan-2026 15:59:21 GMT ModuleEndJob
Summary of seeds computed by the NuRandomService
Random policy: 'random'
master seed: 687636578
seed within: [ 1 ; 900000000 ]
Configured value Last value ModuleLabel.InstanceName
749230970 (same) IonAndScint.ISCalcAlg
554540202 (same) PDFastSim.photon
78147034 (same) PDFastSim.scinttime
230950298 (same) largeant
%MSG
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 1434.47 MB
Peak resident set size usage (VmHWM): 563.069 MB
====================================================================================================
TrigReport ---------- Event summary -------------
TrigReport Events total = 1 passed = 0 failed = 1
TrigReport ---------- Modules in End-path ----------
TrigReport Run Success Error Name
TrigReport 0 0 0 out1
TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 4.158084 Real = 73.133262
MemReport ---------- Memory summary [base-10 MB] ------
MemReport VmPeak = 1434.47 VmHWM = 563.069
%MSG-s ArtException: PostEndJob 21-Jan-2026 15:59:21 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- OtherArt BEGIN
---- PhotonLibrary BEGIN
Error in ttree load, reading photon library: /cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash/PhotonPropagation/LibraryData/lib_Protodunev7_merged_avg.root
The above exception was thrown while processing module PDFastSimPVS/PDFastSim run: 1 subRun: 0 event: 1
---- PhotonLibrary END
---- OtherArt END
Exception going through path simulate
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 0 entries while simb::MCTruths_rain__MPVMPR. has 20 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
Stderr:
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_16_00d00/gdml/protodune_v7.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1148
Info in <TGeoManager::CloseGeometry>: 36176 nodes/ 5435 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
registering to primaryGeneratorActionsMap_
registering to eventActionsMap_
registering to trackingActionsMap_
registering to steppingActionsMap_
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Error in processing g4_stage1
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_16_00d00/gdml/protodune_v7.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1148
Info in <TGeoManager::CloseGeometry>: 36176 nodes/ 5435 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
registering to primaryGeneratorActionsMap_
registering to eventActionsMap_
registering to trackingActionsMap_
registering to steppingActionsMap_
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
ls: cannot access *larcvpdsp*.root: No such file or directory
Output files:
larcv files:
formed
{
"name": "",
"namespace": "usertests",
"metadata": {
"core.file_format": "root",
"core.application.name": "reco",
"core.application.family": "dunesw",
"core.application.version": "v10_16_00d00",
"core.data_tier": "root-tuple-virtual",
"core.runs": null,
"core.runs_subruns": null,
"dune.config_file": "run_supera_pdunesp_mpvmpr_d.fcl",
"dune.campaign": "sp-protodune-spine-reco",
"core.start_time": 1769011023.0,
"core.end_time": 1769011161.0
}
} 