justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 262722.0@dunegpschedd02.fnal.gov

Jobsub ID262722.0@dunegpschedd02.fnal.gov
Workflow ID11230
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes14680064000 (14000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-15 20:03:30
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce15-multicore
Last heartbeat2025-12-15 20:24:52
From worker nodeHostnametdm003.pic.es
cpuinfoAMD EPYC 7662 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes15728640000 (15000 MiB)
Wall seconds limit216000 (60 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-15 20:04:50
Input filesmonte-carlo-011230-000001
JobscriptExit code0
Real time14m (880s)
CPU time36m (2210s = 251%)
Max RSS bytes14393114624 (13726 MiB)
Outputting started2025-12-15 20:19:31
Output filesusertests:prod_cosmics_protodunevd_1_262722_0_1765829127_gen_g4_stage1.root
usertests:prod_cosmics_protodunevd_1_262722_0_1765829127_gen_g4_stage1.err
usertests:prod_cosmics_protodunevd_1_262722_0_1765829127_gen.log
usertests:prod_cosmics_protodunevd_1_262722_0_1765829127_gen.root
usertests:prod_cosmics_protodunevd_1_262722_0_1765829127_gen.err
usertests:prod_cosmics_protodunevd_1_262722_0_1765829127_gen_g4_stage1.log
Finished2025-12-15 20:24:45
Saved logsjustin-logs:262722.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ches of 50000
Processing batch 1 (deposits 0 to 50000)
Processing batch 2 (deposits 50000 to 100000)
Processing batch 3 (deposits 100000 to 150000)
Processing batch 4 (deposits 150000 to 200000)
Processing batch 5 (deposits 200000 to 250000)
Processing batch 6 (deposits 250000 to 300000)
Processing batch 7 (deposits 300000 to 350000)
Processing batch 8 (deposits 350000 to 400000)
Processing batch 9 (deposits 400000 to 450000)
Processing batch 10 (deposits 450000 to 500000)
Processing batch 11 (deposits 500000 to 550000)
Processing batch 12 (deposits 550000 to 600000)
Processing batch 13 (deposits 600000 to 650000)
Processing batch 14 (deposits 650000 to 700000)
Processing batch 15 (deposits 700000 to 750000)
Processing batch 16 (deposits 750000 to 800000)
Processing batch 17 (deposits 800000 to 850000)
Processing batch 18 (deposits 850000 to 900000)
Processing batch 19 (deposits 900000 to 950000)
Processing batch 20 (deposits 950000 to 1000000)
Processing batch 21 (deposits 1000000 to 1050000)
Processing batch 22 (deposits 1050000 to 1100000)
Processing batch 23 (deposits 1100000 to 1150000)
Processing batch 24 (deposits 1150000 to 1200000)
Processing batch 25 (deposits 1200000 to 1250000)
Processing batch 26 (deposits 1250000 to 1300000)
Processing batch 27 (deposits 1300000 to 1350000)
Processing batch 28 (deposits 1350000 to 1400000)
Processing batch 29 (deposits 1400000 to 1450000)
Processing batch 30 (deposits 1450000 to 1500000)
Processing batch 31 (deposits 1500000 to 1550000)
Processing batch 32 (deposits 1550000 to 1600000)
Processing batch 33 (deposits 1600000 to 1650000)
Processing batch 34 (deposits 1650000 to 1700000)
Processing batch 35 (deposits 1700000 to 1750000)
Processing batch 36 (deposits 1750000 to 1800000)
Processing batch 37 (deposits 1800000 to 1850000)
Processing batch 38 (deposits 1850000 to 1900000)
Processing batch 39 (deposits 1900000 to 1950000)
Processing batch 40 (deposits 1950000 to 2000000)
Processing batch 41 (deposits 2000000 to 2050000)
Processing batch 42 (deposits 2050000 to 2100000)
Processing batch 43 (deposits 2100000 to 2150000)
Processing batch 44 (deposits 2150000 to 2200000)
Processing batch 45 (deposits 2200000 to 2250000)
Processing batch 46 (deposits 2250000 to 2300000)
Processing batch 47 (deposits 2300000 to 2350000)
Processing batch 48 (deposits 2350000 to 2400000)
Processing batch 49 (deposits 2400000 to 2450000)
Processing batch 50 (deposits 2450000 to 2500000)
Processing batch 51 (deposits 2500000 to 2550000)
Processing batch 52 (deposits 2550000 to 2600000)
Processing batch 53 (deposits 2600000 to 2650000)
Processing batch 54 (deposits 2650000 to 2700000)
Processing batch 55 (deposits 2700000 to 2750000)
Processing batch 56 (deposits 2750000 to 2800000)
Processing batch 57 (deposits 2800000 to 2850000)
Processing batch 58 (deposits 2850000 to 2900000)
Processing batch 59 (deposits 2900000 to 2950000)
Processing batch 60 (deposits 2950000 to 3000000)
Processing batch 61 (deposits 3000000 to 3050000)
Processing batch 62 (deposits 3050000 to 3100000)
Processing batch 63 (deposits 3100000 to 3150000)
Processing batch 64 (deposits 3150000 to 3200000)
Processing batch 65 (deposits 3200000 to 3250000)
Processing batch 66 (deposits 3250000 to 3300000)
Processing batch 67 (deposits 3300000 to 3350000)
Processing batch 68 (deposits 3350000 to 3400000)
Processing batch 69 (deposits 3400000 to 3450000)
Processing batch 70 (deposits 3450000 to 3500000)
Processing batch 71 (deposits 3500000 to 3507411)
PDFastSimANN module produced 3507411 images...
IonAndScint endJob.
PDFastSimANN endJob.
Close TF session.
15-Dec-2025 21:18:48 CET  Closed input file "prod_cosmics_protodunevd_1_262722_0_1765829127_gen_g4_stage1.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      125.951       134.709       143.467       134.709       8.75787         2     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000394896   0.000673687   0.000952477   0.000673687   0.000278791       2     
simulate:rns:RandomNumberSaver                4.7981e-05    0.000157607   0.000267234   0.000157607   0.000109627       2     
simulate:IonAndScint:IonAndScint                2.71862       3.11994       3.52126       3.11994       0.40132         2     
simulate:PDFastSimAr:PDFastSimANN               97.3827       109.905       122.428       109.905       12.5228         2     
[art]:TriggerResults:TriggerResultInserter     2.856e-05    5.8511e-05    8.8462e-05    5.8511e-05    2.9951e-05        2     
end_path:out1:RootOutput                       3.87e-06     1.42455e-05   2.4621e-05    1.42455e-05   1.03755e-05       2     
end_path:out1:RootOutput(write)                 43.3638       43.3638       43.3638       43.3638          0            1     
================================================================================================================================
%MSG-i NuRandomService:  RootOutput:out1@EndJob 15-Dec-2025 21:18:49 CET  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1
   Configured value          Last value   ModuleLabel.InstanceName
        (per event)           189095806   IonAndScint.ISCalcAlg
        (per event)           771139286   PDFastSimAr.photon
        (per event)           471453000   PDFastSimAr.scinttime

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 16293.8 MB
  Peak resident set size usage (VmHWM): 14393.1 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 2 passed = 2 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport          2          2          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 1741.149417 Real = 327.692327

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 16293.8 VmHWM = 14393.1

%MSG-s ArtException:  PostEndJob 15-Dec-2025 21:19:29 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- FatalRootError BEGIN
    Fatal Root Error: TBufferFile::WriteByteCount
    bytecount too large (more than 1073741822)
    ROOT severity: 3000
  ---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while sim::ParticleAncestryMap_largeant__G4Stage1. has 10 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.

Stderr:
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_12_01d00/gdml/protodunevd_v5_ggd.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16484 nodes/ 2469 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).

Error in processing g4_stage2
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_12_01d00/gdml/protodunevd_v5_ggd.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16484 nodes/ 2469 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).

ls: cannot access *_g4_stage2.root: No such file or directory
ls: cannot access *_reco.root: No such file or directory
Forming g4 metadata
Forming reco metadata
justIN time: 2025-12-18 20:21:59 UTC       justIN version: 01.05.03