Jobsub ID 267372.1@dunegpschedd01.fnal.gov
| Jobsub ID | 267372.1@dunegpschedd01.fnal.gov | |
| Workflow ID | 11159 | |
| Stage ID | 1 | |
| User name | ykermaid@fnal.gov | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 8388608000 (8000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-12-12 15:27:32 | |
| Site | US_FNAL-FermiGrid | |
| Entry | FNAL_GPGrid_ce03_mcore_op_duneonly | |
| Last heartbeat | 2025-12-12 15:50:48 | |
| From worker node | Hostname | dunegli-7890305-0-fnpc23029.fnal.gov |
| cpuinfo | AMD EPYC 7543 32-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 10485760000 (10000 MiB) | |
| Wall seconds limit | 172800 (48 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | finished | |
| Started | 2025-12-12 15:28:45 | |
| Input files | monte-carlo-011159-000001 | |
| Jobscript | Exit code | 0 |
| Real time | 16m (966s) | |
| CPU time | 17m (1049s = 108%) | |
| Max RSS bytes | 11989340160 (11433 MiB) | |
| Outputting started | 2025-12-12 15:44:52 | |
| Output files | usertests:prod_cosmics_protodunevd_1_267372_1_1765553335_gen.root usertests:prod_cosmics_protodunevd_1_267372_1_1765553335_gen.log usertests:prod_cosmics_protodunevd_1_267372_1_1765553335_gen.err usertests:prod_cosmics_protodunevd_1_267372_1_1765553335_gen_g4_stage1.root usertests:prod_cosmics_protodunevd_1_267372_1_1765553335_gen_g4_stage1.log usertests:prod_cosmics_protodunevd_1_267372_1_1765553335_gen_g4_stage1.err | |
| Finished | 2025-12-12 15:50:48 | |
| Saved logs | justin-logs:267372.1-dunegpschedd01.fnal.gov.logs.tgz | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
00000 to 2350000) Processing batch 48 (deposits 2350000 to 2400000) Processing batch 49 (deposits 2400000 to 2450000) Processing batch 50 (deposits 2450000 to 2500000) Processing batch 51 (deposits 2500000 to 2550000) Processing batch 52 (deposits 2550000 to 2600000) Processing batch 53 (deposits 2600000 to 2650000) Processing batch 54 (deposits 2650000 to 2700000) Processing batch 55 (deposits 2700000 to 2750000) Processing batch 56 (deposits 2750000 to 2800000) Processing batch 57 (deposits 2800000 to 2850000) Processing batch 58 (deposits 2850000 to 2900000) Processing batch 59 (deposits 2900000 to 2950000) Processing batch 60 (deposits 2950000 to 2992200) PDFastSimANN module produced 2992200 images... 12-Dec-2025 15:39:11 UTC Opened output file with pattern "prod_cosmics_protodunevd_1_267372_1_1765553335_gen_g4_stage1_g4_stage2.root" %MSG-i FastCloning: PostProcessEvent 12-Dec-2025 15:39:11 UTC run: 1 subRun: 0 event: 1 Fast cloning event data products from input file. %MSG Begin processing the 2nd record. run: 1 subRun: 0 event: 2 at 12-Dec-2025 15:40:21 UTC %MSG-w PhotonBackTrackerService: ProcessEvent 12-Dec-2025 15:40:26 UTC run: 1 subRun: 0 event: 2 Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step. %MSG %MSG-i NuRandomService: IonAndScint:IonAndScint@BeginModule 12-Dec-2025 15:40:26 UTC run: 1 subRun: 0 event: 2 Random seed for this event, engine 'IonAndScint.ISCalcAlg': 80037725 %MSG IonAndScint Module Producer SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolCryostat Skip SimEnergyDeposit in: LArG4DetectorServicevolBeamPlIINi SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActive %MSG-i NuRandomService: PDFastSimANN:PDFastSimAr@BeginModule 12-Dec-2025 15:40:29 UTC run: 1 subRun: 0 event: 2 Random seed for this event, engine 'PDFastSimAr.photon': 470135797 %MSG %MSG-i NuRandomService: PDFastSimANN:PDFastSimAr@BeginModule 12-Dec-2025 15:40:29 UTC run: 1 subRun: 0 event: 2 Random seed for this event, engine 'PDFastSimAr.scinttime': 187645972 %MSG PDFastSimANN Module Producer... Processing 2589292 energy deposits in batches of 50000 Processing batch 1 (deposits 0 to 50000) Processing batch 2 (deposits 50000 to 100000) Processing batch 3 (deposits 100000 to 150000) Processing batch 4 (deposits 150000 to 200000) Processing batch 5 (deposits 200000 to 250000) Processing batch 6 (deposits 250000 to 300000) Processing batch 7 (deposits 300000 to 350000) Processing batch 8 (deposits 350000 to 400000) Processing batch 9 (deposits 400000 to 450000) Processing batch 10 (deposits 450000 to 500000) Processing batch 11 (deposits 500000 to 550000) Processing batch 12 (deposits 550000 to 600000) Processing batch 13 (deposits 600000 to 650000) Processing batch 14 (deposits 650000 to 700000) Processing batch 15 (deposits 700000 to 750000) Processing batch 16 (deposits 750000 to 800000) Processing batch 17 (deposits 800000 to 850000) Processing batch 18 (deposits 850000 to 900000) Processing batch 19 (deposits 900000 to 950000) Processing batch 20 (deposits 950000 to 1000000) Processing batch 21 (deposits 1000000 to 1050000) Processing batch 22 (deposits 1050000 to 1100000) Processing batch 23 (deposits 1100000 to 1150000) Processing batch 24 (deposits 1150000 to 1200000) Processing batch 25 (deposits 1200000 to 1250000) Processing batch 26 (deposits 1250000 to 1300000) Processing batch 27 (deposits 1300000 to 1350000) Processing batch 28 (deposits 1350000 to 1400000) Processing batch 29 (deposits 1400000 to 1450000) Processing batch 30 (deposits 1450000 to 1500000) Processing batch 31 (deposits 1500000 to 1550000) Processing batch 32 (deposits 1550000 to 1600000) Processing batch 33 (deposits 1600000 to 1650000) Processing batch 34 (deposits 1650000 to 1700000) Processing batch 35 (deposits 1700000 to 1750000) Processing batch 36 (deposits 1750000 to 1800000) Processing batch 37 (deposits 1800000 to 1850000) Processing batch 38 (deposits 1850000 to 1900000) Processing batch 39 (deposits 1900000 to 1950000) Processing batch 40 (deposits 1950000 to 2000000) Processing batch 41 (deposits 2000000 to 2050000) Processing batch 42 (deposits 2050000 to 2100000) Processing batch 43 (deposits 2100000 to 2150000) Processing batch 44 (deposits 2150000 to 2200000) Processing batch 45 (deposits 2200000 to 2250000) Processing batch 46 (deposits 2250000 to 2300000) Processing batch 47 (deposits 2300000 to 2350000) Processing batch 48 (deposits 2350000 to 2400000) Processing batch 49 (deposits 2400000 to 2450000) Processing batch 50 (deposits 2450000 to 2500000) Processing batch 51 (deposits 2500000 to 2550000) Processing batch 52 (deposits 2550000 to 2589292) PDFastSimANN module produced 2589292 images... Begin processing the 3rd record. run: 1 subRun: 0 event: 3 at 12-Dec-2025 15:43:45 UTC %MSG-w PhotonBackTrackerService: ProcessEvent 12-Dec-2025 15:43:49 UTC run: 1 subRun: 0 event: 3 Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step. %MSG %MSG-i NuRandomService: IonAndScint:IonAndScint@BeginModule 12-Dec-2025 15:43:49 UTC run: 1 subRun: 0 event: 3 Random seed for this event, engine 'IonAndScint.ISCalcAlg': 128489516 %MSG IonAndScint Module Producer SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolCryostat Skip SimEnergyDeposit in: LArG4DetectorServicevolBeamPlIINi SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActive %MSG-i NuRandomService: PDFastSimANN:PDFastSimAr@BeginModule 12-Dec-2025 15:43:52 UTC run: 1 subRun: 0 event: 3 Random seed for this event, engine 'PDFastSimAr.photon': 373260523 %MSG %MSG-i NuRandomService: PDFastSimANN:PDFastSimAr@BeginModule 12-Dec-2025 15:43:52 UTC run: 1 subRun: 0 event: 3 Random seed for this event, engine 'PDFastSimAr.scinttime': 762744173 %MSG PDFastSimANN Module Producer... Processing 2317889 energy deposits in batches of 50000 Processing batch 1 (deposits 0 to 50000) Processing batch 2 (deposits 50000 to 100000) Processing batch 3 (deposits 100000 to 150000) Processing batch 4 (deposits 150000 to 200000) Processing batch 5 (deposits 200000 to 250000) Processing batch 6 (deposits 250000 to 300000) Processing batch 7 (deposits 300000 to 350000) Processing batch 8 (deposits 350000 to 400000) Processing batch 9 (deposits 400000 to 450000) Processing batch 10 (deposits 450000 to 500000) Processing batch 11 (deposits 500000 to 550000) Processing batch 12 (deposits 550000 to 600000) Processing batch 13 (deposits 600000 to 650000) Processing batch 14 (deposits 650000 to 700000) Processing batch 15 (deposits 700000 to 750000) Processing batch 16 (deposits 750000 to 800000) Processing batch 17 (deposits 800000 to 850000) Processing batch 18 (deposits 850000 to 900000) Processing batch 19 (deposits 900000 to 950000) Processing batch 20 (deposits 950000 to 1000000) Processing batch 21 (deposits 1000000 to 1050000) Processing batch 22 (deposits 1050000 to 1100000) Processing batch 23 (deposits 1100000 to 1150000) Processing batch 24 (deposits 1150000 to 1200000) Processing batch 25 (deposits 1200000 to 1250000) Processing batch 26 (deposits 1250000 to 1300000) Processing batch 27 (deposits 1300000 to 1350000) Processing batch 28 (deposits 1350000 to 1400000) Stderr: Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_12_01d00/gdml/protodunevd_v5_ggd.gdml Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100 Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes... Info in <TGeoManager::CheckGeometry>: ...Nothing to fix Info in <TGeoManager::CloseGeometry>: Counting nodes... Info in <TGeoManager::Voxelize>: Voxelizing... Info in <TGeoManager::CloseGeometry>: Building cache... Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292 Info in <TGeoManager::CloseGeometry>: 16484 nodes/ 2469 volume UID's in Geometry imported from GDML Info in <TGeoManager::CloseGeometry>: ----------------modeler ready---------------- Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak). Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak). Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak). Error in processing g4_stage2 Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_12_01d00/gdml/protodunevd_v5_ggd.gdml Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100 Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes... Info in <TGeoManager::CheckGeometry>: ...Nothing to fix Info in <TGeoManager::CloseGeometry>: Counting nodes... Info in <TGeoManager::Voxelize>: Voxelizing... Info in <TGeoManager::CloseGeometry>: Building cache... Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292 Info in <TGeoManager::CloseGeometry>: 16484 nodes/ 2469 volume UID's in Geometry imported from GDML Info in <TGeoManager::CloseGeometry>: ----------------modeler ready---------------- Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak). Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak). Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak). ls: cannot access *_g4_stage2.root: No such file or directory ls: cannot access *_reco.root: No such file or directory Forming g4 metadata Forming reco metadata