justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 12946.1@dunegpschedd02.fnal.gov

Jobsub ID12946.1@dunegpschedd02.fnal.gov
Workflow ID162
Stage ID1
User nameszh2@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-07-30 18:19:53
SiteBR_CBPF
EntryDUNE_BR_CBPF_ce01
Last heartbeat2025-07-30 18:27:06
From worker nodeHostnamewn61
cpuinfoIntel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-07-30 18:21:10
Input files
JobscriptExit code11
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-07-30 18:27:06
Saved logsjustin-logs:12946.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

bilityService:  Early 30-Jul-2025 15:24:51 -03 JobSetup
PhotonVisbilityService initializing
%MSG
%MSG-i SignalShapingServiceDUNE:  Early 30-Jul-2025 15:24:51 -03 JobSetup
Getting Filter from .fcl file
%MSG
%MSG-i SignalShapingServiceDUNE:  Early 30-Jul-2025 15:24:51 -03 JobSetup
 using the field response provided from a .root file 
%MSG
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
%MSG-i RANDOM:  IonAndScint:IonAndScint@Construction 30-Jul-2025 15:24:52 -03  ModuleConstruction
Instantiated HepJamesRandom engine "IonAndScint:0:ISCalcAlg" with seed 0.
%MSG
%MSG-i NuRandomService:  IonAndScint:IonAndScint@Construction  30-Jul-2025 15:24:52 -03 ModuleConstruction
Seeding HepJamesRandom engine "IonAndScint:ISCalcAlg" with seed 0.
%MSG
IonAndScint Module Construct
==== ParameterSet Keys for IonAndScint ====
 - ISCalcAlg
 - Instances
 - SavePriorSCE
 - module_label
 - module_type
Produce SimEnergyDeposit in volumes: 
 - LArG4DetectorServicevolTPCActiveInner
 - LArG4DetectorServicevolTPCActiveOuter
ScintTimeLAr Tool configure:
Fast rising time: 0, Fast decay time: 6, Slow rising time: 0, Slow decay time: 1600
%MSG-i RANDOM:  PDFastSimANN:PDFastSim@Construction 30-Jul-2025 15:24:53 -03  ModuleConstruction
Instantiated HepJamesRandom engine "PDFastSim:0:photon" with seed 0.
%MSG
%MSG-i NuRandomService:  PDFastSimANN:PDFastSim@Construction  30-Jul-2025 15:24:53 -03 ModuleConstruction
Seeding HepJamesRandom engine "PDFastSim:photon" with seed 0.
%MSG
%MSG-i RANDOM:  PDFastSimANN:PDFastSim@Construction 30-Jul-2025 15:24:53 -03  ModuleConstruction
Instantiated HepJamesRandom engine "PDFastSim:0:scinttime" with seed 0.
%MSG
%MSG-i NuRandomService:  PDFastSimANN:PDFastSim@Construction  30-Jul-2025 15:24:53 -03 ModuleConstruction
Seeding HepJamesRandom engine "PDFastSim:scinttime" with seed 0.
%MSG
PDFastSimANN Module Construct
Use Lite Photon.
G4GDML: Reading '/cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_08_02d00/gdml/protodunehd_v6_refactored.gdml'...
G4GDML: Reading definitions...
G4GDML: Reading materials...
G4GDML: Reading solids...
G4GDML: Reading structure...
G4GDML: Reading setup...
G4GDML: Reading '/cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_08_02d00/gdml/protodunehd_v6_refactored.gdml' done!
Stripping off GDML names of materials, solids and volumes ...
%MSG-i LArG4DetectorService::doBuildLVs:  larg4Main:largeant@Construction  30-Jul-2025 15:24:55 -03 ModuleConstruction
(0,0,0)

Found World:  volWorld_PV
World LV:  volWorld
Found 5537 logical volumes.

Found 8297 physical volumes.

Found 2 volume(s) with auxiliary information.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%MSG
Attaching sensitive Detector: SimEnergyDeposit to Volume:  volTPCActiveInner
%MSG-i AuxUnit:  larg4Main:largeant@Construction 30-Jul-2025 15:24:55 -03  ModuleConstruction
 Unit parsed = cm from unit category: Length
%MSG
%MSG-i AuxUnit:  larg4Main:largeant@Construction 30-Jul-2025 15:24:55 -03  ModuleConstruction
Valid StepLimit unit category obtained: Length
%MSG
%MSG-i fStepLimit:  larg4Main:largeant@Construction 30-Jul-2025 15:24:55 -03  ModuleConstruction
fStepLimit:  4.7625  0.47625 cm

%MSG
%MSG-i AuxUnit:  larg4Main:largeant@Construction 30-Jul-2025 15:24:55 -03  ModuleConstruction
 Unit parsed = V/cm from unit category: Electric field
%MSG
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Attaching sensitive Detector: SimEnergyDeposit to Volume:  volTPCActiveOuter
%MSG-i AuxUnit:  larg4Main:largeant@Construction 30-Jul-2025 15:24:55 -03  ModuleConstruction
 Unit parsed = cm from unit category: Length
%MSG
%MSG-i AuxUnit:  larg4Main:largeant@Construction 30-Jul-2025 15:24:55 -03  ModuleConstruction
Valid StepLimit unit category obtained: Length
%MSG
%MSG-i fStepLimit:  larg4Main:largeant@Construction 30-Jul-2025 15:24:55 -03  ModuleConstruction
fStepLimit:  4.7625  0.47625 cm

%MSG
%MSG-i AuxUnit:  larg4Main:largeant@Construction 30-Jul-2025 15:24:55 -03  ModuleConstruction
 Unit parsed = V/cm from unit category: Electric field
%MSG
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%MSG-w LArG4DetectorService::setStepLimits:  larg4Main:largeant@Construction  30-Jul-2025 15:24:55 -03 ModuleConstruction LArG4Detector.cc:336
Setting step limits from configuration file. This will OVERRIDE redundant stepLimit(s) set in the GDML file. Note that stepLimits are only active if enabled in the physicsListService via the appropriate parameter.
%MSG
%MSG-w LArG4DetectorService::setStepLimits:  larg4Main:largeant@Construction  30-Jul-2025 15:24:55 -03 ModuleConstruction LArG4Detector.cc:366
OVERRIDING PREVIOUSLY SET STEPLIMIT FOR VOLUME : volTPCActiveInner FROM 4.7625 mm TO 0.3 mm
%MSG
%MSG-i LArG4DetectorService::setStepLimits:  larg4Main:largeant@Construction  30-Jul-2025 15:24:55 -03 ModuleConstruction
fStepLimitOverride:  0.3 mm 0.03 cm for volume: volTPCActiveInner
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%MSG
%MSG-w LArG4DetectorService::setStepLimits:  larg4Main:largeant@Construction  30-Jul-2025 15:24:55 -03 ModuleConstruction LArG4Detector.cc:366
OVERRIDING PREVIOUSLY SET STEPLIMIT FOR VOLUME : volTPCActiveOuter FROM 4.7625 mm TO 0.3 mm
%MSG
%MSG-i LArG4DetectorService::setStepLimits:  larg4Main:largeant@Construction  30-Jul-2025 15:24:55 -03 ModuleConstruction
fStepLimitOverride:  0.3 mm 0.03 cm for volume: volTPCActiveOuter
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

%MSG
List SD Tree: 
Volume volTPCActiveInner has the following list of auxiliary information: 
--> Type: SensDet Value: SimEnergyDeposit
--> Type: StepLimit Value: 0.47625*cm
--> Type: Efield Value: 500*V/cm
Volume volTPCActiveOuter has the following list of auxiliary information: 
--> Type: SensDet Value: SimEnergyDeposit
--> Type: StepLimit Value: 0.47625*cm
--> Type: Efield Value: 0*V/cm
/
/volTPCActiveInner_SimEnergyDeposit   *** Active 
/volTPCActiveOuter_SimEnergyDeposit   *** Active 
 Collection Capacity:  0
==================================================
nr of LV ======================:  1
%MSG-i RANDOM:  larg4Main:largeant@Construction 30-Jul-2025 15:24:55 -03  ModuleConstruction
Instantiated G4Engine engine "largeant:0:" with seed 0.
%MSG
%MSG-i NuRandomService:  larg4Main:largeant@Construction  30-Jul-2025 15:24:55 -03 ModuleConstruction
Seeding G4Engine engine "largeant:" with seed 0.
%MSG
30-Jul-2025 15:24:55 -03  Initiating request to open input file "cosmic_gen_1_1.root"
30-Jul-2025 15:24:56 -03  Opened input file "cosmic_gen_1_1.root"
IonAndScint beginJob.
Using Correlated algorithm to calculate IS.
%MSG-i ISCalcCorrelated:  IonAndScint:IonAndScint@BeginJob  30-Jul-2025 15:24:56 -03 ModuleBeginJob ISCalcCorrelated.cxx:40
IonizationAndScintillation/ISCalcCorrelated Initialize.
%MSG
PDFastSimANN beginJob.
larrecodnn:phot::TFLoaderMLP Loading TF Model from: /cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash/PhotonPropagation/ComputableGraph/protodune_hd_128nm_tf2.6, Input Layer: serving_default_pos_x:0 serving_default_pos_y:0 serving_default_pos_z:0 , Output Layer: StatefulPartitionedCall:0
IonAndScint endJob.
PDFastSimANN endJob.
30-Jul-2025 15:26:17 -03  Closed input file "cosmic_gen_1_1.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
[ No processed events ]
====================================================================================================================
%MSG-i NuRandomService:  RootOutput:out1@EndJob 30-Jul-2025 15:26:17 -03  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1
   Configured value          Last value   ModuleLabel.InstanceName
         INVALID!!!                       IonAndScint.ISCalcAlg
         INVALID!!!                       PDFastSim.photon
         INVALID!!!                       PDFastSim.scinttime
         INVALID!!!                       largeant

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2040.34 MB
  Peak resident set size usage (VmHWM): 645.149 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 0 passed = 0 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport          0          0          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 0.639340 Real = 80.939184

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 2040.34 VmHWM = 645.149

%MSG-s ArtException:  PostEndJob 30-Jul-2025 15:26:17 -03 ModuleEndJob
---- OtherArt BEGIN
  An exception was thrown while processing module PDFastSimANN/PDFastSim during beginJob
  ---- TFLoaderMLP BEGIN
    In larrecodnn:phot::TFLoaderMLP: Failed to load SavedModel, status: UNKNOWN: /cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash/PhotonPropagation/ComputableGraph/protodune_hd_128nm_tf2.6/variables/variables.data-00000-of-00001; Input/output error
    	 [[{{function_node __inference__traced_restore_5453091}}{{node RestoreV2}}]]
  ---- TFLoaderMLP END
---- OtherArt END
%MSG
Art has completed and will exit with status 1.
G4 Stage failed
justIN time: 2025-08-04 21:36:43 UTC       justIN version: 01.04.00