Jobsub ID 18933.2@dunegpschedd01.fnal.gov

Jobsub ID	18933.2@dunegpschedd01.fnal.gov
Workflow ID	117
Stage ID	1
User name	szh2@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-29 16:23:29
Site	US_PuertoRico
Entry	CMS_T3_US_PuertoRico_UPRM
Last heartbeat	2025-07-29 22:28:23
From worker node	Hostname	sandy-0-2.local
	cpuinfo	Intel(R) Xeon(R) CPU E5-1620 0 @ 3.60GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	none_processed
Started	2025-07-29 16:24:23
Input files
Outputting started	2025-07-29 22:24:45
Output files
Finished	2025-07-29 22:28:23
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

ernal@Construction  29-Jul-2025 13:10:52 AST ModuleConstruction
Instantiated HepJamesRandom engine "PDFastSimExternal:0:scinttime" with seed 0.
%MSG
%MSG-i NuRandomService:  PDFastSimPVS:PDFastSimExternal@Construction  29-Jul-2025 13:10:52 AST ModuleConstruction
Seeding HepJamesRandom engine "PDFastSimExternal:scinttime" with seed 0.
%MSG
ScintTimeLAr Tool configure:
Fast rising time: 0, Fast decay time: 6, Slow rising time: 0, Slow decay time: 1600
%MSG-i PDFastSimPVS:  PDFastSimPVS:PDFastSimExternal@Construction  29-Jul-2025 13:10:52 AST ModuleConstruction
Initializing PDFastSimPVS.
%MSG
%MSG-i PropagationTimeModel:  PDFastSimPVS:PDFastSimExternal@Construction  29-Jul-2025 13:10:52 AST ModuleConstruction
Initializing Photon propagation time model.
%MSG
%MSG-i PropagationTimeModel:  PDFastSimPVS:PDFastSimExternal@Construction  29-Jul-2025 13:10:52 AST ModuleConstruction
Using VUV timing parameterization
%MSG
%MSG-i PropagationTimeModel:  PDFastSimPVS:PDFastSimExternal@Construction  29-Jul-2025 13:10:52 AST ModuleConstruction
Generating VUV parameters
%MSG
%MSG-i PropagationTimeModel:  PDFastSimPVS:PDFastSimExternal@Construction  29-Jul-2025 13:20:10 AST ModuleConstruction
Photon propagation time model initalized.
%MSG
%MSG-i PDFastSimPVS:  PDFastSimPVS:PDFastSimExternal@Construction  29-Jul-2025 13:20:10 AST ModuleConstruction
Use Lite Photon.
%MSG
%MSG-i PDFastSimPVS:  PDFastSimPVS:PDFastSimExternal@Construction  29-Jul-2025 13:20:10 AST ModuleConstruction
PDFastSimPVS Initialization finish
%MSG
29-Jul-2025 13:20:10 AST  Initiating request to open input file "cosmic_g4_stage1_1.root"
%MSG-i Root_Information:  TriggerResultInserter:TriggerResults@Construction  TClass::Init() 29-Jul-2025 13:20:11 AST ModuleConstruction
no dictionary for class sim::IDE is available
ROOT severity: 2000
%MSG
29-Jul-2025 13:20:11 AST  Opened input file "cosmic_g4_stage1_1.root"
IonAndScint beginJob.
Using Correlated algorithm to calculate IS.
%MSG-i ISCalcCorrelated:  IonAndScint:IonAndScint@BeginJob  29-Jul-2025 13:20:11 AST ModuleBeginJob ISCalcCorrelated.cxx:40
IonizationAndScintillation/ISCalcCorrelated Initialize.
%MSG
IonAndScint beginJob.
Using Correlated algorithm to calculate IS.
%MSG-i ISCalcCorrelated:  IonAndScint:IonAndScintExternal@BeginJob  29-Jul-2025 13:20:11 AST ModuleBeginJob ISCalcCorrelated.cxx:40
IonizationAndScintillation/ISCalcCorrelated Initialize.
%MSG
Begin processing the 1st record. run: 1 subRun: 0 event: 1 at 29-Jul-2025 13:20:11 AST
%MSG-w PhotonBackTrackerService:  ProcessEvent 29-Jul-2025 13:20:23 AST  run: 1 subRun: 0 event: 1
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-i NuRandomService:  IonAndScint:IonAndScint@BeginModule  29-Jul-2025 13:20:23 AST run: 1 subRun: 0 event: 1
Random seed for this event, engine 'IonAndScint.ISCalcAlg': 583913212
%MSG
IonAndScint Module Producer
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveOuter
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveInner
%MSG-i NuRandomService:  IonAndScint:IonAndScintExternal@BeginModule  29-Jul-2025 13:20:32 AST run: 1 subRun: 0 event: 1
Random seed for this event, engine 'IonAndScintExternal.ISCalcAlg': 196478069
%MSG
IonAndScint Module Producer
Skip SimEnergyDeposit in: priorSCE
Skip SimEnergyDeposit in: 
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveOuter
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveInner
%MSG-i NuRandomService:  PDFastSimPAR:PDFastSim@BeginModule  29-Jul-2025 13:20:32 AST run: 1 subRun: 0 event: 1
Random seed for this event, engine 'PDFastSim.photon': 395373845
%MSG
%MSG-i NuRandomService:  PDFastSimPAR:PDFastSim@BeginModule  29-Jul-2025 13:20:32 AST run: 1 subRun: 0 event: 1
Random seed for this event, engine 'PDFastSim.scinttime': 680292027
%MSG
%MSG-i NuRandomService:  PDFastSimPVS:PDFastSimExternal@BeginModule  29-Jul-2025 14:31:17 AST run: 1 subRun: 0 event: 1
Random seed for this event, engine 'PDFastSimExternal.photon': 860912702
%MSG
%MSG-i NuRandomService:  PDFastSimPVS:PDFastSimExternal@BeginModule  29-Jul-2025 14:31:17 AST run: 1 subRun: 0 event: 1
Random seed for this event, engine 'PDFastSimExternal.scinttime': 243180253
%MSG
%MSG-i PhotonVisibilityService:  PDFastSimPVS:PDFastSimExternal@BeginModule  29-Jul-2025 14:49:06 AST run: 1 subRun: 0 event: 1
PhotonVisibilityService Loading photon library from file /cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash/PhotonPropagation/LibraryData/Photon_library_protoDUNEhd_library_protoDUNEhd_v2_refactored_nonActive.root for 760182 voxels and 160 optical detectors.
%MSG
%MSG-i PhotonLibrary:  PDFastSimPVS:PDFastSimExternal@BeginModule  29-Jul-2025 14:50:22 AST run: 1 subRun: 0 event: 1
Reading photon library from input file: /cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash/PhotonPropagation/LibraryData/Photon_library_protoDUNEhd_library_protoDUNEhd_v2_refactored_nonActive.root
%MSG
29-Jul-2025 15:19:59 AST  Opened output file with pattern "cosmic_g4_stage2_1.root"
%MSG-i FastCloning:  PDFastSimPVS:PDFastSimExternal@BeginModule  29-Jul-2025 15:20:38 AST run: 1 subRun: 0 event: 1
Fast cloning event data products from input file.
%MSG
IonAndScint endJob.
IonAndScint endJob.
29-Jul-2025 15:21:14 AST  Closed input file "cosmic_g4_stage1_1.root"
Malformed TimeTracker database.  The TimeEvent table is empty, but
the TimeModule table is not.  This can happen if an exception has
been thrown from a module while processing the first event.  Any
saved database file is suspect and should not be used.
%MSG-i NuRandomService:  RootOutput:out1@EndJob 29-Jul-2025 15:21:14 AST  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1
   Configured value          Last value   ModuleLabel.InstanceName
        (per event)           583913212   IonAndScint.ISCalcAlg
        (per event)           196478069   IonAndScintExternal.ISCalcAlg
        (per event)           395373845   PDFastSim.photon
        (per event)           680292027   PDFastSim.scinttime
        (per event)           860912702   PDFastSimExternal.photon
        (per event)           243180253   PDFastSimExternal.scinttime

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6236.69 MB
  Peak resident set size usage (VmHWM): 2140.32 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 1 passed = 0 failed = 1

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport          0          0          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 276.621535 Real = 7263.659491

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 6236.69 VmHWM = 2140.32

%MSG-s ArtException:  PostEndJob 29-Jul-2025 18:23:53 AST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- OtherArt BEGIN
      ---- PhotonLibrary BEGIN
        Error in ttree load, reading photon library: /cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash/PhotonPropagation/LibraryData/Photon_library_protoDUNEhd_library_protoDUNEhd_v2_refactored_nonActive.root
        The above exception was thrown while processing module PDFastSimPVS/PDFastSimExternal run: 1 subRun: 0 event: 1
      ---- PhotonLibrary END
    ---- OtherArt END
    Exception going through path simulate
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 0 entries while sim::ParticleAncestryMap_largeant__G4Stage1. has 10 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
G4 Stage 2 done and Stage 1 file removed
total 782M
drwxr-xr-x 3 dunepilot gridusers 4.0K Jul 29 12:24 ..
-rw-r--r-- 1 dunepilot gridusers  28K Jul 29 12:34 time.db
-rw-r--r-- 1 dunepilot gridusers 108K Jul 29 12:34 mem.db
-rw-r--r-- 1 dunepilot gridusers 699K Jul 29 12:34 prod_cosmics_protodunehd_hist.root
-rw-r--r-- 1 dunepilot gridusers  519 Jul 29 13:01 g4_protodunehd_stage1_hist.root
-rw-r--r-- 1 dunepilot gridusers 781M Jul 29 18:21 RootOutput-2fa6-945e-27c1-9849.root
-rw-r--r-- 1 dunepilot gridusers  519 Jul 29 18:23 g4_protodunehd_hist.root
drwxr-xr-x 2 dunepilot gridusers 4.0K Jul 29 18:24 .
-rw-r--r-- 1 dunepilot gridusers 233K Jul 29 18:24 jobscript.log
htgettoken: Failure getting token from https://htvaultprod.fnal.gov:8200

Tue Jul 29 18:24:20 2025
 Notice: Unable to find valid grid or kerberos credentials. Later actions will likely fail.
gfal-copy error: 2 (No such file or directory) - Could not stat the source: errno reported by local system call No such file or directory
Perhaps you forgot a -D to indicate destination is a directory? 
Tue Jul 29 18:24:23 2025
 program: www_cp.sh  /home/workspace/cosmic_g4_stage2_1.root https://fndcadoor.fnal.gov:2880/dune/scratch/users/szh2/MC_pdhd_Michel/MY_production//cosmic_g4_stage2_1.rootexited status 2
delaying 19 ...
htgettoken: Failure getting token from https://htvaultprod.fnal.gov:8200

Tue Jul 29 18:24:42 2025
 Notice: Unable to find valid grid or kerberos credentials. Later actions will likely fail.
Tue Jul 29 18:24:42 2025
 retrying...
gfal-copy error: 2 (No such file or directory) - Could not stat the source: errno reported by local system call No such file or directory
Perhaps you forgot a -D to indicate destination is a directory? 
Final G4 file copied and cleaned up