justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 239288.109@dunegpschedd02.fnal.gov

Jobsub ID239288.109@dunegpschedd02.fnal.gov
Workflow ID9846
Stage ID1
User namemdias@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-07 03:47:24
SiteUS_PuertoRico
EntryCMSHTPC_T3_US_PuertoRico_UPRM
Last heartbeat2025-11-07 05:01:03
From worker nodeHostnamecompute-0-5.local
cpuinfoIntel(R) Xeon(R) CPU W3550 @ 3.07GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateaborted
Started2025-11-07 04:23:32
Input filesehn1-beam-np04:H4_v34b_-1GeV_-27.7_001904_227928_75_20250623T163807.root
Outputting started2025-11-07 04:40:55
Output files
Finished2025-11-07 05:01:03
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

g on a generation or simulation step.
%MSG
IonAndScint Module Producer
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveOuter
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveInner
IonAndScint Module Producer
Skip SimEnergyDeposit in: 
Skip SimEnergyDeposit in: priorSCE
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveOuter
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveInner
Begin processing the 4th record. run: 20250623 subRun: 1 event: 43 at 07-Nov-2025 00:38:49 AST
%MSG-w PhotonBackTrackerService:  ProcessEvent 07-Nov-2025 00:38:54 AST  run: 20250623 subRun: 1 event: 43
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
IonAndScint Module Producer
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveOuter
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveInner
IonAndScint Module Producer
Skip SimEnergyDeposit in: 
Skip SimEnergyDeposit in: priorSCE
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveOuter
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveInner
Begin processing the 5th record. run: 20250623 subRun: 1 event: 44 at 07-Nov-2025 00:39:04 AST
%MSG-w PhotonBackTrackerService:  ProcessEvent 07-Nov-2025 00:39:09 AST  run: 20250623 subRun: 1 event: 44
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
IonAndScint Module Producer
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveOuter
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveInner
IonAndScint Module Producer
Skip SimEnergyDeposit in: 
Skip SimEnergyDeposit in: priorSCE
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveOuter
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveInner
Begin processing the 6th record. run: 20250623 subRun: 1 event: 45 at 07-Nov-2025 00:39:23 AST
%MSG-w PhotonBackTrackerService:  ProcessEvent 07-Nov-2025 00:39:28 AST  run: 20250623 subRun: 1 event: 45
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
IonAndScint Module Producer
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveOuter
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveInner
IonAndScint Module Producer
Skip SimEnergyDeposit in: 
Skip SimEnergyDeposit in: priorSCE
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveOuter
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveInner
Begin processing the 7th record. run: 20250623 subRun: 1 event: 46 at 07-Nov-2025 00:39:38 AST
%MSG-w PhotonBackTrackerService:  ProcessEvent 07-Nov-2025 00:39:43 AST  run: 20250623 subRun: 1 event: 46
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
IonAndScint Module Producer
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveOuter
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveInner
IonAndScint Module Producer
Skip SimEnergyDeposit in: 
Skip SimEnergyDeposit in: priorSCE
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveOuter
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveInner
Begin processing the 8th record. run: 20250623 subRun: 1 event: 47 at 07-Nov-2025 00:39:56 AST
%MSG-w PhotonBackTrackerService:  ProcessEvent 07-Nov-2025 00:40:00 AST  run: 20250623 subRun: 1 event: 47
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
IonAndScint Module Producer
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveOuter
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveInner
IonAndScint Module Producer
Skip SimEnergyDeposit in: 
Skip SimEnergyDeposit in: priorSCE
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveOuter
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveInner
Begin processing the 9th record. run: 20250623 subRun: 1 event: 48 at 07-Nov-2025 00:40:12 AST
%MSG-w PhotonBackTrackerService:  ProcessEvent 07-Nov-2025 00:40:18 AST  run: 20250623 subRun: 1 event: 48
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
IonAndScint Module Producer
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveOuter
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveInner
IonAndScint Module Producer
Skip SimEnergyDeposit in: 
Skip SimEnergyDeposit in: priorSCE
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveOuter
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveInner
Begin processing the 10th record. run: 20250623 subRun: 1 event: 49 at 07-Nov-2025 00:40:28 AST
%MSG-w PhotonBackTrackerService:  ProcessEvent 07-Nov-2025 00:40:34 AST  run: 20250623 subRun: 1 event: 49
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
IonAndScint Module Producer
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveOuter
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveInner
IonAndScint Module Producer
Skip SimEnergyDeposit in: 
Skip SimEnergyDeposit in: priorSCE
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveOuter
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveInner
07-Nov-2025 00:40:52 AST  Closed output file "pdhd_prod_beam__239288_109_1_20251107T042337Z_gen_g4_IonScintPDExt.root"
07-Nov-2025 00:40:52 AST  Closed input file "temp_largeant.root"
IonAndScint endJob.
IonAndScint endJob.

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      9.59475       12.0155       21.7605       10.6444       3.51836        10     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000212064    0.0185889     0.183182     0.000231887    0.0548646       10     
simulate:rns:RandomNumberSaver                 3.773e-05    6.4191e-05    0.000261784   4.04705e-05   6.60617e-05      10     
simulate:IonAndScint:IonAndScint                1.78547       2.74104       4.29969       2.56485      0.904565        10     
simulate:IonAndScintExternal:IonAndScint      0.00260212    0.00404775    0.00988834    0.00320965    0.00211392       10     
simulate:PDFastSimExternal:PDFastSimPVS        0.227717       1.23939       9.44168      0.330185       2.73539        10     
[art]:TriggerResults:TriggerResultInserter    1.5447e-05    2.13583e-05   4.5599e-05    1.87645e-05   8.63102e-06      10     
end_path:out1:RootOutput                       2.906e-06    4.6955e-06    1.5579e-05     3.224e-06    3.69756e-06      10     
end_path:out1:RootOutput(write)                 6.15608       8.0119        10.3567       7.76362       1.34824        10     
================================================================================================================================
%MSG-i NuRandomService:  RootOutput:out1@EndJob 07-Nov-2025 00:40:52 AST  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'random'
  master seed: 883080079
  seed within: [ 1 ; 900000000 ]
   Configured value          Last value   ModuleLabel.InstanceName
          333701668              (same)   IonAndScint.ISCalcAlg
          370389936              (same)   IonAndScintExternal.ISCalcAlg
          662295119              (same)   PDFastSimExternal.photon
          786322632              (same)   PDFastSimExternal.scinttime

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5267.66 MB
  Peak resident set size usage (VmHWM): 4127.92 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 10 passed = 10 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         10         10          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 164.792603 Real = 188.373682

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 5267.66 VmHWM = 4127.92

Art has completed and will exit with status 0.
ionscint pdext lar returns 0
==== Start *.json ====
cat: *.json: No such file or directory
==== End *.json ====
.:
total 1854152
-rw-r--r-- 1 dunepilot gridusers     247762 Nov  7 00:40 jobscript.log
-rw-r--r-- 1 dunepilot gridusers        152 Nov  7 00:40 justin-processed-pfns.txt
-rw-r--r-- 1 dunepilot gridusers        519 Nov  7 00:40 g4_protodunehd_hist.root
-rw-r--r-- 1 dunepilot gridusers 1897505540 Nov  7 00:40 pdhd_prod_beam__239288_109_1_20251107T042337Z_gen_g4_IonScintPDExt.root
-rw-r--r-- 1 dunepilot gridusers        519 Nov  7 00:33 g4_protodunehd_stage1_hist.root
-rw-r--r-- 1 dunepilot gridusers     714882 Nov  7 00:26 prod_cosmics_beam_protodunehd_hist.root
-rw-r--r-- 1 dunepilot gridusers     118784 Nov  7 00:26 mem.db
-rw-r--r-- 1 dunepilot gridusers      28672 Nov  7 00:26 time.db
-rw-r--r-- 1 dunepilot gridusers        146 Nov  7 00:23 all-input-dids.txt
justIN time: 2026-02-04 07:45:47 UTC       justIN version: 01.06.00