justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 231877.9@dunegpschedd01.fnal.gov

Jobsub ID231877.9@dunegpschedd01.fnal.gov
Workflow ID8763
Stage ID1
User namemdias@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-08 20:52:02
SiteUS_UCSD
EntryCMSHTPC_T2_US_UCSD_gw7
Last heartbeat2025-10-08 21:13:51
From worker nodeHostnamesdsc-73.t2.ucsd.edu
cpuinfoIntel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statestalled
Started2025-10-08 20:53:50
Input filesehn1-beam-np04:H4_v34b_-1GeV_-27.7_000562_420285_71_20250623T161810.root
Outputting started2025-10-08 21:07:30
Output files
Finished2025-10-08 21:52:08
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ucer
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveOuter
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveInner
IonAndScint Module Producer
Skip SimEnergyDeposit in: 
Skip SimEnergyDeposit in: priorSCE
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveOuter
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveInner
Begin processing the 4th record. run: 20250623 subRun: 1 event: 13 at 08-Oct-2025 14:05:51 PDT
%MSG-w PhotonBackTrackerService:  ProcessEvent 08-Oct-2025 14:05:55 PDT  run: 20250623 subRun: 1 event: 13
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
IonAndScint Module Producer
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveOuter
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveInner
IonAndScint Module Producer
Skip SimEnergyDeposit in: 
Skip SimEnergyDeposit in: priorSCE
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveOuter
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveInner
Begin processing the 5th record. run: 20250623 subRun: 1 event: 14 at 08-Oct-2025 14:06:04 PDT
%MSG-w PhotonBackTrackerService:  ProcessEvent 08-Oct-2025 14:06:11 PDT  run: 20250623 subRun: 1 event: 14
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
IonAndScint Module Producer
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveOuter
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveInner
IonAndScint Module Producer
Skip SimEnergyDeposit in: 
Skip SimEnergyDeposit in: priorSCE
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveOuter
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveInner
Begin processing the 6th record. run: 20250623 subRun: 1 event: 15 at 08-Oct-2025 14:06:20 PDT
%MSG-w PhotonBackTrackerService:  ProcessEvent 08-Oct-2025 14:06:25 PDT  run: 20250623 subRun: 1 event: 15
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
IonAndScint Module Producer
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveOuter
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveInner
IonAndScint Module Producer
Skip SimEnergyDeposit in: 
Skip SimEnergyDeposit in: priorSCE
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveOuter
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveInner
Begin processing the 7th record. run: 20250623 subRun: 1 event: 16 at 08-Oct-2025 14:06:31 PDT
%MSG-w PhotonBackTrackerService:  ProcessEvent 08-Oct-2025 14:06:35 PDT  run: 20250623 subRun: 1 event: 16
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
IonAndScint Module Producer
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveOuter
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveInner
IonAndScint Module Producer
Skip SimEnergyDeposit in: 
Skip SimEnergyDeposit in: priorSCE
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveOuter
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveInner
Begin processing the 8th record. run: 20250623 subRun: 1 event: 17 at 08-Oct-2025 14:06:42 PDT
%MSG-w PhotonBackTrackerService:  ProcessEvent 08-Oct-2025 14:06:46 PDT  run: 20250623 subRun: 1 event: 17
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
IonAndScint Module Producer
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveOuter
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveInner
IonAndScint Module Producer
Skip SimEnergyDeposit in: 
Skip SimEnergyDeposit in: priorSCE
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveOuter
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveInner
Begin processing the 9th record. run: 20250623 subRun: 1 event: 18 at 08-Oct-2025 14:06:55 PDT
%MSG-w PhotonBackTrackerService:  ProcessEvent 08-Oct-2025 14:06:59 PDT  run: 20250623 subRun: 1 event: 18
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
IonAndScint Module Producer
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveOuter
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveInner
IonAndScint Module Producer
Skip SimEnergyDeposit in: 
Skip SimEnergyDeposit in: priorSCE
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveOuter
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveInner
Begin processing the 10th record. run: 20250623 subRun: 1 event: 19 at 08-Oct-2025 14:07:10 PDT
%MSG-w PhotonBackTrackerService:  ProcessEvent 08-Oct-2025 14:07:16 PDT  run: 20250623 subRun: 1 event: 19
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
IonAndScint Module Producer
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveOuter
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveInner
IonAndScint Module Producer
Skip SimEnergyDeposit in: 
Skip SimEnergyDeposit in: priorSCE
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveOuter
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveInner
08-Oct-2025 14:07:29 PDT  Closed output file "pdhd_prod_beam__231877_9_1_20251008T205354Z_gen_g4_IonScintPDExt.root"
08-Oct-2025 14:07:29 PDT  Closed input file "temp_largeant.root"
IonAndScint endJob.
IonAndScint endJob.

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      6.35247       9.19302       12.9121       9.08969       1.93292        10     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000220199   0.000402336   0.000659463   0.000407197   0.000125083      10     
simulate:rns:RandomNumberSaver                 4.037e-05    6.91751e-05   0.000300338   4.24185e-05   7.71487e-05      10     
simulate:IonAndScint:IonAndScint                1.26241       1.61477       2.1061        1.59584      0.266313        10     
simulate:IonAndScintExternal:IonAndScint      0.00135468    0.00305249    0.00722882    0.00264594    0.00164211       10     
simulate:PDFastSimExternal:PDFastSimPVS        0.102363      0.957769       6.37594      0.333299       1.81767        10     
[art]:TriggerResults:TriggerResultInserter    1.4932e-05    1.86499e-05   4.6296e-05    1.57435e-05   9.22256e-06      10     
end_path:out1:RootOutput                       3.237e-06    5.2543e-06    2.0581e-05     3.521e-06    5.11495e-06      10     
end_path:out1:RootOutput(write)                 4.89964       6.61655       8.94663       6.52194       1.3408         10     
================================================================================================================================
%MSG-i NuRandomService:  RootOutput:out1@EndJob 08-Oct-2025 14:07:29 PDT  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'random'
  master seed: 659259793
  seed within: [ 1 ; 900000000 ]
   Configured value          Last value   ModuleLabel.InstanceName
          655603039              (same)   IonAndScint.ISCalcAlg
          322118157              (same)   IonAndScintExternal.ISCalcAlg
           33582209              (same)   PDFastSimExternal.photon
          701720922              (same)   PDFastSimExternal.scinttime

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5317.71 MB
  Peak resident set size usage (VmHWM): 4444.61 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 10 passed = 10 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         10         10          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 140.878446 Real = 152.094364

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 5317.71 VmHWM = 4444.61

Art has completed and will exit with status 0.
ionscint pdext lar returns 0
==== Start *.json ====
cat: *.json: No such file or directory
==== End *.json ====
.:
total 3158960
-rw-r--r--. 1 cuser cuser     248480 Oct  8 14:07 jobscript.log
-rw-r--r--. 1 cuser cuser        149 Oct  8 14:07 justin-processed-pfns.txt
-rw-r--r--. 1 cuser cuser        519 Oct  8 14:07 g4_protodunehd_hist.root
-rw-r--r--. 1 cuser cuser 1924646412 Oct  8 14:07 pdhd_prod_beam__231877_9_1_20251008T205354Z_gen_g4_IonScintPDExt.root
-rw-r--r--. 1 cuser cuser        519 Oct  8 14:02 g4_protodunehd_stage1_hist.root
-rw-r--r--. 1 cuser cuser 1304855862 Oct  8 14:02 temp_largeant.root
-rw-r--r--. 1 cuser cuser     716836 Oct  8 13:56 prod_cosmics_beam_protodunehd_hist.root
-rw-r--r--. 1 cuser cuser     118784 Oct  8 13:56 mem.db
-rw-r--r--. 1 cuser cuser      28672 Oct  8 13:56 time.db
-rw-r--r--. 1 cuser cuser    4052320 Oct  8 13:56 temp_gen.root
-rw-r--r--. 1 cuser cuser         73 Oct  8 13:53 all-input-dids.txt
justIN time: 2025-11-05 08:42:15 UTC       justIN version: 01.05.01