justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 18682.3@dunegpschedd01.fnal.gov

Jobsub ID	18682.3@dunegpschedd01.fnal.gov
Workflow ID	101
Stage ID	1
User name	szh2@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-28 20:58:31
Site	ES_CIEMAT
Entry	DUNE_T2_ES_CIEMAT_condorce2
Last heartbeat	2025-07-28 22:00:01
From worker node	Hostname	gaew0197.ciemat.es
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	232200 (64 hours)
	GPU
	Inner Apptainer?	True
Job state	none_processed
Started	2025-07-28 20:59:33
Input files
Outputting started	2025-07-28 21:59:30
Output files
Finished	2025-07-28 22:00:01
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ule  28-Jul-2025 23:32:04 CEST run: 1 subRun: 0 event: 5
Random seed for this event, engine 'PDFastSim.scinttime': 713240767
%MSG
%MSG-i NuRandomService:  PDFastSimPVS:PDFastSimExternal@BeginModule  28-Jul-2025 23:34:24 CEST run: 1 subRun: 0 event: 5
Random seed for this event, engine 'PDFastSimExternal.photon': 674408777
%MSG
%MSG-i NuRandomService:  PDFastSimPVS:PDFastSimExternal@BeginModule  28-Jul-2025 23:34:24 CEST run: 1 subRun: 0 event: 5
Random seed for this event, engine 'PDFastSimExternal.scinttime': 765202530
%MSG
Begin processing the 6th record. run: 1 subRun: 0 event: 6 at 28-Jul-2025 23:35:35 CEST
%MSG-w PhotonBackTrackerService:  ProcessEvent 28-Jul-2025 23:35:40 CEST  run: 1 subRun: 0 event: 6
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-i NuRandomService:  IonAndScint:IonAndScint@BeginModule  28-Jul-2025 23:35:40 CEST run: 1 subRun: 0 event: 6
Random seed for this event, engine 'IonAndScint.ISCalcAlg': 486501985
%MSG
IonAndScint Module Producer
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveOuter
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveInner
%MSG-i NuRandomService:  IonAndScint:IonAndScintExternal@BeginModule  28-Jul-2025 23:35:42 CEST run: 1 subRun: 0 event: 6
Random seed for this event, engine 'IonAndScintExternal.ISCalcAlg': 621215543
%MSG
IonAndScint Module Producer
Skip SimEnergyDeposit in: priorSCE
Skip SimEnergyDeposit in: 
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveOuter
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveInner
%MSG-i NuRandomService:  PDFastSimPAR:PDFastSim@BeginModule  28-Jul-2025 23:35:42 CEST run: 1 subRun: 0 event: 6
Random seed for this event, engine 'PDFastSim.photon': 554634175
%MSG
%MSG-i NuRandomService:  PDFastSimPAR:PDFastSim@BeginModule  28-Jul-2025 23:35:42 CEST run: 1 subRun: 0 event: 6
Random seed for this event, engine 'PDFastSim.scinttime': 585530388
%MSG
%MSG-i NuRandomService:  PDFastSimPVS:PDFastSimExternal@BeginModule  28-Jul-2025 23:38:19 CEST run: 1 subRun: 0 event: 6
Random seed for this event, engine 'PDFastSimExternal.photon': 350480698
%MSG
%MSG-i NuRandomService:  PDFastSimPVS:PDFastSimExternal@BeginModule  28-Jul-2025 23:38:19 CEST run: 1 subRun: 0 event: 6
Random seed for this event, engine 'PDFastSimExternal.scinttime': 567675144
%MSG
Begin processing the 7th record. run: 1 subRun: 0 event: 7 at 28-Jul-2025 23:39:22 CEST
%MSG-w PhotonBackTrackerService:  ProcessEvent 28-Jul-2025 23:39:29 CEST  run: 1 subRun: 0 event: 7
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-i NuRandomService:  IonAndScint:IonAndScint@BeginModule  28-Jul-2025 23:39:29 CEST run: 1 subRun: 0 event: 7
Random seed for this event, engine 'IonAndScint.ISCalcAlg': 630867021
%MSG
IonAndScint Module Producer
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveOuter
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveInner
%MSG-i NuRandomService:  IonAndScint:IonAndScintExternal@BeginModule  28-Jul-2025 23:39:33 CEST run: 1 subRun: 0 event: 7
Random seed for this event, engine 'IonAndScintExternal.ISCalcAlg': 111366195
%MSG
IonAndScint Module Producer
Skip SimEnergyDeposit in: priorSCE
Skip SimEnergyDeposit in: 
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveOuter
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveInner
%MSG-i NuRandomService:  PDFastSimPAR:PDFastSim@BeginModule  28-Jul-2025 23:39:33 CEST run: 1 subRun: 0 event: 7
Random seed for this event, engine 'PDFastSim.photon': 227224016
%MSG
%MSG-i NuRandomService:  PDFastSimPAR:PDFastSim@BeginModule  28-Jul-2025 23:39:33 CEST run: 1 subRun: 0 event: 7
Random seed for this event, engine 'PDFastSim.scinttime': 157695292
%MSG
%MSG-i NuRandomService:  PDFastSimPVS:PDFastSimExternal@BeginModule  28-Jul-2025 23:42:27 CEST run: 1 subRun: 0 event: 7
Random seed for this event, engine 'PDFastSimExternal.photon': 93001723
%MSG
%MSG-i NuRandomService:  PDFastSimPVS:PDFastSimExternal@BeginModule  28-Jul-2025 23:42:27 CEST run: 1 subRun: 0 event: 7
Random seed for this event, engine 'PDFastSimExternal.scinttime': 716097818
%MSG
Begin processing the 8th record. run: 1 subRun: 0 event: 8 at 28-Jul-2025 23:43:46 CEST
%MSG-w PhotonBackTrackerService:  ProcessEvent 28-Jul-2025 23:43:51 CEST  run: 1 subRun: 0 event: 8
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-i NuRandomService:  IonAndScint:IonAndScint@BeginModule  28-Jul-2025 23:43:51 CEST run: 1 subRun: 0 event: 8
Random seed for this event, engine 'IonAndScint.ISCalcAlg': 193567300
%MSG
IonAndScint Module Producer
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveOuter
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveInner
%MSG-i NuRandomService:  IonAndScint:IonAndScintExternal@BeginModule  28-Jul-2025 23:43:53 CEST run: 1 subRun: 0 event: 8
Random seed for this event, engine 'IonAndScintExternal.ISCalcAlg': 509278098
%MSG
IonAndScint Module Producer
Skip SimEnergyDeposit in: priorSCE
Skip SimEnergyDeposit in: 
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveOuter
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveInner
%MSG-i NuRandomService:  PDFastSimPAR:PDFastSim@BeginModule  28-Jul-2025 23:43:53 CEST run: 1 subRun: 0 event: 8
Random seed for this event, engine 'PDFastSim.photon': 195398312
%MSG
%MSG-i NuRandomService:  PDFastSimPAR:PDFastSim@BeginModule  28-Jul-2025 23:43:53 CEST run: 1 subRun: 0 event: 8
Random seed for this event, engine 'PDFastSim.scinttime': 831716217
%MSG
%MSG-i NuRandomService:  PDFastSimPVS:PDFastSimExternal@BeginModule  28-Jul-2025 23:45:39 CEST run: 1 subRun: 0 event: 8
Random seed for this event, engine 'PDFastSimExternal.photon': 255311634
%MSG
%MSG-i NuRandomService:  PDFastSimPVS:PDFastSimExternal@BeginModule  28-Jul-2025 23:45:39 CEST run: 1 subRun: 0 event: 8
Random seed for this event, engine 'PDFastSimExternal.scinttime': 841473957
%MSG
Begin processing the 9th record. run: 1 subRun: 0 event: 9 at 28-Jul-2025 23:46:25 CEST
%MSG-w PhotonBackTrackerService:  ProcessEvent 28-Jul-2025 23:46:36 CEST  run: 1 subRun: 0 event: 9
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-i NuRandomService:  IonAndScint:IonAndScint@BeginModule  28-Jul-2025 23:46:36 CEST run: 1 subRun: 0 event: 9
Random seed for this event, engine 'IonAndScint.ISCalcAlg': 187036427
%MSG
IonAndScint Module Producer
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveOuter
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveInner
%MSG-i NuRandomService:  IonAndScint:IonAndScintExternal@BeginModule  28-Jul-2025 23:46:49 CEST run: 1 subRun: 0 event: 9
Random seed for this event, engine 'IonAndScintExternal.ISCalcAlg': 358724145
%MSG
IonAndScint Module Producer
Skip SimEnergyDeposit in: priorSCE
Skip SimEnergyDeposit in: 
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveOuter
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveInner
%MSG-i NuRandomService:  PDFastSimPAR:PDFastSim@BeginModule  28-Jul-2025 23:46:49 CEST run: 1 subRun: 0 event: 9
Random seed for this event, engine 'PDFastSim.photon': 337981697
%MSG
%MSG-i NuRandomService:  PDFastSimPAR:PDFastSim@BeginModule  28-Jul-2025 23:46:49 CEST run: 1 subRun: 0 event: 9
Random seed for this event, engine 'PDFastSim.scinttime': 636590603
%MSG
%MSG-i NuRandomService:  PDFastSimPVS:PDFastSimExternal@BeginModule  28-Jul-2025 23:59:00 CEST run: 1 subRun: 0 event: 9
Random seed for this event, engine 'PDFastSimExternal.photon': 140225043
%MSG
%MSG-i NuRandomService:  PDFastSimPVS:PDFastSimExternal@BeginModule  28-Jul-2025 23:59:00 CEST run: 1 subRun: 0 event: 9
Random seed for this event, engine 'PDFastSimExternal.scinttime': 96101666
%MSG
../justin-jobscript: line 35:  1278 Segmentation fault      (core dumped) lar -c standard_g4_protodunehd_stage2.fcl -s cosmic_g4_stage1_${jobid}.root -o cosmic_g4_stage2_${jobid}.root
G4 Stage 2 done and Stage 1 file removed
total 4.9G
drwxr-xr-x 3 dune001 dune 4.0K Jul 28 22:59 ..
-rw-r--r-- 1 dune001 dune  28K Jul 28 23:01 time.db
-rw-r--r-- 1 dune001 dune 108K Jul 28 23:01 mem.db
-rw-r--r-- 1 dune001 dune 700K Jul 28 23:01 prod_cosmics_protodunehd_hist.root
-rw-r--r-- 1 dune001 dune  519 Jul 28 23:08 g4_protodunehd_stage1_hist.root
-rw-r--r-- 1 dune001 dune  274 Jul 28 23:08 TFileService-66c3-b869-d199-2ffe.root
-rw-r--r-- 1 dune001 dune 4.9G Jul 28 23:59 RootOutput-1d00-4146-61b5-e9ac.root
drwxr-xr-x 2 dune001 dune 4.0K Jul 28 23:59 .
-rw-r--r-- 1 dune001 dune 246K Jul 28 23:59 jobscript.log
htgettoken: Failure getting token from https://htvaultprod.fnal.gov:8200

Mon Jul 28 23:59:21 2025
 Notice: Unable to find valid grid or kerberos credentials. Later actions will likely fail.
gfal-copy error: 2 (No such file or directory) - Could not stat the source: errno reported by local system call No such file or directory
Perhaps you forgot a -D to indicate destination is a directory? 
Mon Jul 28 23:59:23 2025
 program: www_cp.sh  /home/workspace/cosmic_g4_stage2_1.root https://fndcadoor.fnal.gov:2880/dune/scratch/users/szh2/MC_pdhd_Michel/MY_production//cosmic_g4_stage2_1.rootexited status 2
delaying 4 ...
htgettoken: Failure getting token from https://htvaultprod.fnal.gov:8200

Mon Jul 28 23:59:28 2025
 Notice: Unable to find valid grid or kerberos credentials. Later actions will likely fail.
Mon Jul 28 23:59:28 2025
 retrying...
gfal-copy error: 2 (No such file or directory) - Could not stat the source: errno reported by local system call No such file or directory
Perhaps you forgot a -D to indicate destination is a directory? 
Final G4 file copied and cleaned up

justIN time: 2025-08-05 00:56:39 UTC       justIN version: 01.04.00