Jobsub ID 18739.1@dunegpschedd01.fnal.gov
| Jobsub ID | 18739.1@dunegpschedd01.fnal.gov |
| Workflow ID | 103 |
| Stage ID | 1 |
| User name | szh2@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 2097152000 (2000 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-07-28 23:20:37 |
| Site | ES_CIEMAT |
| Entry | DUNE_T2_ES_CIEMAT_condorce2 |
| Last heartbeat | 2025-07-28 23:43:01 |
| From worker node | Hostname | gaew0210.ciemat.es |
| cpuinfo | Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 2097152000 (2000 MiB) |
| Wall seconds limit | 232200 (64 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | none_processed |
| Started | 2025-07-28 23:21:06 |
| Input files | |
| Outputting started | 2025-07-28 23:42:35 |
| Output files | |
| Finished | 2025-07-28 23:43:01 |
| List job events Cached HTCondor job logs |
Jobscript log (last 10,000 characters)
hoton lookup table size : 760182 voxels, 160 channels (no voxel geometry included)
%MSG
29-Jul-2025 01:34:33 CEST Opened output file with pattern "cosmic_g4_stage2_1.root"
%MSG-i FastCloning: PostProcessEvent 29-Jul-2025 01:34:34 CEST run: 1 subRun: 0 event: 1
Fast cloning event data products from input file.
%MSG
Begin processing the 2nd record. run: 1 subRun: 0 event: 2 at 29-Jul-2025 01:35:17 CEST
%MSG-w PhotonBackTrackerService: ProcessEvent 29-Jul-2025 01:35:21 CEST run: 1 subRun: 0 event: 2
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-i NuRandomService: IonAndScint:IonAndScint@BeginModule 29-Jul-2025 01:35:21 CEST run: 1 subRun: 0 event: 2
Random seed for this event, engine 'IonAndScint.ISCalcAlg': 834706960
%MSG
IonAndScint Module Producer
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveOuter
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveInner
%MSG-i NuRandomService: IonAndScint:IonAndScintExternal@BeginModule 29-Jul-2025 01:35:22 CEST run: 1 subRun: 0 event: 2
Random seed for this event, engine 'IonAndScintExternal.ISCalcAlg': 664857204
%MSG
IonAndScint Module Producer
Skip SimEnergyDeposit in: priorSCE
Skip SimEnergyDeposit in:
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveOuter
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveInner
%MSG-i NuRandomService: PDFastSimPAR:PDFastSim@BeginModule 29-Jul-2025 01:35:22 CEST run: 1 subRun: 0 event: 2
Random seed for this event, engine 'PDFastSim.photon': 627335545
%MSG
%MSG-i NuRandomService: PDFastSimPAR:PDFastSim@BeginModule 29-Jul-2025 01:35:22 CEST run: 1 subRun: 0 event: 2
Random seed for this event, engine 'PDFastSim.scinttime': 381017845
%MSG
%MSG-i NuRandomService: PDFastSimPVS:PDFastSimExternal@BeginModule 29-Jul-2025 01:37:06 CEST run: 1 subRun: 0 event: 2
Random seed for this event, engine 'PDFastSimExternal.photon': 303663725
%MSG
%MSG-i NuRandomService: PDFastSimPVS:PDFastSimExternal@BeginModule 29-Jul-2025 01:37:06 CEST run: 1 subRun: 0 event: 2
Random seed for this event, engine 'PDFastSimExternal.scinttime': 34603550
%MSG
Begin processing the 3rd record. run: 1 subRun: 0 event: 3 at 29-Jul-2025 01:37:49 CEST
%MSG-w PhotonBackTrackerService: ProcessEvent 29-Jul-2025 01:37:54 CEST run: 1 subRun: 0 event: 3
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-i NuRandomService: IonAndScint:IonAndScint@BeginModule 29-Jul-2025 01:37:54 CEST run: 1 subRun: 0 event: 3
Random seed for this event, engine 'IonAndScint.ISCalcAlg': 721180328
%MSG
IonAndScint Module Producer
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveOuter
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveInner
%MSG-i NuRandomService: IonAndScint:IonAndScintExternal@BeginModule 29-Jul-2025 01:37:56 CEST run: 1 subRun: 0 event: 3
Random seed for this event, engine 'IonAndScintExternal.ISCalcAlg': 351406081
%MSG
IonAndScint Module Producer
Skip SimEnergyDeposit in: priorSCE
Skip SimEnergyDeposit in:
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveOuter
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveInner
%MSG-i NuRandomService: PDFastSimPAR:PDFastSim@BeginModule 29-Jul-2025 01:37:56 CEST run: 1 subRun: 0 event: 3
Random seed for this event, engine 'PDFastSim.photon': 672557656
%MSG
%MSG-i NuRandomService: PDFastSimPAR:PDFastSim@BeginModule 29-Jul-2025 01:37:56 CEST run: 1 subRun: 0 event: 3
Random seed for this event, engine 'PDFastSim.scinttime': 757330915
%MSG
%MSG-i NuRandomService: PDFastSimPVS:PDFastSimExternal@BeginModule 29-Jul-2025 01:41:02 CEST run: 1 subRun: 0 event: 3
Random seed for this event, engine 'PDFastSimExternal.photon': 448025547
%MSG
%MSG-i NuRandomService: PDFastSimPVS:PDFastSimExternal@BeginModule 29-Jul-2025 01:41:02 CEST run: 1 subRun: 0 event: 3
Random seed for this event, engine 'PDFastSimExternal.scinttime': 482635066
%MSG
IonAndScint endJob.
IonAndScint endJob.
29-Jul-2025 01:41:23 CEST Closed input file "cosmic_g4_stage1_1.root"
================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
================================================================================================================================
Full event 138.172 162.217 195.256 153.222 24.1566 3
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000561585 0.00743971 0.015731 0.00602651 0.006273 3
simulate:rns:RandomNumberSaver 3.509e-05 0.000103147 0.000238227 3.6123e-05 9.55172e-05 3
simulate:IonAndScint:IonAndScint 1.35297 1.91486 2.60145 1.79016 0.517263 3
simulate:IonAndScintExternal:IonAndScint 0.00175741 0.00333862 0.00423292 0.00402553 0.00112129 3
simulate:PDFastSim:PDFastSimPAR 104.035 132.04 186.241 105.844 38.3332 3
simulate:PDFastSimExternal:PDFastSimPVS 3.13435 7.82883 13.9505 6.40168 4.52951 3
[art]:TriggerResults:TriggerResultInserter 1.1545e-05 3.09993e-05 6.9715e-05 1.1738e-05 2.73762e-05 3
end_path:out1:RootOutput 2.396e-06 8.05167e-06 1.9273e-05 2.486e-06 7.93477e-06 3
end_path:out1:RootOutput(write) 29.6315 30.6312 31.6309 30.6312 0.999707 2
================================================================================================================================
%MSG-i NuRandomService: RootOutput:out1@EndJob 29-Jul-2025 01:41:23 CEST ModuleEndJob
Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
algorithm version: EventTimestamp_v1
Configured value Last value ModuleLabel.InstanceName
(per event) 721180328 IonAndScint.ISCalcAlg
(per event) 351406081 IonAndScintExternal.ISCalcAlg
(per event) 672557656 PDFastSim.photon
(per event) 757330915 PDFastSim.scinttime
(per event) 448025547 PDFastSimExternal.photon
(per event) 482635066 PDFastSimExternal.scinttime
%MSG
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 13472.2 MB
Peak resident set size usage (VmHWM): 12232.6 MB
====================================================================================================
TrigReport ---------- Event summary -------------
TrigReport Events total = 3 passed = 3 failed = 0
TrigReport ---------- Modules in End-path ----------
TrigReport Run Success Error Name
TrigReport 3 3 0 out1
TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 525.575293 Real = 536.094008
MemReport ---------- Memory summary [base-10 MB] ------
MemReport VmPeak = 13472.2 VmHWM = 12232.6
%MSG-s ArtException: PostEndJob 29-Jul-2025 01:42:13 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 2 entries while sim::ParticleAncestryMap_largeant__G4Stage1. has 10 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
G4 Stage 2 done and Stage 1 file removed
total 2.5G
drwxr-xr-x 3 dune001 dune 4.0K Jul 29 01:21 ..
-rw-r--r-- 1 dune001 dune 28K Jul 29 01:22 time.db
-rw-r--r-- 1 dune001 dune 108K Jul 29 01:22 mem.db
-rw-r--r-- 1 dune001 dune 699K Jul 29 01:22 prod_cosmics_protodunehd_hist.root
-rw-r--r-- 1 dune001 dune 519 Jul 29 01:26 g4_protodunehd_stage1_hist.root
-rw-r--r-- 1 dune001 dune 2.5G Jul 29 01:42 RootOutput-cc5a-e13b-0c9a-9d44.root
-rw-r--r-- 1 dune001 dune 519 Jul 29 01:42 g4_protodunehd_hist.root
drwxr-xr-x 2 dune001 dune 4.0K Jul 29 01:42 .
-rw-r--r-- 1 dune001 dune 239K Jul 29 01:42 jobscript.log
htgettoken: Failure getting token from https://htvaultprod.fnal.gov:8200
Tue Jul 29 01:42:14 2025
Notice: Unable to find valid grid or kerberos credentials. Later actions will likely fail.
gfal-copy error: 2 (No such file or directory) - Could not stat the source: errno reported by local system call No such file or directory
Perhaps you forgot a -D to indicate destination is a directory?
Tue Jul 29 01:42:16 2025
program: www_cp.sh /home/workspace/cosmic_g4_stage2_1.root https://fndcadoor.fnal.gov:2880/dune/scratch/users/szh2/MC_pdhd_Michel/MY_production//cosmic_g4_stage2_1.rootexited status 2
delaying 16 ...
htgettoken: Failure getting token from https://htvaultprod.fnal.gov:8200
Tue Jul 29 01:42:32 2025
Notice: Unable to find valid grid or kerberos credentials. Later actions will likely fail.
Tue Jul 29 01:42:32 2025
retrying...
gfal-copy error: 2 (No such file or directory) - Could not stat the source: errno reported by local system call No such file or directory
Perhaps you forgot a -D to indicate destination is a directory?
Final G4 file copied and cleaned up 