Jobsub ID 247120.12@dunegpschedd02.fnal.gov
| Jobsub ID | 247120.12@dunegpschedd02.fnal.gov | |
| Workflow ID | 10258 | |
| Stage ID | 1 | |
| User name | epennacc@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-11-16 13:29:58 | |
| Site | UK_Edinburgh | |
| Entry | DUNE_UK_SGridECDF_ce1_multicore | |
| Last heartbeat | 2025-11-16 14:19:36 | |
| From worker node | Hostname | node2b03.ecdf.ed.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 171000 (47 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Started | 2025-11-16 13:32:39 | |
| Input files | fardet-hd:prodgenie_nue_dune10kt_1x2x6_20251003T234008Z_gen_000800_g4_detsim.root fardet-hd:prodgenie_nue_dune10kt_1x2x6_20251004T032816Z_gen_006548_g4_detsim.root fardet-hd:prodgenie_nue_dune10kt_1x2x6_20251004T004819Z_gen_002786_g4_detsim.root fardet-hd:prodgenie_nue_dune10kt_1x2x6_20251004T040428Z_gen_007721_g4_detsim.root | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-11-16 14:19:36 | |
| Saved logs | justin-logs:247120.12-dunegpschedd02.fnal.gov.logs.tgz | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
90@root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/67/23/prodgenie_nue_dune10kt_1x2x6_20251004T032816Z_gen_006548_g4_detsim.root?xrdcl.requuid=a8e6250e-7718-48f9-9fbd-3ab83f289287] Error while opening at ccxrootdegee.in2p3.fr:1094: [ERROR] Server responded with an error: [3012] Failed to open file (Queued transfer cancelled: killed by door: transfer interrupted [666])
[2025-11-16 14:01:30.645140 +0000][Debug ][XRootD ] Redirect trace-back:
[2025-11-16 14:01:30.645140 +0000][Debug ][XRootD ] 0. Retrying: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/67/23/prodgenie_nue_dune10kt_1x2x6_20251004T032816Z_gen_006548_g4_detsim.root
[2025-11-16 14:01:30.645140 +0000][Debug ][XRootD ] 1. Retrying: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/67/23/prodgenie_nue_dune10kt_1x2x6_20251004T032816Z_gen_006548_g4_detsim.root
[2025-11-16 14:01:30.645148 +0000][Debug ][ExDbgMsg ] [ccxrootdegee.in2p3.fr:1094] Destroying MsgHandler: 0xcceb710.
16-Nov-2025 14:01:30 GMT Opened output file with pattern "prodgenie_nue_dune10kt_1x2x6_20251004T032816Z_gen_006548_g4_detsim_20251116T133712Z_reco.root"
====================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
====================================================================================================================
[ No processed events ]
====================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 1450.99 MB
Peak resident set size usage (VmHWM): 718.234 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 16-Nov-2025 14:01:31 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- FileReadError BEGIN
---- FatalRootError BEGIN
Fatal Root Error: TNetXNGFile::ReadBuffer
[ERROR] Server responded with an error: [3012] Failed to open file (Queued transfer cancelled: killed by door: transfer interrupted [666])
ROOT severity: 3000
---- FatalRootError END
The above exception was thrown while processing module TriggerPrimitiveMakerTPC/tpmakerTPCsimpleThr run: 8528 subRun: 0 event: 65471
---- FileReadError END
Exception going through path makers
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TNetXNGFile::TNetXNGFile
The remote file is not open
ROOT severity: 3000
---- FatalRootError END
---- FatalRootError BEGIN
Fatal Root Error: TNetXNGFile::Close
[ERROR] Server responded with an error: [3012] Failed to open file (Queued transfer cancelled: killed by door: transfer interrupted [666])
ROOT severity: 3000
---- FatalRootError END
---- FatalRootError BEGIN
Fatal Root Error: TNetXNGFile::Close
[ERROR] Server responded with an error: [3012] Failed to open file (Queued transfer cancelled: killed by door: transfer interrupted [666])
ROOT severity: 3000
---- FatalRootError END
%MSG
[2025-11-16 14:01:31.077129 +0000][Debug ][JobMgr ] Stopping the job manager...
[2025-11-16 14:01:31.077572 +0000][Debug ][JobMgr ] Job manager stopped
[2025-11-16 14:01:31.077628 +0000][Debug ][TaskMgr ] Stopping the task manager...
[2025-11-16 14:01:31.077725 +0000][Debug ][TaskMgr ] Task manager stopped
[2025-11-16 14:01:31.077730 +0000][Debug ][Poller ] Stopping the poller...
[2025-11-16 14:01:31.077935 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30022.0] Closing the socket
[2025-11-16 14:01:31.077945 +0000][Debug ][PostMaster ] [ccdcacli447.in2p3.fr:30022] Destroying stream
[2025-11-16 14:01:31.077985 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30022.0] Closing the socket
[2025-11-16 14:01:31.078021 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30256.0] Closing the socket
[2025-11-16 14:01:31.078026 +0000][Debug ][PostMaster ] [ccdcacli447.in2p3.fr:30256] Destroying stream
[2025-11-16 14:01:31.078032 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30256.0] Closing the socket
[2025-11-16 14:01:31.078042 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30433.0] Closing the socket
[2025-11-16 14:01:31.078046 +0000][Debug ][PostMaster ] [ccdcacli447.in2p3.fr:30433] Destroying stream
[2025-11-16 14:01:31.078051 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30433.0] Closing the socket
[2025-11-16 14:01:31.078062 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30577.0] Closing the socket
[2025-11-16 14:01:31.078066 +0000][Debug ][PostMaster ] [ccdcacli447.in2p3.fr:30577] Destroying stream
[2025-11-16 14:01:31.078074 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30577.0] Closing the socket
[2025-11-16 14:01:31.078088 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30583.0] Closing the socket
[2025-11-16 14:01:31.078092 +0000][Debug ][PostMaster ] [ccdcacli447.in2p3.fr:30583] Destroying stream
[2025-11-16 14:01:31.078096 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30583.0] Closing the socket
[2025-11-16 14:01:31.078107 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30603.0] Closing the socket
[2025-11-16 14:01:31.078111 +0000][Debug ][PostMaster ] [ccdcacli447.in2p3.fr:30603] Destroying stream
[2025-11-16 14:01:31.078115 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30603.0] Closing the socket
[2025-11-16 14:01:31.078124 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30611.0] Closing the socket
[2025-11-16 14:01:31.078129 +0000][Debug ][PostMaster ] [ccdcacli447.in2p3.fr:30611] Destroying stream
[2025-11-16 14:01:31.078133 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30611.0] Closing the socket
[2025-11-16 14:01:31.078142 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30613.0] Closing the socket
[2025-11-16 14:01:31.078146 +0000][Debug ][PostMaster ] [ccdcacli447.in2p3.fr:30613] Destroying stream
[2025-11-16 14:01:31.078150 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30613.0] Closing the socket
[2025-11-16 14:01:31.078159 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30615.0] Closing the socket
[2025-11-16 14:01:31.078173 +0000][Debug ][PostMaster ] [ccdcacli447.in2p3.fr:30615] Destroying stream
[2025-11-16 14:01:31.078178 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30615.0] Closing the socket
[2025-11-16 14:01:31.078187 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30668.0] Closing the socket
[2025-11-16 14:01:31.078191 +0000][Debug ][PostMaster ] [ccdcacli447.in2p3.fr:30668] Destroying stream
[2025-11-16 14:01:31.078195 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30668.0] Closing the socket
[2025-11-16 14:01:31.078203 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30674.0] Closing the socket
[2025-11-16 14:01:31.078206 +0000][Debug ][PostMaster ] [ccdcacli447.in2p3.fr:30674] Destroying stream
[2025-11-16 14:01:31.078211 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30674.0] Closing the socket
[2025-11-16 14:01:31.078220 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30780.0] Closing the socket
[2025-11-16 14:01:31.078224 +0000][Debug ][PostMaster ] [ccdcacli447.in2p3.fr:30780] Destroying stream
[2025-11-16 14:01:31.078229 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30780.0] Closing the socket
[2025-11-16 14:01:31.078237 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30785.0] Closing the socket
[2025-11-16 14:01:31.078241 +0000][Debug ][PostMaster ] [ccdcacli447.in2p3.fr:30785] Destroying stream
[2025-11-16 14:01:31.078245 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30785.0] Closing the socket
[2025-11-16 14:01:31.078253 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30799.0] Closing the socket
[2025-11-16 14:01:31.078257 +0000][Debug ][PostMaster ] [ccdcacli447.in2p3.fr:30799] Destroying stream
[2025-11-16 14:01:31.078260 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30799.0] Closing the socket
[2025-11-16 14:01:31.078268 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30851.0] Closing the socket
[2025-11-16 14:01:31.078272 +0000][Debug ][PostMaster ] [ccdcacli447.in2p3.fr:30851] Destroying stream
[2025-11-16 14:01:31.078275 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30851.0] Closing the socket
[2025-11-16 14:01:31.078284 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30988.0] Closing the socket
[2025-11-16 14:01:31.078287 +0000][Debug ][PostMaster ] [ccdcacli447.in2p3.fr:30988] Destroying stream
[2025-11-16 14:01:31.078290 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30988.0] Closing the socket
[2025-11-16 14:01:31.078300 +0000][Debug ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] Closing the socket
[2025-11-16 14:01:31.078305 +0000][Debug ][Poller ] <[::ffff:192.41.105.36]:41738><--><[::ffff:134.158.209.218]:1094> Removing socket from the poller
[2025-11-16 14:01:31.078428 +0000][Debug ][PostMaster ] [ccxrootdegee.in2p3.fr:1094] Destroying stream
[2025-11-16 14:01:31.078435 +0000][Debug ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] Closing the socket
Art has completed and will exit with status 1.