Jobsub ID 247234.15@dunegpschedd02.fnal.gov
| Jobsub ID | 247234.15@dunegpschedd02.fnal.gov | |
| Workflow ID | 10258 | |
| Stage ID | 1 | |
| User name | epennacc@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-11-16 21:09:18 | |
| Site | UK_Edinburgh | |
| Entry | DUNE_UK_SGridECDF_ce1_multicore | |
| Last heartbeat | 2025-11-16 21:23:19 | |
| From worker node | Hostname | node2b07.ecdf.ed.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 171000 (47 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Started | 2025-11-16 21:10:35 | |
| Input files | fardet-hd:prodgenie_nue_dune10kt_1x2x6_20251003T234543Z_gen_001138_g4_detsim.root fardet-hd:prodgenie_nue_dune10kt_1x2x6_20251004T044945Z_gen_009191_g4_detsim.root fardet-hd:prodgenie_nue_dune10kt_1x2x6_20251004T012045Z_gen_003537_g4_detsim.root fardet-hd:prodgenie_nue_dune10kt_1x2x6_20251004T023348Z_gen_005605_g4_detsim.root | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-11-16 21:23:19 | |
| Saved logs | justin-logs:247234.15-dunegpschedd02.fnal.gov.logs.tgz | |
| List job events (HTCondor job logs unavailable) | ||
Jobscript log (last 10,000 characters)
01.050382 +0000][Debug ][ExDbgMsg ] [ccdcacli447.in2p3.fr:30122] Passing to the thread-pool MsgHandler: 0xa19b0d0 (message: kXR_read (handle: 0x00000000, offset: 41568009, size: 179530197) ).
[2025-11-16 21:23:01.050508 +0000][Debug ][ExDbgMsg ] [ccdcacli447.in2p3.fr:30122] Calling MsgHandler: 0xa19b0d0 (message: kXR_read (handle: 0x00000000, offset: 41568009, size: 179530197) ) with status: [ERROR] Socket error.
[2025-11-16 21:23:01.050539 +0000][Debug ][File ] [0x8549e80@root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/90/96/prodgenie_nue_dune10kt_1x2x6_20251004T044945Z_gen_009191_g4_detsim.root?xrdcl.requuid=f6d242f4-5292-44d4-9b83-41f6b7362f97] Running the recovery procedure
[2025-11-16 21:23:01.050566 +0000][Debug ][ExDbgMsg ] [ccxrootdegee.in2p3.fr:1094] MsgHandler created: 0xa194200 (message: kXR_open (file: pnfs/in2p3.fr/data/dune/disk/fardet-hd/90/96/prodgenie_nue_dune10kt_1x2x6_20251004T044945Z_gen_009191_g4_detsim.root, mode: 00, flags: kXR_open_read ) ).
[2025-11-16 21:23:01.050590 +0000][Debug ][ExDbgMsg ] [ccdcacli447.in2p3.fr:30122] Destroying MsgHandler: 0xa19b0d0.
[2025-11-16 21:23:01.050629 +0000][Debug ][ExDbgMsg ] [ccxrootdegee.in2p3.fr:1094] Moving MsgHandler: 0xa194200 (message: kXR_open (file: pnfs/in2p3.fr/data/dune/disk/fardet-hd/90/96/prodgenie_nue_dune10kt_1x2x6_20251004T044945Z_gen_009191_g4_detsim.root, mode: 00, flags: kXR_open_read ) ) from out-queu to in-queue.
[2025-11-16 21:23:01.123611 +0000][Debug ][ExDbgMsg ] [msg: 0x9a6fc20] Assigned MsgHandler: 0xa194200.
[2025-11-16 21:23:01.123656 +0000][Debug ][ExDbgMsg ] [handler: 0xa194200] Removed MsgHandler: 0xa194200 from the in-queue.
[2025-11-16 21:23:01.123794 +0000][Debug ][XRootD ] [ccxrootdegee.in2p3.fr:1094] Handling error while processing kXR_open (file: pnfs/in2p3.fr/data/dune/disk/fardet-hd/90/96/prodgenie_nue_dune10kt_1x2x6_20251004T044945Z_gen_009191_g4_detsim.root, mode: 00, flags: kXR_open_read ): [ERROR] Error response: bad address.
[2025-11-16 21:23:01.123868 +0000][Debug ][ExDbgMsg ] [ccxrootdegee.in2p3.fr:1094] Calling MsgHandler: 0xa194200 (message: kXR_open (file: pnfs/in2p3.fr/data/dune/disk/fardet-hd/90/96/prodgenie_nue_dune10kt_1x2x6_20251004T044945Z_gen_009191_g4_detsim.root, mode: 00, flags: kXR_open_read ) ) with status: [ERROR] Error response: bad address.
[2025-11-16 21:23:01.123939 +0000][Debug ][File ] [0x8549e80@root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/90/96/prodgenie_nue_dune10kt_1x2x6_20251004T044945Z_gen_009191_g4_detsim.root?xrdcl.requuid=f6d242f4-5292-44d4-9b83-41f6b7362f97] Open has returned with status [ERROR] Server responded with an error: [3012] Internal timeout
[2025-11-16 21:23:01.123947 +0000][Debug ][File ] [0x8549e80@root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/90/96/prodgenie_nue_dune10kt_1x2x6_20251004T044945Z_gen_009191_g4_detsim.root?xrdcl.requuid=f6d242f4-5292-44d4-9b83-41f6b7362f97] Error while opening at ccxrootdegee.in2p3.fr:1094: [ERROR] Server responded with an error: [3012] Internal timeout
[2025-11-16 21:23:01.124022 +0000][Debug ][ExDbgMsg ] [ccxrootdegee.in2p3.fr:1094] Destroying MsgHandler: 0xa194200.
16-Nov-2025 21:23:01 GMT Opened output file with pattern "prodgenie_nue_dune10kt_1x2x6_20251004T044945Z_gen_009191_g4_detsim_20251116T211439Z_reco.root"
====================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
====================================================================================================================
[ No processed events ]
====================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 1399.96 MB
Peak resident set size usage (VmHWM): 668.627 MB
Details saved in: 'mem.db'
====================================================================================================
%MSG-s ArtException: PostEndJob 16-Nov-2025 21:23:01 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- ScheduleExecutionFailure BEGIN
Path: ProcessingStopped.
---- FileReadError BEGIN
---- FatalRootError BEGIN
Fatal Root Error: TNetXNGFile::ReadBuffer
[ERROR] Server responded with an error: [3012] Internal timeout
ROOT severity: 3000
---- FatalRootError END
The above exception was thrown while processing module TriggerPrimitiveMakerTPC/tpmakerTPCsimpleThr run: 8528 subRun: 0 event: 91901
---- FileReadError END
Exception going through path makers
---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TNetXNGFile::TNetXNGFile
The remote file is not open
ROOT severity: 3000
---- FatalRootError END
---- FatalRootError BEGIN
Fatal Root Error: TNetXNGFile::Close
[ERROR] Server responded with an error: [3012] Internal timeout
ROOT severity: 3000
---- FatalRootError END
---- FatalRootError BEGIN
Fatal Root Error: TNetXNGFile::Close
[ERROR] Server responded with an error: [3012] Internal timeout
ROOT severity: 3000
---- FatalRootError END
%MSG
[2025-11-16 21:23:01.513508 +0000][Debug ][JobMgr ] Stopping the job manager...
[2025-11-16 21:23:01.514073 +0000][Debug ][JobMgr ] Job manager stopped
[2025-11-16 21:23:01.514123 +0000][Debug ][TaskMgr ] Stopping the task manager...
[2025-11-16 21:23:01.514197 +0000][Debug ][TaskMgr ] Task manager stopped
[2025-11-16 21:23:01.514202 +0000][Debug ][Poller ] Stopping the poller...
[2025-11-16 21:23:01.514407 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30053.0] Closing the socket
[2025-11-16 21:23:01.514417 +0000][Debug ][PostMaster ] [ccdcacli447.in2p3.fr:30053] Destroying stream
[2025-11-16 21:23:01.514444 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30053.0] Closing the socket
[2025-11-16 21:23:01.514481 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30054.0] Closing the socket
[2025-11-16 21:23:01.514486 +0000][Debug ][PostMaster ] [ccdcacli447.in2p3.fr:30054] Destroying stream
[2025-11-16 21:23:01.514491 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30054.0] Closing the socket
[2025-11-16 21:23:01.514500 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30085.0] Closing the socket
[2025-11-16 21:23:01.514505 +0000][Debug ][PostMaster ] [ccdcacli447.in2p3.fr:30085] Destroying stream
[2025-11-16 21:23:01.514510 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30085.0] Closing the socket
[2025-11-16 21:23:01.514519 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30122.0] Closing the socket
[2025-11-16 21:23:01.514533 +0000][Debug ][PostMaster ] [ccdcacli447.in2p3.fr:30122] Destroying stream
[2025-11-16 21:23:01.514537 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30122.0] Closing the socket
[2025-11-16 21:23:01.514546 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30127.0] Closing the socket
[2025-11-16 21:23:01.514550 +0000][Debug ][PostMaster ] [ccdcacli447.in2p3.fr:30127] Destroying stream
[2025-11-16 21:23:01.514554 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30127.0] Closing the socket
[2025-11-16 21:23:01.514563 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30167.0] Closing the socket
[2025-11-16 21:23:01.514567 +0000][Debug ][PostMaster ] [ccdcacli447.in2p3.fr:30167] Destroying stream
[2025-11-16 21:23:01.514570 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30167.0] Closing the socket
[2025-11-16 21:23:01.514582 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30242.0] Closing the socket
[2025-11-16 21:23:01.514586 +0000][Debug ][PostMaster ] [ccdcacli447.in2p3.fr:30242] Destroying stream
[2025-11-16 21:23:01.514590 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30242.0] Closing the socket
[2025-11-16 21:23:01.514598 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30356.0] Closing the socket
[2025-11-16 21:23:01.514602 +0000][Debug ][PostMaster ] [ccdcacli447.in2p3.fr:30356] Destroying stream
[2025-11-16 21:23:01.514606 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30356.0] Closing the socket
[2025-11-16 21:23:01.514618 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30432.0] Closing the socket
[2025-11-16 21:23:01.514622 +0000][Debug ][PostMaster ] [ccdcacli447.in2p3.fr:30432] Destroying stream
[2025-11-16 21:23:01.514625 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30432.0] Closing the socket
[2025-11-16 21:23:01.514640 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30580.0] Closing the socket
[2025-11-16 21:23:01.514643 +0000][Debug ][PostMaster ] [ccdcacli447.in2p3.fr:30580] Destroying stream
[2025-11-16 21:23:01.514646 +0000][Debug ][AsyncSock ] [ccdcacli447.in2p3.fr:30580.0] Closing the socket
[2025-11-16 21:23:01.514654 +0000][Debug ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] Closing the socket
[2025-11-16 21:23:01.514659 +0000][Debug ][Poller ] <[::ffff:192.41.105.40]:57812><--><[::ffff:134.158.209.218]:1094> Removing socket from the poller
[2025-11-16 21:23:01.514860 +0000][Debug ][PostMaster ] [ccxrootdegee.in2p3.fr:1094] Destroying stream
[2025-11-16 21:23:01.514868 +0000][Debug ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] Closing the socket
Art has completed and will exit with status 1.