Jobsub ID 252803.65@dunegpschedd01.fnal.gov
| Jobsub ID | 252803.65@dunegpschedd01.fnal.gov | |
| Workflow ID | 10258 | |
| Stage ID | 1 | |
| User name | epennacc@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-11-16 13:21:57 | |
| Site | UK_Edinburgh | |
| Entry | DUNE_UK_SGridECDF_ce1_multicore | |
| Last heartbeat | 2025-11-16 14:03:10 | |
| From worker node | Hostname | node2b01.ecdf.ed.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 171000 (47 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Started | 2025-11-16 13:45:01 | |
| Input files | fardet-hd:prodgenie_nue_dune10kt_1x2x6_20251003T225919Z_gen_000234_g4_detsim.root fardet-hd:prodgenie_nue_dune10kt_1x2x6_20251004T025019Z_gen_006132_g4_detsim.root fardet-hd:prodgenie_nue_dune10kt_1x2x6_20251004T032859Z_gen_006586_g4_detsim.root fardet-hd:prodgenie_nue_dune10kt_1x2x6_20251003T234557Z_gen_001160_g4_detsim.root | |
| Jobscript | Exit code | 20 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-11-16 14:03:10 | |
| Saved logs | justin-logs:252803.65-dunegpschedd01.fnal.gov.logs.tgz | |
| List job events (HTCondor job logs unavailable) | ||
Jobscript log (last 10,000 characters)
ket [2025-11-16 13:56:50.633031 +0000][Debug ][Poller ] <[::ffff:192.41.105.34]:41264><--><[::ffff:134.158.209.218]:1094> Removing socket from the poller [2025-11-16 13:56:50.633122 +0000][Debug ][PostMaster ] [ccxrootdegee.in2p3.fr:1094] Recovering error for stream #0: [ERROR] Socket timeout. [2025-11-16 13:56:50.633129 +0000][Debug ][PostMaster ] [ccxrootdegee.in2p3.fr:1094] Reporting disconnection to queued message handlers. [2025-11-16 13:56:50.633136 +0000][Debug ][XRootD ] [ccxrootdegee.in2p3.fr:1094] Handling error while processing kXR_open (file: pnfs/in2p3.fr/data/dune/disk/fardet-hd/50/c9/prodgenie_nue_dune10kt_1x2x6_20251004T025019Z_gen_006132_g4_detsim.root?tried=ccxrootdegee.in2p3.fr,ccxrootdegee.in2p3.fr,ccxrootdegee.in2p3.fr&triedrc=srverr,srverr,srverr, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [ERROR] Socket timeout. [2025-11-16 13:56:50.633144 +0000][Info ][XRootD ] [ccxrootdegee.in2p3.fr:1094] Retrying request: kXR_open (file: pnfs/in2p3.fr/data/dune/disk/fardet-hd/50/c9/prodgenie_nue_dune10kt_1x2x6_20251004T025019Z_gen_006132_g4_detsim.root?tried=ccxrootdegee.in2p3.fr,ccxrootdegee.in2p3.fr,ccxrootdegee.in2p3.fr&triedrc=srverr,srverr,srverr, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ). [2025-11-16 13:56:50.633205 +0000][Debug ][ExDbgMsg ] [ccxrootdegee.in2p3.fr:1094] Retry at server MsgHandler: 0x8f19700 (message: kXR_open (file: pnfs/in2p3.fr/data/dune/disk/fardet-hd/50/c9/prodgenie_nue_dune10kt_1x2x6_20251004T025019Z_gen_006132_g4_detsim.root?tried=ccxrootdegee.in2p3.fr,ccxrootdegee.in2p3.fr,ccxrootdegee.in2p3.fr,ccxrootdegee.in2p3.fr&triedrc=srverr,srverr,srverr,srverr, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ). [2025-11-16 13:56:50.633834 +0000][Debug ][PostMaster ] [ccxrootdegee.in2p3.fr:1094] Found 1 address(es): [::ffff:134.158.209.218]:1094 [2025-11-16 13:56:50.633853 +0000][Debug ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] Attempting connection to [::ffff:134.158.209.218]:1094 [2025-11-16 13:56:50.633875 +0000][Debug ][Poller ] Adding socket 0x959e900 to the poller [2025-11-16 13:56:50.661260 +0000][Debug ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] Async connection call returned [2025-11-16 13:56:50.661292 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Sending out the initial hand shake + kXR_protocol [2025-11-16 13:56:51.159167 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Got the server hand shake response (type: manager [], protocol version 500) [2025-11-16 13:56:51.159208 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] kXR_protocol successful (type: manager [], protocol version 500) [2025-11-16 13:56:51.159827 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Sending out kXR_login request, username: gl05pi8, cgi: xrd.cc=uk&xrd.tz=0&xrd.appname=lar&xrd.info=&xrd.hostname=node2b01.ecdf.ed.ac.uk&xrd.rn=v5.5.5, dual-stack: false, private IPv4: false, private IPv6: false [2025-11-16 13:56:51.159839 +0000][Debug ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] TLS hand-shake exchange. [2025-11-16 13:56:51.226042 +0000][Debug ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] TLS hand-shake exchange. [2025-11-16 13:56:51.254177 +0000][Debug ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] TLS hand-shake exchange. [2025-11-16 13:56:51.254265 +0000][Info ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] TLS hand-shake done. [2025-11-16 13:56:52.624866 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Logged in, session: a4b35d1a79c321776fdbd79873140dbc [2025-11-16 13:56:52.624905 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Authentication is required: &P=ztn,0:4096:&P=gsi,v:10400,c:ssl,ca:5a04724c [2025-11-16 13:56:52.624910 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Sending authentication data [2025-11-16 13:56:52.624934 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Trying to authenticate using ztn [2025-11-16 13:56:52.624949 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Cannot get credentials for protocol ztn: Secztn: No token found; runtime fetch disallowed. [2025-11-16 13:56:52.624971 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Trying to authenticate using gsi [2025-11-16 13:56:52.749078 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Sending more authentication data for gsi [2025-11-16 13:56:54.320445 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Authenticated with gsi. [2025-11-16 13:56:54.320475 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Sending out kXR_endsess for session: ece7f168876879d8676250c872e49e21 [2025-11-16 13:56:54.347786 +0000][Debug ][PostMaster ] [ccxrootdegee.in2p3.fr:1094] Stream 0 connected (IPv4). [2025-11-16 13:56:54.347834 +0000][Debug ][ExDbgMsg ] [ccxrootdegee.in2p3.fr:1094] Moving MsgHandler: 0x8f19700 (message: kXR_open (file: pnfs/in2p3.fr/data/dune/disk/fardet-hd/50/c9/prodgenie_nue_dune10kt_1x2x6_20251004T025019Z_gen_006132_g4_detsim.root?tried=ccxrootdegee.in2p3.fr,ccxrootdegee.in2p3.fr,ccxrootdegee.in2p3.fr,ccxrootdegee.in2p3.fr&triedrc=srverr,srverr,srverr,srverr, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) from out-queu to in-queue. [2025-11-16 13:56:55.902767 +0000][Debug ][ExDbgMsg ] [msg: 0x6eacfd0] Assigned MsgHandler: 0x8f19700. [2025-11-16 13:56:55.902795 +0000][Debug ][ExDbgMsg ] [handler: 0x8f19700] Removed MsgHandler: 0x8f19700 from the in-queue. [2025-11-16 13:56:55.903133 +0000][Debug ][XRootD ] [ccxrootdegee.in2p3.fr:1094] Handling error while processing kXR_open (file: pnfs/in2p3.fr/data/dune/disk/fardet-hd/50/c9/prodgenie_nue_dune10kt_1x2x6_20251004T025019Z_gen_006132_g4_detsim.root?tried=ccxrootdegee.in2p3.fr,ccxrootdegee.in2p3.fr,ccxrootdegee.in2p3.fr,ccxrootdegee.in2p3.fr&triedrc=srverr,srverr,srverr,srverr, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [ERROR] Error response: bad address. [2025-11-16 13:56:55.903291 +0000][Debug ][ExDbgMsg ] [ccxrootdegee.in2p3.fr:1094] Calling MsgHandler: 0x8f19700 (message: kXR_open (file: pnfs/in2p3.fr/data/dune/disk/fardet-hd/50/c9/prodgenie_nue_dune10kt_1x2x6_20251004T025019Z_gen_006132_g4_detsim.root?tried=ccxrootdegee.in2p3.fr,ccxrootdegee.in2p3.fr,ccxrootdegee.in2p3.fr,ccxrootdegee.in2p3.fr&triedrc=srverr,srverr,srverr,srverr, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) with status: [ERROR] Error response: bad address. [2025-11-16 13:56:55.903449 +0000][Debug ][File ] [0x652e4f0@root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/50/c9/prodgenie_nue_dune10kt_1x2x6_20251004T025019Z_gen_006132_g4_detsim.root?xrdcl.requuid=50228bfb-9738-4c1d-9d77-b40c9081022b] Open has returned with status [ERROR] Server responded with an error: [3012] Internal timeout [2025-11-16 13:56:55.903482 +0000][Debug ][File ] [0x652e4f0@root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/50/c9/prodgenie_nue_dune10kt_1x2x6_20251004T025019Z_gen_006132_g4_detsim.root?xrdcl.requuid=50228bfb-9738-4c1d-9d77-b40c9081022b] Error while opening at ccxrootdegee.in2p3.fr:1094: [ERROR] Server responded with an error: [3012] Internal timeout [2025-11-16 13:56:55.903626 +0000][Debug ][XRootD ] Redirect trace-back: [2025-11-16 13:56:55.903626 +0000][Debug ][XRootD ] 0. Retrying: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/50/c9/prodgenie_nue_dune10kt_1x2x6_20251004T025019Z_gen_006132_g4_detsim.root [2025-11-16 13:56:55.903626 +0000][Debug ][XRootD ] 1. Retrying: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/50/c9/prodgenie_nue_dune10kt_1x2x6_20251004T025019Z_gen_006132_g4_detsim.root [2025-11-16 13:56:55.903626 +0000][Debug ][XRootD ] 2. Retrying: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/50/c9/prodgenie_nue_dune10kt_1x2x6_20251004T025019Z_gen_006132_g4_detsim.root [2025-11-16 13:56:55.903626 +0000][Debug ][XRootD ] 3. Retrying: root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/50/c9/prodgenie_nue_dune10kt_1x2x6_20251004T025019Z_gen_006132_g4_detsim.root [2025-11-16 13:56:55.903635 +0000][Debug ][ExDbgMsg ] [ccxrootdegee.in2p3.fr:1094] Destroying MsgHandler: 0x8f19700. %MSG-s ArtException: TriggerResultInserter:TriggerResults@Construction 16-Nov-2025 13:56:55 GMT ModuleConstruction cet::exception caught in art ---- FileOpenError BEGIN RootInputFileSequence::initFile(): Input file root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/50/c9/prodgenie_nue_dune10kt_1x2x6_20251004T025019Z_gen_006132_g4_detsim.root was not found or could not be opened. ---- FileOpenError END %MSG [2025-11-16 13:56:56.029358 +0000][Debug ][JobMgr ] Stopping the job manager... [2025-11-16 13:56:56.029639 +0000][Debug ][JobMgr ] Job manager stopped [2025-11-16 13:56:56.029697 +0000][Debug ][TaskMgr ] Stopping the task manager... [2025-11-16 13:56:56.029761 +0000][Debug ][TaskMgr ] Task manager stopped [2025-11-16 13:56:56.029765 +0000][Debug ][Poller ] Stopping the poller... [2025-11-16 13:56:56.029991 +0000][Debug ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] Closing the socket [2025-11-16 13:56:56.030276 +0000][Debug ][Poller ] <[::ffff:192.41.105.34]:49956><--><[::ffff:134.158.209.218]:1094> Removing socket from the poller [2025-11-16 13:56:56.030406 +0000][Debug ][PostMaster ] [ccxrootdegee.in2p3.fr:1094] Destroying stream [2025-11-16 13:56:56.030447 +0000][Debug ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] Closing the socket Art has completed and will exit with status 20.