Jobsub ID 252776.57@dunegpschedd01.fnal.gov
| Jobsub ID | 252776.57@dunegpschedd01.fnal.gov | |
| Workflow ID | 10257 | |
| Stage ID | 1 | |
| User name | epennacc@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-11-16 12:11:54 | |
| Site | UK_Edinburgh | |
| Entry | DUNE_UK_SGridECDF_ce1_multicore | |
| Last heartbeat | 2025-11-16 12:21:31 | |
| From worker node | Hostname | node2b08.ecdf.ed.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 171000 (47 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Started | 2025-11-16 12:12:29 | |
| Input files | fardet-hd:prodgenie_nu_dune10kt_1x2x6_20251003T205920Z_gen_006464_g4_detsim.root fardet-hd:prodgenie_nu_dune10kt_1x2x6_20251003T202835Z_gen_004564_g4_detsim.root fardet-hd:prodgenie_nu_dune10kt_1x2x6_20251003T223658Z_gen_005853_g4_detsim.root fardet-hd:prodgenie_nu_dune10kt_1x2x6_20251003T195723Z_gen_002568_g4_detsim.root | |
| Jobscript | Exit code | 20 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-11-16 12:21:31 | |
| Saved logs | justin-logs:252776.57-dunegpschedd01.fnal.gov.logs.tgz | |
| List job events (HTCondor job logs unavailable) | ||
Jobscript log (last 10,000 characters)
000][Debug ][PlugInMgr ] Trying to disable plug-in for '*' [2025-11-16 12:21:12.272415 +0000][Debug ][PlugInMgr ] Processing plug-in definitions in /home/.xrootd/client.plugins.d... [2025-11-16 12:21:12.272425 +0000][Debug ][PlugInMgr ] Unable to process directory /home/.xrootd/client.plugins.d: [ERROR] OS Error: no such file or directory [2025-11-16 12:21:12.385179 +0000][Debug ][Utility ] Env: overriding entry: connectionwindow=120 with 30 [2025-11-16 12:21:12.385228 +0000][Debug ][Utility ] Env: overriding entry: requesttimeout=1800 with 4096 [2025-11-16 12:21:12.385237 +0000][Debug ][Utility ] Env: overriding entry: requesttimeout=4096 with 14400 [2025-11-16 12:21:12.385277 +0000][Debug ][Utility ] Env: overriding entry: redirectlimit=16 with 64 [2025-11-16 12:21:12.385296 +0000][Debug ][Utility ] Env: overriding entry: multiprotocol=0 with 1 [2025-11-16 12:21:12.385707 +0000][Debug ][File ] [0x83dc500@root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/97/a0/prodgenie_nu_dune10kt_1x2x6_20251003T223658Z_gen_005853_g4_detsim.root?xrdcl.requuid=e06f2826-925f-469f-833a-34905a460132] Sending an open command [2025-11-16 12:21:12.386117 +0000][Debug ][Utility ] Env: trying to get a non-existent string entry: pollerpreference [2025-11-16 12:21:12.386190 +0000][Debug ][Poller ] Available pollers: built-in [2025-11-16 12:21:12.386194 +0000][Debug ][Poller ] Attempting to create a poller according to preference: built-in [2025-11-16 12:21:12.386199 +0000][Debug ][Poller ] Creating poller: built-in [2025-11-16 12:21:12.386263 +0000][Debug ][Poller ] Creating and starting the built-in poller... [2025-11-16 12:21:12.386865 +0000][Debug ][Poller ] Using 1 poller threads [2025-11-16 12:21:12.386901 +0000][Debug ][TaskMgr ] Starting the task manager... [2025-11-16 12:21:12.386966 +0000][Debug ][TaskMgr ] Task manager started [2025-11-16 12:21:12.386992 +0000][Debug ][JobMgr ] Starting the job manager... [2025-11-16 12:21:12.387108 +0000][Debug ][JobMgr ] Job manager started, 3 workers [2025-11-16 12:21:12.387183 +0000][Debug ][TaskMgr ] Registering task: "FileTimer task" to be run at: [2025-11-16 12:21:12 +0000] [2025-11-16 12:21:12.387431 +0000][Debug ][ExDbgMsg ] [ccxrootdegee.in2p3.fr:1094] MsgHandler created: 0x8d346a0 (message: kXR_open (file: pnfs/in2p3.fr/data/dune/disk/fardet-hd/97/a0/prodgenie_nu_dune10kt_1x2x6_20251003T223658Z_gen_005853_g4_detsim.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ). [2025-11-16 12:21:12.387769 +0000][Debug ][PostMaster ] Creating new channel to: root://ccxrootdegee.in2p3.fr:1094/ [2025-11-16 12:21:12.388063 +0000][Debug ][PostMaster ] [ccxrootdegee.in2p3.fr:1094] Stream parameters: Network Stack: IPAuto, Connection Window: 30, ConnectionRetry: 5, Stream Error Window: 1800 [2025-11-16 12:21:12.388169 +0000][Debug ][TaskMgr ] Registering task: "TickGeneratorTask for: root://ccxrootdegee.in2p3.fr:1094/" to be run at: [2025-11-16 12:21:27 +0000] [2025-11-16 12:21:12.389127 +0000][Debug ][PostMaster ] [ccxrootdegee.in2p3.fr:1094] Found 1 address(es): [::ffff:134.158.209.218]:1094 [2025-11-16 12:21:12.389201 +0000][Debug ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] Attempting connection to [::ffff:134.158.209.218]:1094 [2025-11-16 12:21:12.389269 +0000][Debug ][Poller ] Adding socket 0x649b270 to the poller [2025-11-16 12:21:12.418204 +0000][Debug ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] Async connection call returned [2025-11-16 12:21:12.418479 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Sending out the initial hand shake + kXR_protocol [2025-11-16 12:21:12.447152 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Got the server hand shake response (type: manager [], protocol version 500) [2025-11-16 12:21:12.450470 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] kXR_protocol successful (type: manager [], protocol version 500) [2025-11-16 12:21:12.451349 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Sending out kXR_login request, username: gl05pi8, cgi: xrd.cc=uk&xrd.tz=0&xrd.appname=lar&xrd.info=&xrd.hostname=node2b08.ecdf.ed.ac.uk&xrd.rn=v5.5.5, dual-stack: false, private IPv4: false, private IPv6: false [2025-11-16 12:21:12.451391 +0000][Debug ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] TLS hand-shake exchange. [2025-11-16 12:21:12.492720 +0000][Debug ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] TLS hand-shake exchange. [2025-11-16 12:21:12.523132 +0000][Debug ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] TLS hand-shake exchange. [2025-11-16 12:21:12.523464 +0000][Info ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] TLS hand-shake done. [2025-11-16 12:21:12.552168 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Logged in, session: 143db2a5edb5c96fac5f28c2f4a6fb7b [2025-11-16 12:21:12.552214 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Authentication is required: &P=ztn,0:4096:&P=gsi,v:10400,c:ssl,ca:5a04724c [2025-11-16 12:21:12.552246 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Sending authentication data [2025-11-16 12:21:12.562899 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Trying to authenticate using ztn [2025-11-16 12:21:12.562945 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Cannot get credentials for protocol ztn: Secztn: No token found; runtime fetch disallowed. [2025-11-16 12:21:12.576094 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Trying to authenticate using gsi [2025-11-16 12:21:13.188304 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Sending more authentication data for gsi [2025-11-16 12:21:13.251790 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Authenticated with gsi. [2025-11-16 12:21:13.251924 +0000][Debug ][PostMaster ] [ccxrootdegee.in2p3.fr:1094] Stream 0 connected (IPv4). [2025-11-16 12:21:13.251978 +0000][Debug ][Utility ] Monitor library name not set. No monitoring [2025-11-16 12:21:13.252258 +0000][Debug ][ExDbgMsg ] [ccxrootdegee.in2p3.fr:1094] Moving MsgHandler: 0x8d346a0 (message: kXR_open (file: pnfs/in2p3.fr/data/dune/disk/fardet-hd/97/a0/prodgenie_nu_dune10kt_1x2x6_20251003T223658Z_gen_005853_g4_detsim.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) from out-queu to in-queue. [2025-11-16 12:21:13.348743 +0000][Debug ][ExDbgMsg ] [msg: 0x80442f0] Assigned MsgHandler: 0x8d346a0. [2025-11-16 12:21:13.348791 +0000][Debug ][ExDbgMsg ] [handler: 0x8d346a0] Removed MsgHandler: 0x8d346a0 from the in-queue. [2025-11-16 12:21:13.349146 +0000][Debug ][XRootD ] [ccxrootdegee.in2p3.fr:1094] Handling error while processing kXR_open (file: pnfs/in2p3.fr/data/dune/disk/fardet-hd/97/a0/prodgenie_nu_dune10kt_1x2x6_20251003T223658Z_gen_005853_g4_detsim.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [ERROR] Error response: bad address. [2025-11-16 12:21:13.349332 +0000][Debug ][ExDbgMsg ] [ccxrootdegee.in2p3.fr:1094] Calling MsgHandler: 0x8d346a0 (message: kXR_open (file: pnfs/in2p3.fr/data/dune/disk/fardet-hd/97/a0/prodgenie_nu_dune10kt_1x2x6_20251003T223658Z_gen_005853_g4_detsim.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) with status: [ERROR] Error response: bad address. [2025-11-16 12:21:13.349527 +0000][Debug ][File ] [0x83dc500@root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/97/a0/prodgenie_nu_dune10kt_1x2x6_20251003T223658Z_gen_005853_g4_detsim.root?xrdcl.requuid=e06f2826-925f-469f-833a-34905a460132] Open has returned with status [ERROR] Server responded with an error: [3012] Internal timeout [2025-11-16 12:21:13.349564 +0000][Debug ][File ] [0x83dc500@root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/97/a0/prodgenie_nu_dune10kt_1x2x6_20251003T223658Z_gen_005853_g4_detsim.root?xrdcl.requuid=e06f2826-925f-469f-833a-34905a460132] Error while opening at ccxrootdegee.in2p3.fr:1094: [ERROR] Server responded with an error: [3012] Internal timeout [2025-11-16 12:21:13.349674 +0000][Debug ][ExDbgMsg ] [ccxrootdegee.in2p3.fr:1094] Destroying MsgHandler: 0x8d346a0. %MSG-s ArtException: TriggerResultInserter:TriggerResults@Construction 16-Nov-2025 12:21:13 GMT ModuleConstruction cet::exception caught in art ---- FileOpenError BEGIN RootInputFileSequence::initFile(): Input file root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/97/a0/prodgenie_nu_dune10kt_1x2x6_20251003T223658Z_gen_005853_g4_detsim.root was not found or could not be opened. ---- FileOpenError END %MSG [2025-11-16 12:21:13.491361 +0000][Debug ][JobMgr ] Stopping the job manager... [2025-11-16 12:21:13.492007 +0000][Debug ][JobMgr ] Job manager stopped [2025-11-16 12:21:13.492083 +0000][Debug ][TaskMgr ] Stopping the task manager... [2025-11-16 12:21:13.492163 +0000][Debug ][TaskMgr ] Task manager stopped [2025-11-16 12:21:13.492169 +0000][Debug ][Poller ] Stopping the poller... [2025-11-16 12:21:13.492474 +0000][Debug ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] Closing the socket [2025-11-16 12:21:13.492875 +0000][Debug ][Poller ] <[::ffff:192.41.105.41]:43002><--><[::ffff:134.158.209.218]:1094> Removing socket from the poller [2025-11-16 12:21:13.493182 +0000][Debug ][PostMaster ] [ccxrootdegee.in2p3.fr:1094] Destroying stream [2025-11-16 12:21:13.493244 +0000][Debug ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] Closing the socket Art has completed and will exit with status 20.