Jobsub ID 252805.63@dunegpschedd01.fnal.gov
| Jobsub ID | 252805.63@dunegpschedd01.fnal.gov | |
| Workflow ID | 10257 | |
| Stage ID | 1 | |
| User name | epennacc@fnal.gov | |
| HTCondor Group | group_dune.prod_mcsim | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-11-16 13:23:56 | |
| Site | UK_Edinburgh | |
| Entry | DUNE_UK_SGridECDF_ce1_multicore | |
| Last heartbeat | 2025-11-16 13:49:19 | |
| From worker node | Hostname | node2b01.ecdf.ed.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 171000 (47 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Started | 2025-11-16 13:47:56 | |
| Input files | fardet-hd:prodgenie_nu_dune10kt_1x2x6_20251003T234302Z_gen_006511_g4_detsim.root fardet-hd:prodgenie_nu_dune10kt_1x2x6_20251003T202336Z_gen_004220_g4_detsim.root fardet-hd:prodgenie_nu_dune10kt_1x2x6_20251003T223148Z_gen_009738_g4_detsim.root fardet-hd:prodgenie_nu_dune10kt_1x2x6_20251003T220614Z_gen_009254_g4_detsim.root | |
| Jobscript | Exit code | 20 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-11-16 13:49:19 | |
| Saved logs | justin-logs:252805.63-dunegpschedd01.fnal.gov.logs.tgz | |
| List job events (HTCondor job logs unavailable) | ||
Jobscript log (last 10,000 characters)
000][Debug ][PlugInMgr ] Trying to disable plug-in for '*' [2025-11-16 13:48:15.219446 +0000][Debug ][PlugInMgr ] Processing plug-in definitions in /home/.xrootd/client.plugins.d... [2025-11-16 13:48:15.219456 +0000][Debug ][PlugInMgr ] Unable to process directory /home/.xrootd/client.plugins.d: [ERROR] OS Error: no such file or directory [2025-11-16 13:48:15.325317 +0000][Debug ][Utility ] Env: overriding entry: connectionwindow=120 with 30 [2025-11-16 13:48:15.325348 +0000][Debug ][Utility ] Env: overriding entry: requesttimeout=1800 with 4096 [2025-11-16 13:48:15.325356 +0000][Debug ][Utility ] Env: overriding entry: requesttimeout=4096 with 14400 [2025-11-16 13:48:15.325371 +0000][Debug ][Utility ] Env: overriding entry: redirectlimit=16 with 64 [2025-11-16 13:48:15.325386 +0000][Debug ][Utility ] Env: overriding entry: multiprotocol=0 with 1 [2025-11-16 13:48:15.325761 +0000][Debug ][File ] [0x74a4050@root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/d7/74/prodgenie_nu_dune10kt_1x2x6_20251003T234302Z_gen_006511_g4_detsim.root?xrdcl.requuid=9c15c321-cf59-4ae1-a836-62cf9db301aa] Sending an open command [2025-11-16 13:48:15.326151 +0000][Debug ][Utility ] Env: trying to get a non-existent string entry: pollerpreference [2025-11-16 13:48:15.326220 +0000][Debug ][Poller ] Available pollers: built-in [2025-11-16 13:48:15.326224 +0000][Debug ][Poller ] Attempting to create a poller according to preference: built-in [2025-11-16 13:48:15.326229 +0000][Debug ][Poller ] Creating poller: built-in [2025-11-16 13:48:15.326289 +0000][Debug ][Poller ] Creating and starting the built-in poller... [2025-11-16 13:48:15.326929 +0000][Debug ][Poller ] Using 1 poller threads [2025-11-16 13:48:15.326974 +0000][Debug ][TaskMgr ] Starting the task manager... [2025-11-16 13:48:15.327023 +0000][Debug ][TaskMgr ] Task manager started [2025-11-16 13:48:15.327047 +0000][Debug ][JobMgr ] Starting the job manager... [2025-11-16 13:48:15.327150 +0000][Debug ][JobMgr ] Job manager started, 3 workers [2025-11-16 13:48:15.327218 +0000][Debug ][TaskMgr ] Registering task: "FileTimer task" to be run at: [2025-11-16 13:48:15 +0000] [2025-11-16 13:48:15.327447 +0000][Debug ][ExDbgMsg ] [ccxrootdegee.in2p3.fr:1094] MsgHandler created: 0x5b60130 (message: kXR_open (file: pnfs/in2p3.fr/data/dune/disk/fardet-hd/d7/74/prodgenie_nu_dune10kt_1x2x6_20251003T234302Z_gen_006511_g4_detsim.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ). [2025-11-16 13:48:15.327757 +0000][Debug ][PostMaster ] Creating new channel to: root://ccxrootdegee.in2p3.fr:1094/ [2025-11-16 13:48:15.328056 +0000][Debug ][PostMaster ] [ccxrootdegee.in2p3.fr:1094] Stream parameters: Network Stack: IPAuto, Connection Window: 30, ConnectionRetry: 5, Stream Error Window: 1800 [2025-11-16 13:48:15.328158 +0000][Debug ][TaskMgr ] Registering task: "TickGeneratorTask for: root://ccxrootdegee.in2p3.fr:1094/" to be run at: [2025-11-16 13:48:30 +0000] [2025-11-16 13:48:15.329323 +0000][Debug ][PostMaster ] [ccxrootdegee.in2p3.fr:1094] Found 1 address(es): [::ffff:134.158.209.218]:1094 [2025-11-16 13:48:15.329394 +0000][Debug ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] Attempting connection to [::ffff:134.158.209.218]:1094 [2025-11-16 13:48:15.329455 +0000][Debug ][Poller ] Adding socket 0x74a6c00 to the poller [2025-11-16 13:48:15.357811 +0000][Debug ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] Async connection call returned [2025-11-16 13:48:15.358071 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Sending out the initial hand shake + kXR_protocol [2025-11-16 13:48:15.386736 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Got the server hand shake response (type: manager [], protocol version 500) [2025-11-16 13:48:15.386890 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] kXR_protocol successful (type: manager [], protocol version 500) [2025-11-16 13:48:15.387789 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Sending out kXR_login request, username: gl05pi8, cgi: xrd.cc=uk&xrd.tz=0&xrd.appname=lar&xrd.info=&xrd.hostname=node2b01.ecdf.ed.ac.uk&xrd.rn=v5.5.5, dual-stack: false, private IPv4: false, private IPv6: false [2025-11-16 13:48:15.387827 +0000][Debug ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] TLS hand-shake exchange. [2025-11-16 13:48:15.433010 +0000][Debug ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] TLS hand-shake exchange. [2025-11-16 13:48:15.463108 +0000][Debug ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] TLS hand-shake exchange. [2025-11-16 13:48:15.463326 +0000][Info ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] TLS hand-shake done. [2025-11-16 13:48:15.492212 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Logged in, session: c6c98c4869598d12d5e94843f5e79f17 [2025-11-16 13:48:15.492242 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Authentication is required: &P=ztn,0:4096:&P=gsi,v:10400,c:ssl,ca:5a04724c [2025-11-16 13:48:15.492271 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Sending authentication data [2025-11-16 13:48:15.497226 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Trying to authenticate using ztn [2025-11-16 13:48:15.497270 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Cannot get credentials for protocol ztn: Secztn: No token found; runtime fetch disallowed. [2025-11-16 13:48:15.505347 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Trying to authenticate using gsi [2025-11-16 13:48:16.377248 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Sending more authentication data for gsi [2025-11-16 13:48:16.506135 +0000][Debug ][XRootDTransport ] [ccxrootdegee.in2p3.fr:1094.0] Authenticated with gsi. [2025-11-16 13:48:16.506216 +0000][Debug ][PostMaster ] [ccxrootdegee.in2p3.fr:1094] Stream 0 connected (IPv4). [2025-11-16 13:48:16.506246 +0000][Debug ][Utility ] Monitor library name not set. No monitoring [2025-11-16 13:48:16.506445 +0000][Debug ][ExDbgMsg ] [ccxrootdegee.in2p3.fr:1094] Moving MsgHandler: 0x5b60130 (message: kXR_open (file: pnfs/in2p3.fr/data/dune/disk/fardet-hd/d7/74/prodgenie_nu_dune10kt_1x2x6_20251003T234302Z_gen_006511_g4_detsim.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) from out-queu to in-queue. [2025-11-16 13:48:17.420829 +0000][Debug ][ExDbgMsg ] [msg: 0x5b6a320] Assigned MsgHandler: 0x5b60130. [2025-11-16 13:48:17.420867 +0000][Debug ][ExDbgMsg ] [handler: 0x5b60130] Removed MsgHandler: 0x5b60130 from the in-queue. [2025-11-16 13:48:17.421286 +0000][Debug ][XRootD ] [ccxrootdegee.in2p3.fr:1094] Handling error while processing kXR_open (file: pnfs/in2p3.fr/data/dune/disk/fardet-hd/d7/74/prodgenie_nu_dune10kt_1x2x6_20251003T234302Z_gen_006511_g4_detsim.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [ERROR] Error response: bad address. [2025-11-16 13:48:17.421455 +0000][Debug ][ExDbgMsg ] [ccxrootdegee.in2p3.fr:1094] Calling MsgHandler: 0x5b60130 (message: kXR_open (file: pnfs/in2p3.fr/data/dune/disk/fardet-hd/d7/74/prodgenie_nu_dune10kt_1x2x6_20251003T234302Z_gen_006511_g4_detsim.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) with status: [ERROR] Error response: bad address. [2025-11-16 13:48:17.421623 +0000][Debug ][File ] [0x74a4050@root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/d7/74/prodgenie_nu_dune10kt_1x2x6_20251003T234302Z_gen_006511_g4_detsim.root?xrdcl.requuid=9c15c321-cf59-4ae1-a836-62cf9db301aa] Open has returned with status [ERROR] Server responded with an error: [3012] Internal timeout [2025-11-16 13:48:17.421654 +0000][Debug ][File ] [0x74a4050@root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/d7/74/prodgenie_nu_dune10kt_1x2x6_20251003T234302Z_gen_006511_g4_detsim.root?xrdcl.requuid=9c15c321-cf59-4ae1-a836-62cf9db301aa] Error while opening at ccxrootdegee.in2p3.fr:1094: [ERROR] Server responded with an error: [3012] Internal timeout [2025-11-16 13:48:17.421751 +0000][Debug ][ExDbgMsg ] [ccxrootdegee.in2p3.fr:1094] Destroying MsgHandler: 0x5b60130. %MSG-s ArtException: TriggerResultInserter:TriggerResults@Construction 16-Nov-2025 13:48:18 GMT ModuleConstruction cet::exception caught in art ---- FileOpenError BEGIN RootInputFileSequence::initFile(): Input file root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/d7/74/prodgenie_nu_dune10kt_1x2x6_20251003T234302Z_gen_006511_g4_detsim.root was not found or could not be opened. ---- FileOpenError END %MSG [2025-11-16 13:48:18.318971 +0000][Debug ][JobMgr ] Stopping the job manager... [2025-11-16 13:48:18.319345 +0000][Debug ][JobMgr ] Job manager stopped [2025-11-16 13:48:18.319400 +0000][Debug ][TaskMgr ] Stopping the task manager... [2025-11-16 13:48:18.319469 +0000][Debug ][TaskMgr ] Task manager stopped [2025-11-16 13:48:18.319474 +0000][Debug ][Poller ] Stopping the poller... [2025-11-16 13:48:18.319740 +0000][Debug ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] Closing the socket [2025-11-16 13:48:18.319801 +0000][Debug ][Poller ] <[::ffff:192.41.105.34]:47396><--><[::ffff:134.158.209.218]:1094> Removing socket from the poller [2025-11-16 13:48:18.320320 +0000][Debug ][PostMaster ] [ccxrootdegee.in2p3.fr:1094] Destroying stream [2025-11-16 13:48:18.320386 +0000][Debug ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] Closing the socket Art has completed and will exit with status 20.