Jobsub ID 19310.5@dunegpschedd01.fnal.gov

Jobsub ID	19310.5@dunegpschedd01.fnal.gov
Workflow ID	111
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-30 10:46:30
Site	UK_Edinburgh
Entry	DUNE_UK_SGridECDF_ce1
Last heartbeat	2025-07-30 10:48:51
From worker node	Hostname	node2b01.ecdf.ed.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-07-30 10:47:30
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250122T152224Z_gen_000164_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	20
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-07-30 10:48:51
Saved logs	justin-logs:19310.5-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

ity           ] Env: overriding entry: requesttimeout=4096 with 14400
[2025-07-30 11:48:02.846364 +0100][Debug  ][Utility           ] Env: overriding entry: redirectlimit=16 with 64
[2025-07-30 11:48:02.846388 +0100][Debug  ][Utility           ] Env: overriding entry: multiprotocol=0 with 1
[2025-07-30 11:48:02.846751 +0100][Debug  ][File              ] [0x9f46ef0@root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/7e/a8/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250122T152224Z_gen_000164_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=2ab85077-535f-4fdc-864b-ed22e212a684] Sending an open command
[2025-07-30 11:48:02.847246 +0100][Debug  ][Utility           ] Env: trying to get a non-existent string entry: pollerpreference
[2025-07-30 11:48:02.847333 +0100][Debug  ][Poller            ] Available pollers: built-in
[2025-07-30 11:48:02.847338 +0100][Debug  ][Poller            ] Attempting to create a poller according to preference: built-in
[2025-07-30 11:48:02.847342 +0100][Debug  ][Poller            ] Creating poller: built-in
[2025-07-30 11:48:02.847417 +0100][Debug  ][Poller            ] Creating and starting the built-in poller...
[2025-07-30 11:48:02.847930 +0100][Debug  ][Poller            ] Using 1 poller threads
[2025-07-30 11:48:02.847971 +0100][Debug  ][TaskMgr           ] Starting the task manager...
[2025-07-30 11:48:02.848269 +0100][Debug  ][TaskMgr           ] Task manager started
[2025-07-30 11:48:02.848311 +0100][Debug  ][JobMgr            ] Starting the job manager...
[2025-07-30 11:48:02.848406 +0100][Debug  ][JobMgr            ] Job manager started, 3 workers
[2025-07-30 11:48:02.848490 +0100][Debug  ][TaskMgr           ] Registering task: "FileTimer task" to be run at: [2025-07-30 11:48:02 +0100]
[2025-07-30 11:48:02.848790 +0100][Debug  ][ExDbgMsg          ] [fndca1.fnal.gov:1094] MsgHandler created: 0xeb0c6a0 (message: kXR_open (file: pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/7e/a8/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250122T152224Z_gen_000164_supernova_g4stage1_g4stage2_detsim_reco.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ).
[2025-07-30 11:48:02.849225 +0100][Debug  ][PostMaster        ] Creating new channel to: root://fndca1.fnal.gov:1094/
[2025-07-30 11:48:02.849515 +0100][Debug  ][PostMaster        ] [fndca1.fnal.gov:1094] Stream parameters: Network Stack: IPAuto, Connection Window: 30, ConnectionRetry: 5, Stream Error Window: 1800
[2025-07-30 11:48:02.849664 +0100][Debug  ][TaskMgr           ] Registering task: "TickGeneratorTask for: root://fndca1.fnal.gov:1094/" to be run at: [2025-07-30 11:48:17 +0100]
[2025-07-30 11:48:02.850928 +0100][Debug  ][PostMaster        ] [fndca1.fnal.gov:1094] Found 1 address(es): [::ffff:131.225.69.121]:1094
[2025-07-30 11:48:02.851006 +0100][Debug  ][AsyncSock         ] [fndca1.fnal.gov:1094.0] Attempting connection to [::ffff:131.225.69.121]:1094
[2025-07-30 11:48:02.851082 +0100][Debug  ][Poller            ] Adding socket 0xeb3c420 to the poller
[2025-07-30 11:48:02.971093 +0100][Debug  ][AsyncSock         ] [fndca1.fnal.gov:1094.0] Async connection call returned
[2025-07-30 11:48:02.971350 +0100][Debug  ][XRootDTransport   ] [fndca1.fnal.gov:1094.0] Sending out the initial hand shake + kXR_protocol
[2025-07-30 11:48:03.091451 +0100][Debug  ][XRootDTransport   ] [fndca1.fnal.gov:1094.0] Got the server hand shake response (type: manager [], protocol version 500)
[2025-07-30 11:48:03.091617 +0100][Debug  ][XRootDTransport   ] [fndca1.fnal.gov:1094.0] kXR_protocol successful (type: manager [], protocol version 500)
[2025-07-30 11:48:03.092656 +0100][Debug  ][XRootDTransport   ] [fndca1.fnal.gov:1094.0] Sending out kXR_login request, username: gl05pi6, cgi: xrd.cc=uk&xrd.tz=0&xrd.appname=lar&xrd.info=&xrd.hostname=node2b01.ecdf.ed.ac.uk&xrd.rn=v5.5.5, dual-stack: false, private IPv4: false, private IPv6: false
[2025-07-30 11:48:03.092706 +0100][Debug  ][AsyncSock         ] [fndca1.fnal.gov:1094.0] TLS hand-shake exchange.
[2025-07-30 11:48:03.223266 +0100][Debug  ][AsyncSock         ] [fndca1.fnal.gov:1094.0] TLS hand-shake exchange.
[2025-07-30 11:48:03.346076 +0100][Debug  ][AsyncSock         ] [fndca1.fnal.gov:1094.0] TLS hand-shake exchange.
[2025-07-30 11:48:03.346326 +0100][Info   ][AsyncSock         ] [fndca1.fnal.gov:1094.0] TLS hand-shake done.
[2025-07-30 11:48:03.466480 +0100][Debug  ][XRootDTransport   ] [fndca1.fnal.gov:1094.0] Logged in, session: ffd47d9cef784ee4dfba6e7d9d3ec129
[2025-07-30 11:48:03.466496 +0100][Debug  ][XRootDTransport   ] [fndca1.fnal.gov:1094.0] Authentication is required: &P=ztn,0:4096:&P=gsi,v:10400,c:ssl,ca:3cbc995f
[2025-07-30 11:48:03.466531 +0100][Debug  ][XRootDTransport   ] [fndca1.fnal.gov:1094.0] Sending authentication data
[2025-07-30 11:48:03.479041 +0100][Debug  ][XRootDTransport   ] [fndca1.fnal.gov:1094.0] Trying to authenticate using ztn
[2025-07-30 11:48:03.479077 +0100][Debug  ][XRootDTransport   ] [fndca1.fnal.gov:1094.0] Cannot get credentials for protocol ztn: Secztn: No token found; runtime fetch disallowed.
[2025-07-30 11:48:03.496036 +0100][Debug  ][XRootDTransport   ] [fndca1.fnal.gov:1094.0] Trying to authenticate using gsi
[2025-07-30 11:48:03.756226 +0100][Debug  ][XRootDTransport   ] [fndca1.fnal.gov:1094.0] Sending more authentication data for gsi
[2025-07-30 11:48:03.897811 +0100][Debug  ][XRootDTransport   ] [fndca1.fnal.gov:1094.0] Authenticated with gsi.
[2025-07-30 11:48:03.897927 +0100][Debug  ][PostMaster        ] [fndca1.fnal.gov:1094] Stream 0 connected (IPv4).
[2025-07-30 11:48:03.897969 +0100][Debug  ][Utility           ] Monitor library name not set. No monitoring
[2025-07-30 11:48:03.898258 +0100][Debug  ][ExDbgMsg          ] [fndca1.fnal.gov:1094] Moving MsgHandler: 0xeb0c6a0 (message: kXR_open (file: pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/7e/a8/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250122T152224Z_gen_000164_supernova_g4stage1_g4stage2_detsim_reco.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) from out-queu to in-queue.
[2025-07-30 11:48:04.028016 +0100][Debug  ][ExDbgMsg          ] [msg: 0xeb3f9a0] Assigned MsgHandler: 0xeb0c6a0.
[2025-07-30 11:48:04.028054 +0100][Debug  ][ExDbgMsg          ] [handler: 0xeb0c6a0] Removed MsgHandler: 0xeb0c6a0 from the in-queue.
[2025-07-30 11:48:04.028345 +0100][Debug  ][XRootD            ] [fndca1.fnal.gov:1094] Handling error while processing kXR_open (file: pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/7e/a8/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250122T152224Z_gen_000164_supernova_g4stage1_g4stage2_detsim_reco.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [ERROR] Error response: bad address.
[2025-07-30 11:48:04.028550 +0100][Debug  ][ExDbgMsg          ] [fndca1.fnal.gov:1094] Calling MsgHandler: 0xeb0c6a0 (message: kXR_open (file: pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/7e/a8/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250122T152224Z_gen_000164_supernova_g4stage1_g4stage2_detsim_reco.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) with status: [ERROR] Error response: bad address.
[2025-07-30 11:48:04.028759 +0100][Debug  ][File              ] [0x9f46ef0@root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/7e/a8/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250122T152224Z_gen_000164_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=2ab85077-535f-4fdc-864b-ed22e212a684] Open has returned with status [ERROR] Server responded with an error: [3012] Failed to open file (Pool unavailable [1010])
[2025-07-30 11:48:04.028801 +0100][Debug  ][File              ] [0x9f46ef0@root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/7e/a8/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250122T152224Z_gen_000164_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=2ab85077-535f-4fdc-864b-ed22e212a684] Error while opening at fndca1.fnal.gov:1094: [ERROR] Server responded with an error: [3012] Failed to open file (Pool unavailable [1010])
[2025-07-30 11:48:04.028927 +0100][Debug  ][ExDbgMsg          ] [fndca1.fnal.gov:1094] Destroying MsgHandler: 0xeb0c6a0.
MultiPandoraApiImpl::DeletePandoraInstances - unable to find daughter instances associated with primary 0
%MSG-s ArtException:  TriggerResultInserter:TriggerResults@Construction  30-Jul-2025 11:48:04 BST ModuleConstruction
cet::exception caught in art
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/7e/a8/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250122T152224Z_gen_000164_supernova_g4stage1_g4stage2_detsim_reco.root was not found or could not be opened.
---- FileOpenError END
%MSG
[2025-07-30 11:48:04.121664 +0100][Debug  ][JobMgr            ] Stopping the job manager...
[2025-07-30 11:48:04.121981 +0100][Debug  ][JobMgr            ] Job manager stopped
[2025-07-30 11:48:04.122046 +0100][Debug  ][TaskMgr           ] Stopping the task manager...
[2025-07-30 11:48:04.122182 +0100][Debug  ][TaskMgr           ] Task manager stopped
[2025-07-30 11:48:04.122198 +0100][Debug  ][Poller            ] Stopping the poller...
[2025-07-30 11:48:04.122608 +0100][Debug  ][AsyncSock         ] [fndca1.fnal.gov:1094.0] Closing the socket
[2025-07-30 11:48:04.122848 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.34]:59872><--><[::ffff:131.225.69.121]:1094> Removing socket from the poller
[2025-07-30 11:48:04.123157 +0100][Debug  ][PostMaster        ] [fndca1.fnal.gov:1094] Destroying stream
[2025-07-30 11:48:04.123240 +0100][Debug  ][AsyncSock         ] [fndca1.fnal.gov:1094.0] Closing the socket
Art has completed and will exit with status 20.