Jobsub ID 266863.181@dunegpschedd02.fnal.gov

Jobsub ID	266863.181@dunegpschedd02.fnal.gov
Workflow ID	11601
Stage ID	1
User name	imawby@fnal.gov
Requested	Processors	1
	GPU	No
	RSS bytes	1048576000 (1000 MiB)
	Wall seconds limit	7200 (2 hours)
Submitted time	2026-01-08 10:22:55
Site	UK_Edinburgh
Entry	DUNE_UK_SGridECDF_ce1_multicore
Last heartbeat	2026-01-08 11:10:20
From worker node	Hostname	node2b06.ecdf.ed.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	1048576000 (1000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2026-01-08 11:09:12
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_856_20231202T151741Z_gen_g4_detsim_hitreco__20240508T030513Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2026-01-08 11:10:20
Saved logs	justin-logs:266863.181-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

g request to open input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/8b/d0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_856_20231202T151741Z_gen_g4_detsim_hitreco__20240508T030513Z_reco2.root"
[2026-01-08 11:09:52.670213 +0000][Debug  ][Utility           ][ 1307] Env: overriding entry: connectionwindow=120 with 30
[2026-01-08 11:09:52.670256 +0000][Debug  ][Utility           ][ 1307] Env: overriding entry: requesttimeout=1800 with 4096
[2026-01-08 11:09:52.670265 +0000][Debug  ][Utility           ][ 1307] Env: overriding entry: requesttimeout=4096 with 14400
[2026-01-08 11:09:52.670287 +0000][Debug  ][Utility           ][ 1307] Env: overriding entry: redirectlimit=16 with 64
[2026-01-08 11:09:52.670305 +0000][Debug  ][Utility           ][ 1307] Env: overriding entry: multiprotocol=0 with 1
[2026-01-08 11:09:52.670585 +0000][Debug  ][File              ][ 1307] [0x6a418f0@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/8b/d0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_856_20231202T151741Z_gen_g4_detsim_hitreco__20240508T030513Z_reco2.root?xrdcl.requuid=85f0ed30-4d58-4ee8-a10d-7a771c2a8fe3] Sending an open command
[2026-01-08 11:09:52.670703 +0000][Debug  ][Utility           ][ 1307] Env: trying to get a non-existent string entry: pollerpreference
[2026-01-08 11:09:52.670725 +0000][Debug  ][Poller            ][ 1307] Available pollers: built-in
[2026-01-08 11:09:52.670730 +0000][Debug  ][Poller            ][ 1307] Attempting to create a poller according to preference: built-in
[2026-01-08 11:09:52.670734 +0000][Debug  ][Poller            ][ 1307] Creating poller: built-in
[2026-01-08 11:09:52.670753 +0000][Debug  ][Poller            ][ 1307] Creating and starting the built-in poller...
[2026-01-08 11:09:52.671187 +0000][Debug  ][Poller            ][ 1307] Using 1 poller threads
[2026-01-08 11:09:52.671206 +0000][Debug  ][TaskMgr           ][ 1307] Starting the task manager...
[2026-01-08 11:09:52.671249 +0000][Debug  ][TaskMgr           ][ 1307] Task manager started
[2026-01-08 11:09:52.671261 +0000][Debug  ][JobMgr            ][ 1307] Starting the job manager...
[2026-01-08 11:09:52.671372 +0000][Debug  ][JobMgr            ][ 1307] Job manager started, 3 workers
[2026-01-08 11:09:52.671395 +0000][Debug  ][TaskMgr           ][ 1307] Registering task: "FileTimer task" to be run at: [2026-01-08 11:09:52 +0000]
[2026-01-08 11:09:52.671476 +0000][Debug  ][ExDbgMsg          ][ 1307] [se1.farm.particle.cz:1094] MsgHandler created: 0xb6a5210 (message: kXR_open (file: /dune/RSE/fardet-hd/8b/d0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_856_20231202T151741Z_gen_g4_detsim_hitreco__20240508T030513Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ).
[2026-01-08 11:09:52.671582 +0000][Debug  ][PostMaster        ][ 1307] Creating new channel to: root://se1.farm.particle.cz:1094/
[2026-01-08 11:09:52.671786 +0000][Debug  ][PostMaster        ][ 1307] [se1.farm.particle.cz:1094] Stream parameters: Network Stack: IPAuto, Connection Window: 30, ConnectionRetry: 5, Stream Error Window: 1800
[2026-01-08 11:09:52.671824 +0000][Debug  ][TaskMgr           ][ 1307] Registering task: "TickGeneratorTask for: root://se1.farm.particle.cz:1094/" to be run at: [2026-01-08 11:10:07 +0000]
[2026-01-08 11:09:52.672385 +0000][Debug  ][PostMaster        ][ 1307] [se1.farm.particle.cz:1094] Found 1 address(es): [::ffff:147.231.25.100]:1094
[2026-01-08 11:09:52.672424 +0000][Debug  ][AsyncSock         ][ 1307] [se1.farm.particle.cz:1094.0] Attempting connection to [::ffff:147.231.25.100]:1094
[2026-01-08 11:09:52.672462 +0000][Debug  ][Poller            ][ 1307] Adding socket 0xdd6ea70 to the poller
[2026-01-08 11:09:52.704298 +0000][Debug  ][AsyncSock         ][ 1307] [se1.farm.particle.cz:1094.0] Async connection call returned
[2026-01-08 11:09:52.704432 +0000][Debug  ][XRootDTransport   ][ 1307] [se1.farm.particle.cz:1094.0] Sending out the initial hand shake + kXR_protocol
[2026-01-08 11:09:52.736413 +0000][Debug  ][XRootDTransport   ][ 1307] [se1.farm.particle.cz:1094.0] Got the server hand shake response (type: manager [], protocol version 500)
[2026-01-08 11:09:52.736498 +0000][Debug  ][XRootDTransport   ][ 1307] [se1.farm.particle.cz:1094.0] kXR_protocol successful (type: manager [], protocol version 500)
[2026-01-08 11:09:52.737043 +0000][Debug  ][XRootDTransport   ][ 1307] [se1.farm.particle.cz:1094.0] Sending out kXR_login request, username: gl05pi8, cgi: xrd.cc=uk&xrd.tz=0&xrd.appname=lar&xrd.info=&xrd.hostname=node2b06.ecdf.ed.ac.uk&xrd.rn=v5.5.5, dual-stack: false, private IPv4: false, private IPv6: false
[2026-01-08 11:09:52.768924 +0000][Debug  ][XRootDTransport   ][ 1307] [se1.farm.particle.cz:1094.0] Logged in, session: f406d37e5d84b80c1f15e922779ca498
[2026-01-08 11:09:52.768939 +0000][Debug  ][XRootDTransport   ][ 1307] [se1.farm.particle.cz:1094.0] Authentication is required: &P=gsi,v:10400,c:ssl,ca:8d903d8c&P=unix
[2026-01-08 11:09:52.768950 +0000][Debug  ][XRootDTransport   ][ 1307] [se1.farm.particle.cz:1094.0] Sending authentication data
[2026-01-08 11:09:52.785849 +0000][Debug  ][XRootDTransport   ][ 1307] [se1.farm.particle.cz:1094.0] Trying to authenticate using gsi
[2026-01-08 11:09:53.374295 +0000][Debug  ][XRootDTransport   ][ 1307] [se1.farm.particle.cz:1094.0] Sending more authentication data for gsi
[2026-01-08 11:09:53.421685 +0000][Debug  ][XRootDTransport   ][ 1307] [se1.farm.particle.cz:1094.0] Authenticated with gsi.
[2026-01-08 11:09:53.421757 +0000][Debug  ][PostMaster        ][ 1307] [se1.farm.particle.cz:1094] Stream 0 connected (IPv4).
[2026-01-08 11:09:53.421774 +0000][Debug  ][Utility           ][ 1307] Monitor library name not set. No monitoring
[2026-01-08 11:09:53.421871 +0000][Debug  ][ExDbgMsg          ][ 1307] [se1.farm.particle.cz:1094] Moving MsgHandler: 0xb6a5210 (message: kXR_open (file: /dune/RSE/fardet-hd/8b/d0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_856_20231202T151741Z_gen_g4_detsim_hitreco__20240508T030513Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) from out-queu to in-queue.
[2026-01-08 11:09:53.462484 +0000][Debug  ][ExDbgMsg          ][ 1307] [msg: 0xa642d20] Assigned MsgHandler: 0xb6a5210.
[2026-01-08 11:09:53.462535 +0000][Debug  ][ExDbgMsg          ][ 1307] [handler: 0xb6a5210] Removed MsgHandler: 0xb6a5210 from the in-queue.
[2026-01-08 11:09:53.462628 +0000][Debug  ][XRootD            ][ 1307] [se1.farm.particle.cz:1094] Handling error while processing kXR_open (file: /dune/RSE/fardet-hd/8b/d0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_856_20231202T151741Z_gen_g4_detsim_hitreco__20240508T030513Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [ERROR] Error response: no such file or directory.
[2026-01-08 11:09:53.462679 +0000][Debug  ][ExDbgMsg          ][ 1307] [se1.farm.particle.cz:1094] Calling MsgHandler: 0xb6a5210 (message: kXR_open (file: /dune/RSE/fardet-hd/8b/d0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_856_20231202T151741Z_gen_g4_detsim_hitreco__20240508T030513Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) with status: [ERROR] Error response: no such file or directory.
[2026-01-08 11:09:53.462765 +0000][Debug  ][File              ][ 1307] [0x6a418f0@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/8b/d0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_856_20231202T151741Z_gen_g4_detsim_hitreco__20240508T030513Z_reco2.root?xrdcl.requuid=85f0ed30-4d58-4ee8-a10d-7a771c2a8fe3] Open has returned with status [ERROR] Server responded with an error: [3011] No such file
[2026-01-08 11:09:53.462780 +0000][Debug  ][File              ][ 1307] [0x6a418f0@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/8b/d0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_856_20231202T151741Z_gen_g4_detsim_hitreco__20240508T030513Z_reco2.root?xrdcl.requuid=85f0ed30-4d58-4ee8-a10d-7a771c2a8fe3] Error while opening at se1.farm.particle.cz:1094: [ERROR] Server responded with an error: [3011] No such file
[2026-01-08 11:09:53.462828 +0000][Debug  ][ExDbgMsg          ][ 1307] [se1.farm.particle.cz:1094] Destroying MsgHandler: 0xb6a5210.
MultiPandoraApiImpl::DeletePandoraInstances - unable to find daughter instances associated with primary 0
%MSG-s ArtException:  TriggerResultInserter:TriggerResults@Construction  08-Jan-2026 11:09:53 GMT ModuleConstruction
cet::exception caught in art
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/8b/d0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_856_20231202T151741Z_gen_g4_detsim_hitreco__20240508T030513Z_reco2.root was not found or could not be opened.
---- FileOpenError END
%MSG
Art has completed and will exit with status 20.
[2026-01-08 11:09:53.663820 +0000][Debug  ][JobMgr            ][ 1307] Stopping the job manager...
[2026-01-08 11:09:53.664051 +0000][Debug  ][JobMgr            ][ 1307] Job manager stopped
[2026-01-08 11:09:53.664088 +0000][Debug  ][TaskMgr           ][ 1307] Stopping the task manager...
[2026-01-08 11:09:53.664213 +0000][Debug  ][TaskMgr           ][ 1307] Task manager stopped
[2026-01-08 11:09:53.664328 +0000][Debug  ][Poller            ][ 1307] Stopping the poller...
[2026-01-08 11:09:53.664663 +0000][Debug  ][AsyncSock         ][ 1307] [se1.farm.particle.cz:1094.0] Closing the socket
[2026-01-08 11:09:53.664690 +0000][Debug  ][Poller            ][ 1307] <[::ffff:192.41.105.39]:46382><--><[::ffff:147.231.25.100]:1094> Removing socket from the poller
[2026-01-08 11:09:53.664741 +0000][Debug  ][PostMaster        ][ 1307] [se1.farm.particle.cz:1094] Destroying stream
[2026-01-08 11:09:53.664767 +0000][Debug  ][AsyncSock         ][ 1307] [se1.farm.particle.cz:1094.0] Closing the socket
=== End last 100 lines of lar log file ===
mv: cannot stat 'MLPHierarchyValidationFile.root': No such file or directory
mv: cannot stat 'Validation.root': No such file or directory