Jobsub ID 21973.3@dunegpschedd01.fnal.gov

Jobsub ID	21973.3@dunegpschedd01.fnal.gov
Workflow ID	650
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-08-08 06:46:37
Site	UK_Edinburgh
Entry	DUNE_UK_SGridECDF_ce1
Last heartbeat	2025-08-08 07:25:50
From worker node	Hostname	node2b01.ecdf.ed.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-08-08 06:48:02
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6456061_739_20231207T133810Z_gen_g4_detsim_hitreco__20240510T060025Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413332_136_20231203T041041Z_gen_g4_detsim_hitreco__20240508T063433Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_174_20231203T125330Z_gen_g4_detsim_hitreco__20240508T075131Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_649_20231208T010548Z_gen_g4_detsim_hitreco__20240510T031542Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472771_367_20231208T085858Z_gen_g4_detsim_hitreco__20240510T061642Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74568471_31_20231207T151324Z_gen_g4_detsim_hitreco__20240510T030319Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481120_396_20231201T125849Z_gen_g4_detsim_hitreco__20240507T195220Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496970_765_20231202T063902Z_gen_g4_detsim_hitreco__20240508T054706Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6494524_161_20231208T063606Z_gen_g4_detsim_hitreco__20240510T052540Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50620606_380_20231207T105527Z_gen_g4_detsim_hitreco__20240510T035532Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-08-08 07:25:50
Saved logs	justin-logs:21973.3-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

08:25:30.513236 +0100][Debug  ][ExDbgMsg          ] [handler: 0xf210e20] Removed MsgHandler: 0xf210e20 from the in-queue.
[2025-08-08 08:25:30.513272 +0100][Debug  ][ExDbgMsg          ] [ccdcacli529.in2p3.fr:30573] Calling MsgHandler: 0xf210e20 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] .
[2025-08-08 08:25:30.513285 +0100][Debug  ][File              ] [0xc8a6a00@root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/7b/31/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496970_765_20231202T063902Z_gen_g4_detsim_hitreco__20240508T054706Z_reco2.root?xrdcl.requuid=d699e228-4794-4d82-89a5-48130b00b541] Close returned from ccdcacli529.in2p3.fr:30573 with: [SUCCESS] 
[2025-08-08 08:25:30.513319 +0100][Debug  ][ExDbgMsg          ] [ccdcacli529.in2p3.fr:30573] Destroying MsgHandler: 0xf210e20.
[2025-08-08 08:25:30.515303 +0100][Debug  ][Utility           ] Env: overriding entry: connectionwindow=30 with 30
[2025-08-08 08:25:30.515319 +0100][Debug  ][Utility           ] Env: overriding entry: requesttimeout=14400 with 4096
[2025-08-08 08:25:30.515327 +0100][Debug  ][Utility           ] Env: overriding entry: requesttimeout=4096 with 14400
[2025-08-08 08:25:30.515341 +0100][Debug  ][Utility           ] Env: overriding entry: redirectlimit=64 with 64
[2025-08-08 08:25:30.515363 +0100][Debug  ][Utility           ] Env: overriding entry: multiprotocol=1 with 1
[2025-08-08 08:25:30.515449 +0100][Debug  ][File              ] [0xb06d780@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/be/f7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6494524_161_20231208T063606Z_gen_g4_detsim_hitreco__20240510T052540Z_reco2.root?xrdcl.requuid=82ad8fa2-9619-4233-a262-e9fce2af7832] Sending an open command
[2025-08-08 08:25:30.515469 +0100][Debug  ][ExDbgMsg          ] [se1.farm.particle.cz:1094] MsgHandler created: 0xf210e20 (message: kXR_open (file: /dune/RSE/fardet-hd/be/f7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6494524_161_20231208T063606Z_gen_g4_detsim_hitreco__20240510T052540Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ).
[2025-08-08 08:25:30.515519 +0100][Debug  ][ExDbgMsg          ] [se1.farm.particle.cz:1094] Moving MsgHandler: 0xf210e20 (message: kXR_open (file: /dune/RSE/fardet-hd/be/f7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6494524_161_20231208T063606Z_gen_g4_detsim_hitreco__20240510T052540Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) from out-queu to in-queue.
[2025-08-08 08:25:30.552100 +0100][Debug  ][ExDbgMsg          ] [msg: 0xef4a2d0] Assigned MsgHandler: 0xf210e20.
[2025-08-08 08:25:30.552107 +0100][Debug  ][ExDbgMsg          ] [handler: 0xf210e20] Removed MsgHandler: 0xf210e20 from the in-queue.
[2025-08-08 08:25:30.552130 +0100][Debug  ][XRootD            ] [se1.farm.particle.cz:1094] Handling error while processing kXR_open (file: /dune/RSE/fardet-hd/be/f7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6494524_161_20231208T063606Z_gen_g4_detsim_hitreco__20240510T052540Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [ERROR] Error response: no such file or directory.
[2025-08-08 08:25:30.552306 +0100][Debug  ][ExDbgMsg          ] [se1.farm.particle.cz:1094] Calling MsgHandler: 0xf210e20 (message: kXR_open (file: /dune/RSE/fardet-hd/be/f7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6494524_161_20231208T063606Z_gen_g4_detsim_hitreco__20240510T052540Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) with status: [ERROR] Error response: no such file or directory.
[2025-08-08 08:25:30.552372 +0100][Debug  ][File              ] [0xb06d780@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/be/f7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6494524_161_20231208T063606Z_gen_g4_detsim_hitreco__20240510T052540Z_reco2.root?xrdcl.requuid=82ad8fa2-9619-4233-a262-e9fce2af7832] Open has returned with status [ERROR] Server responded with an error: [3011] No such file
[2025-08-08 08:25:30.552381 +0100][Debug  ][File              ] [0xb06d780@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/be/f7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6494524_161_20231208T063606Z_gen_g4_detsim_hitreco__20240510T052540Z_reco2.root?xrdcl.requuid=82ad8fa2-9619-4233-a262-e9fce2af7832] Error while opening at se1.farm.particle.cz:1094: [ERROR] Server responded with an error: [3011] No such file
[2025-08-08 08:25:30.552427 +0100][Debug  ][ExDbgMsg          ] [se1.farm.particle.cz:1094] Destroying MsgHandler: 0xf210e20.
[2025-08-08 08:25:30.840507 +0100][Debug  ][JobMgr            ] Stopping the job manager...
[2025-08-08 08:25:30.840764 +0100][Debug  ][JobMgr            ] Job manager stopped
[2025-08-08 08:25:30.840814 +0100][Debug  ][TaskMgr           ] Stopping the task manager...
[2025-08-08 08:25:30.840883 +0100][Debug  ][TaskMgr           ] Task manager stopped
[2025-08-08 08:25:30.840897 +0100][Debug  ][Poller            ] Stopping the poller...
[2025-08-08 08:25:30.841141 +0100][Debug  ][AsyncSock         ] [ccdcacli529.in2p3.fr:30573.0] Closing the socket
[2025-08-08 08:25:30.841153 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.34]:36326><--><[::ffff:134.158.209.139]:30573> Removing socket from the poller
[2025-08-08 08:25:30.841190 +0100][Debug  ][PostMaster        ] [ccdcacli529.in2p3.fr:30573] Destroying stream
[2025-08-08 08:25:30.841201 +0100][Debug  ][AsyncSock         ] [ccdcacli529.in2p3.fr:30573.0] Closing the socket
[2025-08-08 08:25:30.841225 +0100][Debug  ][AsyncSock         ] [ccxrootdegee.in2p3.fr:1094.0] Closing the socket
[2025-08-08 08:25:30.841230 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.34]:42082><--><[::ffff:134.158.209.218]:1094> Removing socket from the poller
[2025-08-08 08:25:30.841513 +0100][Debug  ][PostMaster        ] [ccxrootdegee.in2p3.fr:1094] Destroying stream
[2025-08-08 08:25:30.841520 +0100][Debug  ][AsyncSock         ] [ccxrootdegee.in2p3.fr:1094.0] Closing the socket
[2025-08-08 08:25:30.841586 +0100][Debug  ][AsyncSock         ] [ds26.farm.particle.cz:23381.0] Closing the socket
[2025-08-08 08:25:30.841612 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.34]:48952><--><[::ffff:147.231.25.164]:23381> Removing socket from the poller
[2025-08-08 08:25:30.841623 +0100][Debug  ][PostMaster        ] [ds26.farm.particle.cz:23381] Destroying stream
[2025-08-08 08:25:30.841627 +0100][Debug  ][AsyncSock         ] [ds26.farm.particle.cz:23381.0] Closing the socket
[2025-08-08 08:25:30.841645 +0100][Debug  ][AsyncSock         ] [se1.farm.particle.cz:1094.0] Closing the socket
[2025-08-08 08:25:30.841649 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.34]:58290><--><[::ffff:147.231.25.100]:1094> Removing socket from the poller
[2025-08-08 08:25:30.841659 +0100][Debug  ][PostMaster        ] [se1.farm.particle.cz:1094] Destroying stream
[2025-08-08 08:25:30.841668 +0100][Debug  ][AsyncSock         ] [se1.farm.particle.cz:1094.0] Closing the socket
lar exit code 1
=== Start last 100 lines of lar log file ===
tail: cannot open '_reco_20250808T064810Z.log' for reading: No such file or directory
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/0b/78/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6456061_739_20231207T133810Z_gen_g4_detsim_hitreco__20240510T060025Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/cf/6b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413332_136_20231203T041041Z_gen_g4_detsim_hitreco__20240508T063433Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/7f/fa/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_174_20231203T125330Z_gen_g4_detsim_hitreco__20240508T075131Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/82/ac/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_649_20231208T010548Z_gen_g4_detsim_hitreco__20240510T031542Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/04/44/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472771_367_20231208T085858Z_gen_g4_detsim_hitreco__20240510T061642Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/71/f0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74568471_31_20231207T151324Z_gen_g4_detsim_hitreco__20240510T030319Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/79/05/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481120_396_20231201T125849Z_gen_g4_detsim_hitreco__20240507T195220Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/7b/31/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496970_765_20231202T063902Z_gen_g4_detsim_hitreco__20240508T054706Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/be/f7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6494524_161_20231208T063606Z_gen_g4_detsim_hitreco__20240510T052540Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/cc/15/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50620606_380_20231207T105527Z_gen_g4_detsim_hitreco__20240510T035532Z_reco2.root
.:
total 126416
-rw-r--r-- 1 gl05pi6 eddie_users 87182992 Aug  8 08:25 jobscript.log
-rw-r--r-- 1 gl05pi6 eddie_users 13114039 Aug  8 08:25 flatcaf.root
-rw-r--r-- 1 gl05pi6 eddie_users 10924560 Aug  8 08:25 caf.root
-rw-r--r-- 1 gl05pi6 eddie_users 10924560 Aug  8 08:25 caf_fd_hd_atmo_650_20250808T064810Z.root
-rw-r--r-- 1 gl05pi6 eddie_users  7246368 Aug  8 08:25 caf_20250808T064810Z.log
-rw-r--r-- 1 gl05pi6 eddie_users     1914 Aug  8 08:25 caf_20250808T064810Z.file
-rw-r--r-- 1 gl05pi6 eddie_users     1914 Aug  8 08:25 caf_20250808T064810Z.pfns
-rw-r--r-- 1 gl05pi6 eddie_users     1914 Aug  8 07:48 file.list
-rw-r--r-- 1 gl05pi6 eddie_users     1914 Aug  8 08:25 justin-processed-pfns.txt
-rw-r--r-- 1 gl05pi6 eddie_users     1374 Aug  8 07:48 all-input-dids.txt
-rw-r--r-- 1 gl05pi6 eddie_users     1374 Aug  8 08:25 caf_20250808T064810Z.did
-rw-r--r-- 1 gl05pi6 eddie_users     1374 Aug  8 07:48 did.list