Jobsub ID 15441.6@dunegpschedd02.fnal.gov

Jobsub ID	15441.6@dunegpschedd02.fnal.gov
Workflow ID	647
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-08-08 03:52:27
Site	UK_Edinburgh
Entry	DUNE_UK_SGridECDF_ce1
Last heartbeat	2025-08-08 04:02:53
From worker node	Hostname	node2b04.ecdf.ed.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-08-08 03:52:44
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_505_20231202T150608Z_gen_g4_detsim_hitreco__20240508T025740Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542214_69_20231201T220539Z_gen_g4_detsim_hitreco__20240507T204116Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505548_238_20231202T141421Z_gen_g4_detsim_hitreco__20240508T055022Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571580_627_20231203T024638Z_gen_g4_detsim_hitreco__20240508T071351Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_8_20231204T054817Z_gen_g4_detsim_hitreco__20240509T220035Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_0_20231207T045731Z_gen_g4_detsim_hitreco__20240510T060911Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420351_1028_20231204T045059Z_gen_g4_detsim_hitreco__20240509T191821Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_656_20231208T053303Z_gen_g4_detsim_hitreco__20240510T065437Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6435715_1_20231207T041035Z_gen_g4_detsim_hitreco__20240510T044851Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6208838_206_20231122T053441Z_gen_g4_detsim_hitreco__20240503T061003Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-08-08 04:02:53
Saved logs	justin-logs:15441.6-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

rm.particle.cz:24961
[2025-08-08 05:02:35.767160 +0100][Debug  ][ExDbgMsg          ] [dpmpool20.farm.particle.cz:24961] MsgHandler created: 0xc63fae0 (message: kXR_close (handle: 0x00000000) ).
[2025-08-08 05:02:35.767253 +0100][Debug  ][ExDbgMsg          ] [dpmpool20.farm.particle.cz:24961] Moving MsgHandler: 0xc63fae0 (message: kXR_close (handle: 0x00000000) ) from out-queu to in-queue.
[2025-08-08 05:02:35.798980 +0100][Debug  ][ExDbgMsg          ] [msg: 0x1d421ee0] Assigned MsgHandler: 0xc63fae0.
[2025-08-08 05:02:35.798998 +0100][Debug  ][ExDbgMsg          ] [handler: 0xc63fae0] Removed MsgHandler: 0xc63fae0 from the in-queue.
[2025-08-08 05:02:35.799019 +0100][Debug  ][ExDbgMsg          ] [dpmpool20.farm.particle.cz:24961] Calling MsgHandler: 0xc63fae0 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] .
[2025-08-08 05:02:35.799028 +0100][Debug  ][File              ] [0x14531950@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/4f/08/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542214_69_20231201T220539Z_gen_g4_detsim_hitreco__20240507T204116Z_reco2.root?xrdcl.requuid=ce2bfe73-35d4-4342-928c-d9b5bd737c7d] Close returned from dpmpool20.farm.particle.cz:24961 with: [SUCCESS] 
[2025-08-08 05:02:35.799053 +0100][Debug  ][ExDbgMsg          ] [dpmpool20.farm.particle.cz:24961] Destroying MsgHandler: 0xc63fae0.
[2025-08-08 05:02:35.801024 +0100][Debug  ][Utility           ] Env: overriding entry: connectionwindow=30 with 30
[2025-08-08 05:02:35.801042 +0100][Debug  ][Utility           ] Env: overriding entry: requesttimeout=14400 with 4096
[2025-08-08 05:02:35.801049 +0100][Debug  ][Utility           ] Env: overriding entry: requesttimeout=4096 with 14400
[2025-08-08 05:02:35.801068 +0100][Debug  ][Utility           ] Env: overriding entry: redirectlimit=64 with 64
[2025-08-08 05:02:35.801082 +0100][Debug  ][Utility           ] Env: overriding entry: multiprotocol=1 with 1
[2025-08-08 05:02:35.801169 +0100][Debug  ][File              ] [0x14da0010@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/5a/d3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505548_238_20231202T141421Z_gen_g4_detsim_hitreco__20240508T055022Z_reco2.root?xrdcl.requuid=648b2434-dd3f-44e1-9b9d-9f42de9dd3a7] Sending an open command
[2025-08-08 05:02:35.801187 +0100][Debug  ][ExDbgMsg          ] [se1.farm.particle.cz:1094] MsgHandler created: 0xc63fae0 (message: kXR_open (file: /dune/RSE/fardet-hd/5a/d3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505548_238_20231202T141421Z_gen_g4_detsim_hitreco__20240508T055022Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ).
[2025-08-08 05:02:35.801229 +0100][Debug  ][ExDbgMsg          ] [se1.farm.particle.cz:1094] Moving MsgHandler: 0xc63fae0 (message: kXR_open (file: /dune/RSE/fardet-hd/5a/d3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505548_238_20231202T141421Z_gen_g4_detsim_hitreco__20240508T055022Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) from out-queu to in-queue.
[2025-08-08 05:02:35.837428 +0100][Debug  ][ExDbgMsg          ] [msg: 0x950f8d0] Assigned MsgHandler: 0xc63fae0.
[2025-08-08 05:02:35.837444 +0100][Debug  ][ExDbgMsg          ] [handler: 0xc63fae0] Removed MsgHandler: 0xc63fae0 from the in-queue.
[2025-08-08 05:02:35.837471 +0100][Debug  ][XRootD            ] [se1.farm.particle.cz:1094] Handling error while processing kXR_open (file: /dune/RSE/fardet-hd/5a/d3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505548_238_20231202T141421Z_gen_g4_detsim_hitreco__20240508T055022Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [ERROR] Error response: no such file or directory.
[2025-08-08 05:02:35.837866 +0100][Debug  ][ExDbgMsg          ] [se1.farm.particle.cz:1094] Calling MsgHandler: 0xc63fae0 (message: kXR_open (file: /dune/RSE/fardet-hd/5a/d3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505548_238_20231202T141421Z_gen_g4_detsim_hitreco__20240508T055022Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) with status: [ERROR] Error response: no such file or directory.
[2025-08-08 05:02:35.837932 +0100][Debug  ][File              ] [0x14da0010@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/5a/d3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505548_238_20231202T141421Z_gen_g4_detsim_hitreco__20240508T055022Z_reco2.root?xrdcl.requuid=648b2434-dd3f-44e1-9b9d-9f42de9dd3a7] Open has returned with status [ERROR] Server responded with an error: [3011] No such file
[2025-08-08 05:02:35.837941 +0100][Debug  ][File              ] [0x14da0010@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/5a/d3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505548_238_20231202T141421Z_gen_g4_detsim_hitreco__20240508T055022Z_reco2.root?xrdcl.requuid=648b2434-dd3f-44e1-9b9d-9f42de9dd3a7] Error while opening at se1.farm.particle.cz:1094: [ERROR] Server responded with an error: [3011] No such file
[2025-08-08 05:02:35.837993 +0100][Debug  ][ExDbgMsg          ] [se1.farm.particle.cz:1094] Destroying MsgHandler: 0xc63fae0.
[2025-08-08 05:02:36.055869 +0100][Debug  ][JobMgr            ] Stopping the job manager...
[2025-08-08 05:02:36.056145 +0100][Debug  ][JobMgr            ] Job manager stopped
[2025-08-08 05:02:36.056205 +0100][Debug  ][TaskMgr           ] Stopping the task manager...
[2025-08-08 05:02:36.056301 +0100][Debug  ][TaskMgr           ] Task manager stopped
[2025-08-08 05:02:36.056319 +0100][Debug  ][Poller            ] Stopping the poller...
[2025-08-08 05:02:36.056731 +0100][Debug  ][AsyncSock         ] [dpmpool20.farm.particle.cz:23044.0] Closing the socket
[2025-08-08 05:02:36.056818 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.37]:58064><--><[::ffff:147.231.25.91]:23044> Removing socket from the poller
[2025-08-08 05:02:36.056922 +0100][Debug  ][PostMaster        ] [dpmpool20.farm.particle.cz:23044] Destroying stream
[2025-08-08 05:02:36.057000 +0100][Debug  ][AsyncSock         ] [dpmpool20.farm.particle.cz:23044.0] Closing the socket
[2025-08-08 05:02:36.057050 +0100][Debug  ][AsyncSock         ] [dpmpool20.farm.particle.cz:24961.0] Closing the socket
[2025-08-08 05:02:36.057056 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.37]:44138><--><[::ffff:147.231.25.91]:24961> Removing socket from the poller
[2025-08-08 05:02:36.057069 +0100][Debug  ][PostMaster        ] [dpmpool20.farm.particle.cz:24961] Destroying stream
[2025-08-08 05:02:36.057075 +0100][Debug  ][AsyncSock         ] [dpmpool20.farm.particle.cz:24961.0] Closing the socket
[2025-08-08 05:02:36.057087 +0100][Debug  ][AsyncSock         ] [se1.farm.particle.cz:1094.0] Closing the socket
[2025-08-08 05:02:36.057097 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.37]:37870><--><[::ffff:147.231.25.100]:1094> Removing socket from the poller
[2025-08-08 05:02:36.057106 +0100][Debug  ][PostMaster        ] [se1.farm.particle.cz:1094] Destroying stream
[2025-08-08 05:02:36.057110 +0100][Debug  ][AsyncSock         ] [se1.farm.particle.cz:1094.0] Closing the socket
lar exit code 1
=== Start last 100 lines of lar log file ===
tail: cannot open '_reco_20250808T035252Z.log' for reading: No such file or directory
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a8/3f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6409602_505_20231202T150608Z_gen_g4_detsim_hitreco__20240508T025740Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/4f/08/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542214_69_20231201T220539Z_gen_g4_detsim_hitreco__20240507T204116Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/5a/d3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505548_238_20231202T141421Z_gen_g4_detsim_hitreco__20240508T055022Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/3f/c0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571580_627_20231203T024638Z_gen_g4_detsim_hitreco__20240508T071351Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/80/66/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_8_20231204T054817Z_gen_g4_detsim_hitreco__20240509T220035Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/0c/e2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_0_20231207T045731Z_gen_g4_detsim_hitreco__20240510T060911Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a9/42/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420351_1028_20231204T045059Z_gen_g4_detsim_hitreco__20240509T191821Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f4/42/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_656_20231208T053303Z_gen_g4_detsim_hitreco__20240510T065437Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/be/50/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6435715_1_20231207T041035Z_gen_g4_detsim_hitreco__20240510T044851Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/52/85/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6208838_206_20231122T053441Z_gen_g4_detsim_hitreco__20240503T061003Z_reco2.root
.:
total 33400
-rw-r--r-- 1 gl05pi6 eddie_users 21897025 Aug  8 05:02 jobscript.log
-rw-r--r-- 1 gl05pi6 eddie_users  3990433 Aug  8 05:02 flatcaf.root
-rw-r--r-- 1 gl05pi6 eddie_users  3220419 Aug  8 05:02 caf.root
-rw-r--r-- 1 gl05pi6 eddie_users  3220419 Aug  8 05:02 caf_fd_hd_atmo_647_20250808T035252Z.root
-rw-r--r-- 1 gl05pi6 eddie_users  1818722 Aug  8 05:02 caf_20250808T035252Z.log
-rw-r--r-- 1 gl05pi6 eddie_users     1860 Aug  8 05:02 caf_20250808T035252Z.file
-rw-r--r-- 1 gl05pi6 eddie_users     1860 Aug  8 05:02 caf_20250808T035252Z.pfns
-rw-r--r-- 1 gl05pi6 eddie_users     1860 Aug  8 04:52 file.list
-rw-r--r-- 1 gl05pi6 eddie_users     1860 Aug  8 05:02 justin-processed-pfns.txt
-rw-r--r-- 1 gl05pi6 eddie_users     1370 Aug  8 04:52 all-input-dids.txt
-rw-r--r-- 1 gl05pi6 eddie_users     1370 Aug  8 05:02 caf_20250808T035252Z.did
-rw-r--r-- 1 gl05pi6 eddie_users     1370 Aug  8 04:52 did.list