Jobsub ID 21853.5@dunegpschedd01.fnal.gov

Jobsub ID	21853.5@dunegpschedd01.fnal.gov
Workflow ID	647
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-08-08 01:54:22
Site	UK_Edinburgh
Entry	DUNE_UK_SGridECDF_ce1
Last heartbeat	2025-08-08 02:05:59
From worker node	Hostname	node2b08.ecdf.ed.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-08-08 01:55:42
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74525168_18_20231205T103524Z_gen_g4_detsim_hitreco__20240510T062727Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_161_20231207T022510Z_gen_g4_detsim_hitreco__20240510T034428Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_472_20231207T050122Z_gen_g4_detsim_hitreco__20240510T043607Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574864_246_20231203T153645Z_gen_g4_detsim_hitreco__20240508T025319Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6468412_721_20231207T081039Z_gen_g4_detsim_hitreco__20240510T064324Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577113_524_20231204T014815Z_gen_g4_detsim_hitreco__20240509T215642Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66029763_432_20231207T032134Z_gen_g4_detsim_hitreco__20240510T024434Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258005_58_20231121T152819Z_gen_g4_detsim_hitreco__20240507T173851Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66049673_600_20231207T155354Z_gen_g4_detsim_hitreco__20240510T044753Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_546_20231203T041555Z_gen_g4_detsim_hitreco__20240508T072053Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-08-08 02:05:59
Saved logs	justin-logs:21853.5-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

:41.762471 +0100][Debug  ][ExDbgMsg          ] [dpmpool20.farm.particle.cz:23044] MsgHandler created: 0x1d218520 (message: kXR_close (handle: 0x00000000) ).
[2025-08-08 03:05:41.762546 +0100][Debug  ][ExDbgMsg          ] [dpmpool20.farm.particle.cz:23044] Moving MsgHandler: 0x1d218520 (message: kXR_close (handle: 0x00000000) ) from out-queu to in-queue.
[2025-08-08 03:05:41.794110 +0100][Debug  ][ExDbgMsg          ] [msg: 0x1f18d9b0] Assigned MsgHandler: 0x1d218520.
[2025-08-08 03:05:41.794136 +0100][Debug  ][ExDbgMsg          ] [handler: 0x1d218520] Removed MsgHandler: 0x1d218520 from the in-queue.
[2025-08-08 03:05:41.794161 +0100][Debug  ][ExDbgMsg          ] [dpmpool20.farm.particle.cz:23044] Calling MsgHandler: 0x1d218520 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] .
[2025-08-08 03:05:41.794171 +0100][Debug  ][File              ] [0xadcd560@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/e6/e5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_161_20231207T022510Z_gen_g4_detsim_hitreco__20240510T034428Z_reco2.root?xrdcl.requuid=37806e37-c858-4c5f-9baa-5e8f1fc098c9] Close returned from dpmpool20.farm.particle.cz:23044 with: [SUCCESS] 
[2025-08-08 03:05:41.794181 +0100][Debug  ][ExDbgMsg          ] [dpmpool20.farm.particle.cz:23044] Destroying MsgHandler: 0x1d218520.
[2025-08-08 03:05:41.795730 +0100][Debug  ][Utility           ] Env: overriding entry: connectionwindow=30 with 30
[2025-08-08 03:05:41.795747 +0100][Debug  ][Utility           ] Env: overriding entry: requesttimeout=14400 with 4096
[2025-08-08 03:05:41.795756 +0100][Debug  ][Utility           ] Env: overriding entry: requesttimeout=4096 with 14400
[2025-08-08 03:05:41.795785 +0100][Debug  ][Utility           ] Env: overriding entry: redirectlimit=64 with 64
[2025-08-08 03:05:41.795800 +0100][Debug  ][Utility           ] Env: overriding entry: multiprotocol=1 with 1
[2025-08-08 03:05:41.795894 +0100][Debug  ][File              ] [0x7d77290@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/17/80/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_472_20231207T050122Z_gen_g4_detsim_hitreco__20240510T043607Z_reco2.root?xrdcl.requuid=0061686b-a52f-4a74-9e3d-f349abedd05e] Sending an open command
[2025-08-08 03:05:41.795913 +0100][Debug  ][ExDbgMsg          ] [se1.farm.particle.cz:1094] MsgHandler created: 0x1d218520 (message: kXR_open (file: /dune/RSE/fardet-hd/17/80/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_472_20231207T050122Z_gen_g4_detsim_hitreco__20240510T043607Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ).
[2025-08-08 03:05:41.795968 +0100][Debug  ][ExDbgMsg          ] [se1.farm.particle.cz:1094] Moving MsgHandler: 0x1d218520 (message: kXR_open (file: /dune/RSE/fardet-hd/17/80/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_472_20231207T050122Z_gen_g4_detsim_hitreco__20240510T043607Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) from out-queu to in-queue.
[2025-08-08 03:05:41.833487 +0100][Debug  ][ExDbgMsg          ] [msg: 0xa7aaa50] Assigned MsgHandler: 0x1d218520.
[2025-08-08 03:05:41.833512 +0100][Debug  ][ExDbgMsg          ] [handler: 0x1d218520] Removed MsgHandler: 0x1d218520 from the in-queue.
[2025-08-08 03:05:41.833551 +0100][Debug  ][XRootD            ] [se1.farm.particle.cz:1094] Handling error while processing kXR_open (file: /dune/RSE/fardet-hd/17/80/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_472_20231207T050122Z_gen_g4_detsim_hitreco__20240510T043607Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [ERROR] Error response: no such file or directory.
[2025-08-08 03:05:41.833890 +0100][Debug  ][ExDbgMsg          ] [se1.farm.particle.cz:1094] Calling MsgHandler: 0x1d218520 (message: kXR_open (file: /dune/RSE/fardet-hd/17/80/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_472_20231207T050122Z_gen_g4_detsim_hitreco__20240510T043607Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) with status: [ERROR] Error response: no such file or directory.
[2025-08-08 03:05:41.833976 +0100][Debug  ][File              ] [0x7d77290@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/17/80/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_472_20231207T050122Z_gen_g4_detsim_hitreco__20240510T043607Z_reco2.root?xrdcl.requuid=0061686b-a52f-4a74-9e3d-f349abedd05e] Open has returned with status [ERROR] Server responded with an error: [3011] No such file
[2025-08-08 03:05:41.833985 +0100][Debug  ][File              ] [0x7d77290@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/17/80/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_472_20231207T050122Z_gen_g4_detsim_hitreco__20240510T043607Z_reco2.root?xrdcl.requuid=0061686b-a52f-4a74-9e3d-f349abedd05e] Error while opening at se1.farm.particle.cz:1094: [ERROR] Server responded with an error: [3011] No such file
[2025-08-08 03:05:41.834040 +0100][Debug  ][ExDbgMsg          ] [se1.farm.particle.cz:1094] Destroying MsgHandler: 0x1d218520.
[2025-08-08 03:05:42.058594 +0100][Debug  ][JobMgr            ] Stopping the job manager...
[2025-08-08 03:05:42.058884 +0100][Debug  ][JobMgr            ] Job manager stopped
[2025-08-08 03:05:42.058944 +0100][Debug  ][TaskMgr           ] Stopping the task manager...
[2025-08-08 03:05:42.059025 +0100][Debug  ][TaskMgr           ] Task manager stopped
[2025-08-08 03:05:42.059031 +0100][Debug  ][Poller            ] Stopping the poller...
[2025-08-08 03:05:42.059540 +0100][Debug  ][AsyncSock         ] [dpmpool20.farm.particle.cz:23044.0] Closing the socket
[2025-08-08 03:05:42.059639 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.41]:44504><--><[::ffff:147.231.25.91]:23044> Removing socket from the poller
[2025-08-08 03:05:42.059761 +0100][Debug  ][PostMaster        ] [dpmpool20.farm.particle.cz:23044] Destroying stream
[2025-08-08 03:05:42.059868 +0100][Debug  ][AsyncSock         ] [dpmpool20.farm.particle.cz:23044.0] Closing the socket
[2025-08-08 03:05:42.059922 +0100][Debug  ][AsyncSock         ] [dpmpool24.farm.particle.cz:23318.0] Closing the socket
[2025-08-08 03:05:42.059935 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.41]:47902><--><[::ffff:147.231.25.95]:23318> Removing socket from the poller
[2025-08-08 03:05:42.059949 +0100][Debug  ][PostMaster        ] [dpmpool24.farm.particle.cz:23318] Destroying stream
[2025-08-08 03:05:42.059953 +0100][Debug  ][AsyncSock         ] [dpmpool24.farm.particle.cz:23318.0] Closing the socket
[2025-08-08 03:05:42.059962 +0100][Debug  ][AsyncSock         ] [se1.farm.particle.cz:1094.0] Closing the socket
[2025-08-08 03:05:42.059969 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.41]:35236><--><[::ffff:147.231.25.100]:1094> Removing socket from the poller
[2025-08-08 03:05:42.059980 +0100][Debug  ][PostMaster        ] [se1.farm.particle.cz:1094] Destroying stream
[2025-08-08 03:05:42.059984 +0100][Debug  ][AsyncSock         ] [se1.farm.particle.cz:1094.0] Closing the socket
lar exit code 1
=== Start last 100 lines of lar log file ===
tail: cannot open '_reco_20250808T015550Z.log' for reading: No such file or directory
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a6/0c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74525168_18_20231205T103524Z_gen_g4_detsim_hitreco__20240510T062727Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/e6/e5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_161_20231207T022510Z_gen_g4_detsim_hitreco__20240510T034428Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/17/80/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_472_20231207T050122Z_gen_g4_detsim_hitreco__20240510T043607Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/fe/9a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574864_246_20231203T153645Z_gen_g4_detsim_hitreco__20240508T025319Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/53/31/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6468412_721_20231207T081039Z_gen_g4_detsim_hitreco__20240510T064324Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/1f/fb/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577113_524_20231204T014815Z_gen_g4_detsim_hitreco__20240509T215642Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/27/4d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66029763_432_20231207T032134Z_gen_g4_detsim_hitreco__20240510T024434Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/66/04/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258005_58_20231121T152819Z_gen_g4_detsim_hitreco__20240507T173851Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/8e/02/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66049673_600_20231207T155354Z_gen_g4_detsim_hitreco__20240510T044753Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/35/da/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_546_20231203T041555Z_gen_g4_detsim_hitreco__20240508T072053Z_reco2.root
.:
total 35400
-rw-r--r-- 1 gl05pi6 eddie_users 21982479 Aug  8 03:05 jobscript.log
-rw-r--r-- 1 gl05pi6 eddie_users  4684873 Aug  8 03:05 flatcaf.root
-rw-r--r-- 1 gl05pi6 eddie_users  3841451 Aug  8 03:05 caf.root
-rw-r--r-- 1 gl05pi6 eddie_users  3841451 Aug  8 03:05 caf_fd_hd_atmo_647_20250808T015550Z.root
-rw-r--r-- 1 gl05pi6 eddie_users  1848537 Aug  8 03:05 caf_20250808T015550Z.log
-rw-r--r-- 1 gl05pi6 eddie_users     1895 Aug  8 03:05 caf_20250808T015550Z.file
-rw-r--r-- 1 gl05pi6 eddie_users     1895 Aug  8 03:05 caf_20250808T015550Z.pfns
-rw-r--r-- 1 gl05pi6 eddie_users     1895 Aug  8 02:55 file.list
-rw-r--r-- 1 gl05pi6 eddie_users     1895 Aug  8 03:05 justin-processed-pfns.txt
-rw-r--r-- 1 gl05pi6 eddie_users     1375 Aug  8 02:55 all-input-dids.txt
-rw-r--r-- 1 gl05pi6 eddie_users     1375 Aug  8 03:05 caf_20250808T015550Z.did
-rw-r--r-- 1 gl05pi6 eddie_users     1375 Aug  8 02:55 did.list