Jobsub ID 21891.0@dunegpschedd01.fnal.gov

Jobsub ID	21891.0@dunegpschedd01.fnal.gov
Workflow ID	649
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-08-08 03:28:27
Site	UK_Edinburgh
Entry	DUNE_UK_SGridECDF_ce1
Last heartbeat	2025-08-08 03:49:02
From worker node	Hostname	node2b08.ecdf.ed.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-08-08 03:29:00
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74540126_733_20231207T064706Z_gen_g4_detsim_hitreco__20240510T055848Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_46_20231205T071817Z_gen_g4_detsim_hitreco__20240510T035231Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50253389_5_20231120T123600Z_gen_g4_detsim_hitreco__20240503T062514Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_947_20231203T124140Z_gen_g4_detsim_hitreco__20240509T201428Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_602_20231202T162415Z_gen_g4_detsim_hitreco__20240508T044114Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50579427_791_20231204T045537Z_gen_g4_detsim_hitreco__20240509T211025Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571580_986_20231203T030504Z_gen_g4_detsim_hitreco__20240508T072401Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_962_20231205T080610Z_gen_g4_detsim_hitreco__20240509T222129Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_159_20231207T163055Z_gen_g4_detsim_hitreco__20240510T030113Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_437_20231203T042358Z_gen_g4_detsim_hitreco__20240508T045227Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-08-08 03:48:59
Saved logs	justin-logs:21891.0-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

-08-08 04:48:43.702543 +0100][Debug  ][ExDbgMsg          ] [dpmpool24.farm.particle.cz:23318] Destroying MsgHandler: 0x182cf690.
[2025-08-08 04:48:43.711661 +0100][Debug  ][File              ] [0x9e7f9e0@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c5/04/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50253389_5_20231120T123600Z_gen_g4_detsim_hitreco__20240503T062514Z_reco2.root?xrdcl.requuid=fc15cc28-a6fd-4c0e-b0bd-f8c24099e160] Sending a close command for handle 0x0 to dpmpool24.farm.particle.cz:23318
[2025-08-08 04:48:43.711816 +0100][Debug  ][ExDbgMsg          ] [dpmpool24.farm.particle.cz:23318] MsgHandler created: 0xccd5d60 (message: kXR_close (handle: 0x00000000) ).
[2025-08-08 04:48:43.711884 +0100][Debug  ][ExDbgMsg          ] [dpmpool24.farm.particle.cz:23318] Moving MsgHandler: 0xccd5d60 (message: kXR_close (handle: 0x00000000) ) from out-queu to in-queue.
[2025-08-08 04:48:43.743795 +0100][Debug  ][ExDbgMsg          ] [msg: 0x153fb7e0] Assigned MsgHandler: 0xccd5d60.
[2025-08-08 04:48:43.743854 +0100][Debug  ][ExDbgMsg          ] [handler: 0xccd5d60] Removed MsgHandler: 0xccd5d60 from the in-queue.
[2025-08-08 04:48:43.743891 +0100][Debug  ][ExDbgMsg          ] [dpmpool24.farm.particle.cz:23318] Calling MsgHandler: 0xccd5d60 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] .
[2025-08-08 04:48:43.743901 +0100][Debug  ][File              ] [0x9e7f9e0@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c5/04/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50253389_5_20231120T123600Z_gen_g4_detsim_hitreco__20240503T062514Z_reco2.root?xrdcl.requuid=fc15cc28-a6fd-4c0e-b0bd-f8c24099e160] Close returned from dpmpool24.farm.particle.cz:23318 with: [SUCCESS] 
[2025-08-08 04:48:43.743917 +0100][Debug  ][ExDbgMsg          ] [dpmpool24.farm.particle.cz:23318] Destroying MsgHandler: 0xccd5d60.
[2025-08-08 04:48:43.745943 +0100][Debug  ][Utility           ] Env: overriding entry: connectionwindow=30 with 30
[2025-08-08 04:48:43.745962 +0100][Debug  ][Utility           ] Env: overriding entry: requesttimeout=14400 with 4096
[2025-08-08 04:48:43.745969 +0100][Debug  ][Utility           ] Env: overriding entry: requesttimeout=4096 with 14400
[2025-08-08 04:48:43.745989 +0100][Debug  ][Utility           ] Env: overriding entry: redirectlimit=64 with 64
[2025-08-08 04:48:43.746004 +0100][Debug  ][Utility           ] Env: overriding entry: multiprotocol=1 with 1
[2025-08-08 04:48:43.746094 +0100][Debug  ][File              ] [0x15168e20@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/2d/bc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_947_20231203T124140Z_gen_g4_detsim_hitreco__20240509T201428Z_reco2.root?xrdcl.requuid=ec7c62b7-ad20-4085-8724-3b928b702300] Sending an open command
[2025-08-08 04:48:43.746114 +0100][Debug  ][ExDbgMsg          ] [se1.farm.particle.cz:1094] MsgHandler created: 0x11889280 (message: kXR_open (file: /dune/RSE/fardet-hd/2d/bc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_947_20231203T124140Z_gen_g4_detsim_hitreco__20240509T201428Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ).
[2025-08-08 04:48:43.746166 +0100][Debug  ][ExDbgMsg          ] [se1.farm.particle.cz:1094] Moving MsgHandler: 0x11889280 (message: kXR_open (file: /dune/RSE/fardet-hd/2d/bc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_947_20231203T124140Z_gen_g4_detsim_hitreco__20240509T201428Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) from out-queu to in-queue.
[2025-08-08 04:48:43.784361 +0100][Debug  ][ExDbgMsg          ] [msg: 0x151c5f80] Assigned MsgHandler: 0x11889280.
[2025-08-08 04:48:43.784370 +0100][Debug  ][ExDbgMsg          ] [handler: 0x11889280] Removed MsgHandler: 0x11889280 from the in-queue.
[2025-08-08 04:48:43.784400 +0100][Debug  ][XRootD            ] [se1.farm.particle.cz:1094] Handling error while processing kXR_open (file: /dune/RSE/fardet-hd/2d/bc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_947_20231203T124140Z_gen_g4_detsim_hitreco__20240509T201428Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [ERROR] Error response: no such file or directory.
[2025-08-08 04:48:43.784800 +0100][Debug  ][ExDbgMsg          ] [se1.farm.particle.cz:1094] Calling MsgHandler: 0x11889280 (message: kXR_open (file: /dune/RSE/fardet-hd/2d/bc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_947_20231203T124140Z_gen_g4_detsim_hitreco__20240509T201428Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) with status: [ERROR] Error response: no such file or directory.
[2025-08-08 04:48:43.784884 +0100][Debug  ][File              ] [0x15168e20@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/2d/bc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_947_20231203T124140Z_gen_g4_detsim_hitreco__20240509T201428Z_reco2.root?xrdcl.requuid=ec7c62b7-ad20-4085-8724-3b928b702300] Open has returned with status [ERROR] Server responded with an error: [3011] No such file
[2025-08-08 04:48:43.784894 +0100][Debug  ][File              ] [0x15168e20@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/2d/bc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_947_20231203T124140Z_gen_g4_detsim_hitreco__20240509T201428Z_reco2.root?xrdcl.requuid=ec7c62b7-ad20-4085-8724-3b928b702300] Error while opening at se1.farm.particle.cz:1094: [ERROR] Server responded with an error: [3011] No such file
[2025-08-08 04:48:43.784949 +0100][Debug  ][ExDbgMsg          ] [se1.farm.particle.cz:1094] Destroying MsgHandler: 0x11889280.
[2025-08-08 04:48:44.029921 +0100][Debug  ][JobMgr            ] Stopping the job manager...
[2025-08-08 04:48:44.030251 +0100][Debug  ][JobMgr            ] Job manager stopped
[2025-08-08 04:48:44.030317 +0100][Debug  ][TaskMgr           ] Stopping the task manager...
[2025-08-08 04:48:44.030406 +0100][Debug  ][TaskMgr           ] Task manager stopped
[2025-08-08 04:48:44.030412 +0100][Debug  ][Poller            ] Stopping the poller...
[2025-08-08 04:48:44.030685 +0100][Debug  ][AsyncSock         ] [dpmpool24.farm.particle.cz:23318.0] Closing the socket
[2025-08-08 04:48:44.030704 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.41]:53858><--><[::ffff:147.231.25.95]:23318> Removing socket from the poller
[2025-08-08 04:48:44.030743 +0100][Debug  ][PostMaster        ] [dpmpool24.farm.particle.cz:23318] Destroying stream
[2025-08-08 04:48:44.030750 +0100][Debug  ][AsyncSock         ] [dpmpool24.farm.particle.cz:23318.0] Closing the socket
[2025-08-08 04:48:44.030765 +0100][Debug  ][AsyncSock         ] [se1.farm.particle.cz:1094.0] Closing the socket
[2025-08-08 04:48:44.030772 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.41]:36570><--><[::ffff:147.231.25.100]:1094> Removing socket from the poller
[2025-08-08 04:48:44.030782 +0100][Debug  ][PostMaster        ] [se1.farm.particle.cz:1094] Destroying stream
[2025-08-08 04:48:44.030786 +0100][Debug  ][AsyncSock         ] [se1.farm.particle.cz:1094.0] Closing the socket
lar exit code 1
=== Start last 100 lines of lar log file ===
tail: cannot open '_reco_20250808T032908Z.log' for reading: No such file or directory
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a6/0b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74540126_733_20231207T064706Z_gen_g4_detsim_hitreco__20240510T055848Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/65/22/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_46_20231205T071817Z_gen_g4_detsim_hitreco__20240510T035231Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c5/04/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50253389_5_20231120T123600Z_gen_g4_detsim_hitreco__20240503T062514Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/2d/bc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_947_20231203T124140Z_gen_g4_detsim_hitreco__20240509T201428Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d4/aa/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_602_20231202T162415Z_gen_g4_detsim_hitreco__20240508T044114Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d9/37/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50579427_791_20231204T045537Z_gen_g4_detsim_hitreco__20240509T211025Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/8c/cc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571580_986_20231203T030504Z_gen_g4_detsim_hitreco__20240508T072401Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/36/c0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_962_20231205T080610Z_gen_g4_detsim_hitreco__20240509T222129Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/05/da/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6470038_159_20231207T163055Z_gen_g4_detsim_hitreco__20240510T030113Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/44/fb/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_437_20231203T042358Z_gen_g4_detsim_hitreco__20240508T045227Z_reco2.root
.:
total 65160
-rw-r--r-- 1 gl05pi6 eddie_users 44024717 Aug  8 04:48 jobscript.log
-rw-r--r-- 1 gl05pi6 eddie_users  7178013 Aug  8 04:48 flatcaf.root
-rw-r--r-- 1 gl05pi6 eddie_users  5920420 Aug  8 04:48 caf.root
-rw-r--r-- 1 gl05pi6 eddie_users  5920420 Aug  8 04:48 caf_fd_hd_atmo_649_20250808T032908Z.root
-rw-r--r-- 1 gl05pi6 eddie_users  3623034 Aug  8 04:48 caf_20250808T032908Z.log
-rw-r--r-- 1 gl05pi6 eddie_users     1884 Aug  8 04:48 caf_20250808T032908Z.file
-rw-r--r-- 1 gl05pi6 eddie_users     1884 Aug  8 04:48 caf_20250808T032908Z.pfns
-rw-r--r-- 1 gl05pi6 eddie_users     1884 Aug  8 04:29 file.list
-rw-r--r-- 1 gl05pi6 eddie_users     1884 Aug  8 04:48 justin-processed-pfns.txt
-rw-r--r-- 1 gl05pi6 eddie_users     1374 Aug  8 04:29 all-input-dids.txt
-rw-r--r-- 1 gl05pi6 eddie_users     1374 Aug  8 04:48 caf_20250808T032908Z.did
-rw-r--r-- 1 gl05pi6 eddie_users     1374 Aug  8 04:29 did.list