Jobsub ID 21919.6@dunegpschedd01.fnal.gov

Jobsub ID	21919.6@dunegpschedd01.fnal.gov
Workflow ID	649
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-08-08 03:54:28
Site	UK_Edinburgh
Entry	DUNE_UK_SGridECDF_ce1
Last heartbeat	2025-08-08 04:17:59
From worker node	Hostname	node2b04.ecdf.ed.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-08-08 03:55:45
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74568471_322_20231207T153504Z_gen_g4_detsim_hitreco__20240510T030037Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_46_20231205T071817Z_gen_g4_detsim_hitreco__20240510T035231Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515284_132_20231201T114106Z_gen_g4_detsim_hitreco__20240507T185548Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_437_20231203T042358Z_gen_g4_detsim_hitreco__20240508T045227Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_466_20231208T055947Z_gen_g4_detsim_hitreco__20240510T050420Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74508011_752_20231203T051611Z_gen_g4_detsim_hitreco__20240508T074806Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74508011_76_20231203T025010Z_gen_g4_detsim_hitreco__20240508T074954Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-08-08 04:17:59
Saved logs	justin-logs:21919.6-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

handle 0x0 to dpmpool23.farm.particle.cz:22880
[2025-08-08 05:17:40.504630 +0100][Debug  ][ExDbgMsg          ] [dpmpool23.farm.particle.cz:22880] MsgHandler created: 0xbf41c40 (message: kXR_close (handle: 0x00000000) ).
[2025-08-08 05:17:40.504682 +0100][Debug  ][ExDbgMsg          ] [dpmpool23.farm.particle.cz:22880] Moving MsgHandler: 0xbf41c40 (message: kXR_close (handle: 0x00000000) ) from out-queu to in-queue.
[2025-08-08 05:17:40.536500 +0100][Debug  ][ExDbgMsg          ] [msg: 0xa62eb20] Assigned MsgHandler: 0xbf41c40.
[2025-08-08 05:17:40.536527 +0100][Debug  ][ExDbgMsg          ] [handler: 0xbf41c40] Removed MsgHandler: 0xbf41c40 from the in-queue.
[2025-08-08 05:17:40.536554 +0100][Debug  ][ExDbgMsg          ] [dpmpool23.farm.particle.cz:22880] Calling MsgHandler: 0xbf41c40 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] .
[2025-08-08 05:17:40.536571 +0100][Debug  ][File              ] [0xb101a90@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/48/f7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_466_20231208T055947Z_gen_g4_detsim_hitreco__20240510T050420Z_reco2.root?xrdcl.requuid=1cd06098-e5c2-46f3-a5e3-e95c5ef62249] Close returned from dpmpool23.farm.particle.cz:22880 with: [SUCCESS] 
[2025-08-08 05:17:40.536584 +0100][Debug  ][ExDbgMsg          ] [dpmpool23.farm.particle.cz:22880] Destroying MsgHandler: 0xbf41c40.
[2025-08-08 05:17:40.538013 +0100][Debug  ][Utility           ] Env: overriding entry: connectionwindow=30 with 30
[2025-08-08 05:17:40.538028 +0100][Debug  ][Utility           ] Env: overriding entry: requesttimeout=14400 with 4096
[2025-08-08 05:17:40.538036 +0100][Debug  ][Utility           ] Env: overriding entry: requesttimeout=4096 with 14400
[2025-08-08 05:17:40.538051 +0100][Debug  ][Utility           ] Env: overriding entry: redirectlimit=64 with 64
[2025-08-08 05:17:40.538065 +0100][Debug  ][Utility           ] Env: overriding entry: multiprotocol=1 with 1
[2025-08-08 05:17:40.538151 +0100][Debug  ][File              ] [0x189a7130@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/40/27/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74508011_752_20231203T051611Z_gen_g4_detsim_hitreco__20240508T074806Z_reco2.root?xrdcl.requuid=f2b0e485-2472-4a89-9337-d96dbd353635] Sending an open command
[2025-08-08 05:17:40.538170 +0100][Debug  ][ExDbgMsg          ] [se1.farm.particle.cz:1094] MsgHandler created: 0xbf41c40 (message: kXR_open (file: /dune/RSE/fardet-hd/40/27/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74508011_752_20231203T051611Z_gen_g4_detsim_hitreco__20240508T074806Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ).
[2025-08-08 05:17:40.538212 +0100][Debug  ][ExDbgMsg          ] [se1.farm.particle.cz:1094] Moving MsgHandler: 0xbf41c40 (message: kXR_open (file: /dune/RSE/fardet-hd/40/27/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74508011_752_20231203T051611Z_gen_g4_detsim_hitreco__20240508T074806Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) from out-queu to in-queue.
[2025-08-08 05:17:40.575233 +0100][Debug  ][ExDbgMsg          ] [msg: 0x19a4b310] Assigned MsgHandler: 0xbf41c40.
[2025-08-08 05:17:40.575240 +0100][Debug  ][ExDbgMsg          ] [handler: 0xbf41c40] Removed MsgHandler: 0xbf41c40 from the in-queue.
[2025-08-08 05:17:40.575265 +0100][Debug  ][XRootD            ] [se1.farm.particle.cz:1094] Handling error while processing kXR_open (file: /dune/RSE/fardet-hd/40/27/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74508011_752_20231203T051611Z_gen_g4_detsim_hitreco__20240508T074806Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [ERROR] Error response: no such file or directory.
[2025-08-08 05:17:40.575610 +0100][Debug  ][ExDbgMsg          ] [se1.farm.particle.cz:1094] Calling MsgHandler: 0xbf41c40 (message: kXR_open (file: /dune/RSE/fardet-hd/40/27/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74508011_752_20231203T051611Z_gen_g4_detsim_hitreco__20240508T074806Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) with status: [ERROR] Error response: no such file or directory.
[2025-08-08 05:17:40.575691 +0100][Debug  ][File              ] [0x189a7130@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/40/27/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74508011_752_20231203T051611Z_gen_g4_detsim_hitreco__20240508T074806Z_reco2.root?xrdcl.requuid=f2b0e485-2472-4a89-9337-d96dbd353635] Open has returned with status [ERROR] Server responded with an error: [3011] No such file
[2025-08-08 05:17:40.575701 +0100][Debug  ][File              ] [0x189a7130@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/40/27/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74508011_752_20231203T051611Z_gen_g4_detsim_hitreco__20240508T074806Z_reco2.root?xrdcl.requuid=f2b0e485-2472-4a89-9337-d96dbd353635] Error while opening at se1.farm.particle.cz:1094: [ERROR] Server responded with an error: [3011] No such file
[2025-08-08 05:17:40.575761 +0100][Debug  ][ExDbgMsg          ] [se1.farm.particle.cz:1094] Destroying MsgHandler: 0xbf41c40.
[2025-08-08 05:17:40.829130 +0100][Debug  ][JobMgr            ] Stopping the job manager...
[2025-08-08 05:17:40.829444 +0100][Debug  ][JobMgr            ] Job manager stopped
[2025-08-08 05:17:40.829504 +0100][Debug  ][TaskMgr           ] Stopping the task manager...
[2025-08-08 05:17:40.829595 +0100][Debug  ][TaskMgr           ] Task manager stopped
[2025-08-08 05:17:40.829617 +0100][Debug  ][Poller            ] Stopping the poller...
[2025-08-08 05:17:40.829877 +0100][Debug  ][AsyncSock         ] [dpmpool23.farm.particle.cz:22880.0] Closing the socket
[2025-08-08 05:17:40.829895 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.37]:57862><--><[::ffff:147.231.25.94]:22880> Removing socket from the poller
[2025-08-08 05:17:40.829918 +0100][Debug  ][PostMaster        ] [dpmpool23.farm.particle.cz:22880] Destroying stream
[2025-08-08 05:17:40.829924 +0100][Debug  ][AsyncSock         ] [dpmpool23.farm.particle.cz:22880.0] Closing the socket
[2025-08-08 05:17:40.829941 +0100][Debug  ][AsyncSock         ] [dpmpool24.farm.particle.cz:23458.0] Closing the socket
[2025-08-08 05:17:40.829948 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.37]:37930><--><[::ffff:147.231.25.95]:23458> Removing socket from the poller
[2025-08-08 05:17:40.829963 +0100][Debug  ][PostMaster        ] [dpmpool24.farm.particle.cz:23458] Destroying stream
[2025-08-08 05:17:40.829967 +0100][Debug  ][AsyncSock         ] [dpmpool24.farm.particle.cz:23458.0] Closing the socket
[2025-08-08 05:17:40.829979 +0100][Debug  ][AsyncSock         ] [dune.dcache.nikhef.nl:1094.0] Closing the socket
[2025-08-08 05:17:40.829988 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.37]:33858><--><[::ffff:194.171.96.149]:1094> Removing socket from the poller
[2025-08-08 05:17:40.829999 +0100][Debug  ][PostMaster        ] [dune.dcache.nikhef.nl:1094] Destroying stream
[2025-08-08 05:17:40.830005 +0100][Debug  ][AsyncSock         ] [dune.dcache.nikhef.nl:1094.0] Closing the socket
[2025-08-08 05:17:40.830015 +0100][Debug  ][AsyncSock         ] [se1.farm.particle.cz:1094.0] Closing the socket
[2025-08-08 05:17:40.830024 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.37]:38360><--><[::ffff:147.231.25.100]:1094> Removing socket from the poller
[2025-08-08 05:17:40.830035 +0100][Debug  ][PostMaster        ] [se1.farm.particle.cz:1094] Destroying stream
[2025-08-08 05:17:40.830039 +0100][Debug  ][AsyncSock         ] [se1.farm.particle.cz:1094.0] Closing the socket
lar exit code 1
=== Start last 100 lines of lar log file ===
tail: cannot open '_reco_20250808T035552Z.log' for reading: No such file or directory
=== End last 100 lines of lar log file ===
processed files
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/57/50/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74568471_322_20231207T153504Z_gen_g4_detsim_hitreco__20240510T030037Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/65/22/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_46_20231205T071817Z_gen_g4_detsim_hitreco__20240510T035231Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/8d/cd/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50515284_132_20231201T114106Z_gen_g4_detsim_hitreco__20240507T185548Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/44/fb/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_437_20231203T042358Z_gen_g4_detsim_hitreco__20240508T045227Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/48/f7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_466_20231208T055947Z_gen_g4_detsim_hitreco__20240510T050420Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/40/27/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74508011_752_20231203T051611Z_gen_g4_detsim_hitreco__20240508T074806Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/f9/ce/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74508011_76_20231203T025010Z_gen_g4_detsim_hitreco__20240508T074954Z_reco2.root
.:
total 81796
-rw-r--r-- 1 gl05pi6 eddie_users 54952114 Aug  8 05:17 jobscript.log
-rw-r--r-- 1 gl05pi6 eddie_users  9111813 Aug  8 05:17 flatcaf.root
-rw-r--r-- 1 gl05pi6 eddie_users  7547346 Aug  8 05:17 caf.root
-rw-r--r-- 1 gl05pi6 eddie_users  7547346 Aug  8 05:17 caf_fd_hd_atmo_649_20250808T035552Z.root
-rw-r--r-- 1 gl05pi6 eddie_users  4546204 Aug  8 05:17 caf_20250808T035552Z.log
-rw-r--r-- 1 gl05pi6 eddie_users     1335 Aug  8 05:17 caf_20250808T035552Z.file
-rw-r--r-- 1 gl05pi6 eddie_users     1335 Aug  8 05:17 caf_20250808T035552Z.pfns
-rw-r--r-- 1 gl05pi6 eddie_users     1335 Aug  8 04:55 file.list
-rw-r--r-- 1 gl05pi6 eddie_users     1335 Aug  8 05:17 justin-processed-pfns.txt
-rw-r--r-- 1 gl05pi6 eddie_users      962 Aug  8 04:55 all-input-dids.txt
-rw-r--r-- 1 gl05pi6 eddie_users      962 Aug  8 05:17 caf_20250808T035552Z.did
-rw-r--r-- 1 gl05pi6 eddie_users      962 Aug  8 04:55 did.list