Jobsub ID 15367.21@dunegpschedd02.fnal.gov

Jobsub ID	15367.21@dunegpschedd02.fnal.gov
Workflow ID	648
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-08-08 01:42:21
Site	UK_Edinburgh
Entry	DUNE_UK_SGridECDF_ce1
Last heartbeat	2025-08-08 01:52:27
From worker node	Hostname	node2b08.ecdf.ed.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-08-08 01:43:46
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494003_586_20231201T170338Z_gen_g4_detsim_hitreco__20240507T201551Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_608_20231207T120529Z_gen_g4_detsim_hitreco__20240510T055437Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6468412_299_20231207T075006Z_gen_g4_detsim_hitreco__20240510T060802Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_1111_20231203T125003Z_gen_g4_detsim_hitreco__20240509T192323Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74581430_697_20231208T082350Z_gen_g4_detsim_hitreco__20240510T062742Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_984_20231202T140222Z_gen_g4_detsim_hitreco__20240507T223606Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419624_1189_20231204T032040Z_gen_g4_detsim_hitreco__20240509T212405Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_730_20231207T031744Z_gen_g4_detsim_hitreco__20240510T050721Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-08-08 01:52:27
Saved logs	justin-logs:15367.21-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

xDbgMsg          ] [dpmpool25.farm.particle.cz:20065] Moving MsgHandler: 0x174d1b50 (message: kXR_close (handle: 0x00000000) ) from out-queu to in-queue.
[2025-08-08 02:52:10.340393 +0100][Debug  ][ExDbgMsg          ] [msg: 0xc2eff40] Assigned MsgHandler: 0x174d1b50.
[2025-08-08 02:52:10.340401 +0100][Debug  ][ExDbgMsg          ] [handler: 0x174d1b50] Removed MsgHandler: 0x174d1b50 from the in-queue.
[2025-08-08 02:52:10.340424 +0100][Debug  ][ExDbgMsg          ] [dpmpool25.farm.particle.cz:20065] Calling MsgHandler: 0x174d1b50 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] .
[2025-08-08 02:52:10.340435 +0100][Debug  ][File              ] [0xc9fa9b0@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/9e/f5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_608_20231207T120529Z_gen_g4_detsim_hitreco__20240510T055437Z_reco2.root?xrdcl.requuid=d58bbd72-4967-483f-8af7-6c0cf3d037cb] Close returned from dpmpool25.farm.particle.cz:20065 with: [SUCCESS] 
[2025-08-08 02:52:10.340447 +0100][Debug  ][ExDbgMsg          ] [dpmpool25.farm.particle.cz:20065] Destroying MsgHandler: 0x174d1b50.
[2025-08-08 02:52:10.342654 +0100][Debug  ][Utility           ] Env: overriding entry: connectionwindow=30 with 30
[2025-08-08 02:52:10.342671 +0100][Debug  ][Utility           ] Env: overriding entry: requesttimeout=14400 with 4096
[2025-08-08 02:52:10.342680 +0100][Debug  ][Utility           ] Env: overriding entry: requesttimeout=4096 with 14400
[2025-08-08 02:52:10.342697 +0100][Debug  ][Utility           ] Env: overriding entry: redirectlimit=64 with 64
[2025-08-08 02:52:10.342713 +0100][Debug  ][Utility           ] Env: overriding entry: multiprotocol=1 with 1
[2025-08-08 02:52:10.342809 +0100][Debug  ][File              ] [0xc99d740@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/16/f0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6468412_299_20231207T075006Z_gen_g4_detsim_hitreco__20240510T060802Z_reco2.root?xrdcl.requuid=ff63d18a-3e66-49c0-ba4e-18ac96deb20f] Sending an open command
[2025-08-08 02:52:10.342827 +0100][Debug  ][ExDbgMsg          ] [se1.farm.particle.cz:1094] MsgHandler created: 0x174d1b50 (message: kXR_open (file: /dune/RSE/fardet-hd/16/f0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6468412_299_20231207T075006Z_gen_g4_detsim_hitreco__20240510T060802Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ).
[2025-08-08 02:52:10.342871 +0100][Debug  ][ExDbgMsg          ] [se1.farm.particle.cz:1094] Moving MsgHandler: 0x174d1b50 (message: kXR_open (file: /dune/RSE/fardet-hd/16/f0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6468412_299_20231207T075006Z_gen_g4_detsim_hitreco__20240510T060802Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) from out-queu to in-queue.
[2025-08-08 02:52:10.380590 +0100][Debug  ][ExDbgMsg          ] [msg: 0x14420f90] Assigned MsgHandler: 0x174d1b50.
[2025-08-08 02:52:10.380613 +0100][Debug  ][ExDbgMsg          ] [handler: 0x174d1b50] Removed MsgHandler: 0x174d1b50 from the in-queue.
[2025-08-08 02:52:10.380640 +0100][Debug  ][XRootD            ] [se1.farm.particle.cz:1094] Handling error while processing kXR_open (file: /dune/RSE/fardet-hd/16/f0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6468412_299_20231207T075006Z_gen_g4_detsim_hitreco__20240510T060802Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [ERROR] Error response: no such file or directory.
[2025-08-08 02:52:10.381039 +0100][Debug  ][ExDbgMsg          ] [se1.farm.particle.cz:1094] Calling MsgHandler: 0x174d1b50 (message: kXR_open (file: /dune/RSE/fardet-hd/16/f0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6468412_299_20231207T075006Z_gen_g4_detsim_hitreco__20240510T060802Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) with status: [ERROR] Error response: no such file or directory.
[2025-08-08 02:52:10.381120 +0100][Debug  ][File              ] [0xc99d740@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/16/f0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6468412_299_20231207T075006Z_gen_g4_detsim_hitreco__20240510T060802Z_reco2.root?xrdcl.requuid=ff63d18a-3e66-49c0-ba4e-18ac96deb20f] Open has returned with status [ERROR] Server responded with an error: [3011] No such file
[2025-08-08 02:52:10.381130 +0100][Debug  ][File              ] [0xc99d740@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/16/f0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6468412_299_20231207T075006Z_gen_g4_detsim_hitreco__20240510T060802Z_reco2.root?xrdcl.requuid=ff63d18a-3e66-49c0-ba4e-18ac96deb20f] Error while opening at se1.farm.particle.cz:1094: [ERROR] Server responded with an error: [3011] No such file
[2025-08-08 02:52:10.381187 +0100][Debug  ][ExDbgMsg          ] [se1.farm.particle.cz:1094] Destroying MsgHandler: 0x174d1b50.
[2025-08-08 02:52:10.682687 +0100][Debug  ][JobMgr            ] Stopping the job manager...
[2025-08-08 02:52:10.683004 +0100][Debug  ][JobMgr            ] Job manager stopped
[2025-08-08 02:52:10.683065 +0100][Debug  ][TaskMgr           ] Stopping the task manager...
[2025-08-08 02:52:10.683161 +0100][Debug  ][TaskMgr           ] Task manager stopped
[2025-08-08 02:52:10.683167 +0100][Debug  ][Poller            ] Stopping the poller...
[2025-08-08 02:52:10.683666 +0100][Debug  ][AsyncSock         ] [dpmpool25.farm.particle.cz:20065.0] Closing the socket
[2025-08-08 02:52:10.683798 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.41]:59236><--><[::ffff:147.231.25.70]:20065> Removing socket from the poller
[2025-08-08 02:52:10.683920 +0100][Debug  ][PostMaster        ] [dpmpool25.farm.particle.cz:20065] Destroying stream
[2025-08-08 02:52:10.684014 +0100][Debug  ][AsyncSock         ] [dpmpool25.farm.particle.cz:20065.0] Closing the socket
[2025-08-08 02:52:10.684072 +0100][Debug  ][AsyncSock         ] [dune.dcache.nikhef.nl:1094.0] Closing the socket
[2025-08-08 02:52:10.684087 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.41]:52344><--><[::ffff:194.171.96.149]:1094> Removing socket from the poller
[2025-08-08 02:52:10.684101 +0100][Debug  ][PostMaster        ] [dune.dcache.nikhef.nl:1094] Destroying stream
[2025-08-08 02:52:10.684105 +0100][Debug  ][AsyncSock         ] [dune.dcache.nikhef.nl:1094.0] Closing the socket
[2025-08-08 02:52:10.684116 +0100][Debug  ][AsyncSock         ] [hooipolder-04.nikhef.nl:33207.0] Closing the socket
[2025-08-08 02:52:10.684120 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.41]:51938><--><[::ffff:194.171.96.152]:33207> Removing socket from the poller
[2025-08-08 02:52:10.684130 +0100][Debug  ][PostMaster        ] [hooipolder-04.nikhef.nl:33207] Destroying stream
[2025-08-08 02:52:10.684133 +0100][Debug  ][AsyncSock         ] [hooipolder-04.nikhef.nl:33207.0] Closing the socket
[2025-08-08 02:52:10.684143 +0100][Debug  ][AsyncSock         ] [se1.farm.particle.cz:1094.0] Closing the socket
[2025-08-08 02:52:10.684148 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.41]:33638><--><[::ffff:147.231.25.100]:1094> Removing socket from the poller
[2025-08-08 02:52:10.684169 +0100][Debug  ][PostMaster        ] [se1.farm.particle.cz:1094] Destroying stream
[2025-08-08 02:52:10.684173 +0100][Debug  ][AsyncSock         ] [se1.farm.particle.cz:1094.0] Closing the socket
lar exit code 1
=== Start last 100 lines of lar log file ===
tail: cannot open '_reco_20250808T014355Z.log' for reading: No such file or directory
=== End last 100 lines of lar log file ===
processed files
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/bc/b7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74494003_586_20231201T170338Z_gen_g4_detsim_hitreco__20240507T201551Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/9e/f5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_608_20231207T120529Z_gen_g4_detsim_hitreco__20240510T055437Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/16/f0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6468412_299_20231207T075006Z_gen_g4_detsim_hitreco__20240510T060802Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/8f/b4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_1111_20231203T125003Z_gen_g4_detsim_hitreco__20240509T192323Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/6e/d0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74581430_697_20231208T082350Z_gen_g4_detsim_hitreco__20240510T062742Z_reco2.root
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/45/bb/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_984_20231202T140222Z_gen_g4_detsim_hitreco__20240507T223606Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/00/ab/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419624_1189_20231204T032040Z_gen_g4_detsim_hitreco__20240509T212405Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/09/dc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_730_20231207T031744Z_gen_g4_detsim_hitreco__20240510T050721Z_reco2.root
.:
total 37552
-rw-r--r-- 1 gl05pi6 eddie_users 21911626 Aug  8 02:52 jobscript.log
-rw-r--r-- 1 gl05pi6 eddie_users  5532619 Aug  8 02:52 flatcaf.root
-rw-r--r-- 1 gl05pi6 eddie_users  4536403 Aug  8 02:52 caf.root
-rw-r--r-- 1 gl05pi6 eddie_users  4536403 Aug  8 02:52 caf_fd_hd_atmo_648_20250808T014355Z.root
-rw-r--r-- 1 gl05pi6 eddie_users  1882791 Aug  8 02:52 caf_20250808T014355Z.log
-rw-r--r-- 1 gl05pi6 eddie_users     1604 Aug  8 02:52 caf_20250808T014355Z.file
-rw-r--r-- 1 gl05pi6 eddie_users     1604 Aug  8 02:52 caf_20250808T014355Z.pfns
-rw-r--r-- 1 gl05pi6 eddie_users     1604 Aug  8 02:43 file.list
-rw-r--r-- 1 gl05pi6 eddie_users     1604 Aug  8 02:52 justin-processed-pfns.txt
-rw-r--r-- 1 gl05pi6 eddie_users     1102 Aug  8 02:43 all-input-dids.txt
-rw-r--r-- 1 gl05pi6 eddie_users     1102 Aug  8 02:52 caf_20250808T014355Z.did
-rw-r--r-- 1 gl05pi6 eddie_users     1102 Aug  8 02:43 did.list