Jobsub ID 15495.4@dunegpschedd02.fnal.gov

Jobsub ID	15495.4@dunegpschedd02.fnal.gov
Workflow ID	650
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-08-08 06:34:36
Site	UK_Edinburgh
Entry	DUNE_UK_SGridECDF_ce1
Last heartbeat	2025-08-08 06:39:22
From worker node	Hostname	node2b08.ecdf.ed.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-08-08 06:35:56
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6404651_244_20231202T093135Z_gen_g4_detsim_hitreco__20240507T214220Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6402377_697_20231202T111356Z_gen_g4_detsim_hitreco__20240507T211103Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_332_20231205T024539Z_gen_g4_detsim_hitreco__20240509T221331Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66027687_480_20231207T003136Z_gen_g4_detsim_hitreco__20240510T061701Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6403947_550_20231202T121116Z_gen_g4_detsim_hitreco__20240507T212505Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_389_20231202T180518Z_gen_g4_detsim_hitreco__20240508T063718Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50231162_142_20231118T173342Z_gen_g4_detsim_hitreco__20240503T051724Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_477_20231122T063338Z_gen_g4_detsim_hitreco__20240507T171325Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_239_20231202T202525Z_gen_g4_detsim_hitreco__20240508T061713Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50261805_342_20231122T120832Z_gen_g4_detsim_hitreco__20240503T055655Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-08-08 06:39:22
Saved logs	justin-logs:15495.4-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ostMaster        ] [se1.farm.particle.cz:1094] Stream parameters: Network Stack: IPAuto, Connection Window: 30, ConnectionRetry: 5, Stream Error Window: 1800
[2025-08-08 07:39:05.796828 +0100][Debug  ][TaskMgr           ] Registering task: "TickGeneratorTask for: root://se1.farm.particle.cz:1094/" to be run at: [2025-08-08 07:39:20 +0100]
[2025-08-08 07:39:05.797307 +0100][Debug  ][PostMaster        ] [se1.farm.particle.cz:1094] Found 1 address(es): [::ffff:147.231.25.100]:1094
[2025-08-08 07:39:05.797321 +0100][Debug  ][AsyncSock         ] [se1.farm.particle.cz:1094.0] Attempting connection to [::ffff:147.231.25.100]:1094
[2025-08-08 07:39:05.797341 +0100][Debug  ][Poller            ] Adding socket 0xd2c5690 to the poller
[2025-08-08 07:39:05.828997 +0100][Debug  ][AsyncSock         ] [se1.farm.particle.cz:1094.0] Async connection call returned
[2025-08-08 07:39:05.829009 +0100][Debug  ][XRootDTransport   ] [se1.farm.particle.cz:1094.0] Sending out the initial hand shake + kXR_protocol
[2025-08-08 07:39:05.860747 +0100][Debug  ][XRootDTransport   ] [se1.farm.particle.cz:1094.0] Got the server hand shake response (type: manager [], protocol version 500)
[2025-08-08 07:39:05.860883 +0100][Debug  ][XRootDTransport   ] [se1.farm.particle.cz:1094.0] kXR_protocol successful (type: manager [], protocol version 500)
[2025-08-08 07:39:05.861859 +0100][Debug  ][XRootDTransport   ] [se1.farm.particle.cz:1094.0] Sending out kXR_login request, username: gl05pi6, cgi: xrd.cc=uk&xrd.tz=0&xrd.appname=lar&xrd.info=&xrd.hostname=node2b08.ecdf.ed.ac.uk&xrd.rn=v5.5.5, dual-stack: false, private IPv4: false, private IPv6: false
[2025-08-08 07:39:05.893608 +0100][Debug  ][XRootDTransport   ] [se1.farm.particle.cz:1094.0] Logged in, session: 11adbd4855b9298c5cd1b414b3059bb6
[2025-08-08 07:39:05.893615 +0100][Debug  ][XRootDTransport   ] [se1.farm.particle.cz:1094.0] Authentication is required: &P=gsi,v:10400,c:ssl,ca:9c979c2b&P=unix
[2025-08-08 07:39:05.893627 +0100][Debug  ][XRootDTransport   ] [se1.farm.particle.cz:1094.0] Sending authentication data
[2025-08-08 07:39:05.893656 +0100][Debug  ][XRootDTransport   ] [se1.farm.particle.cz:1094.0] Trying to authenticate using gsi
[2025-08-08 07:39:05.945618 +0100][Debug  ][XRootDTransport   ] [se1.farm.particle.cz:1094.0] Sending more authentication data for gsi
[2025-08-08 07:39:05.997465 +0100][Debug  ][XRootDTransport   ] [se1.farm.particle.cz:1094.0] Authenticated with gsi.
[2025-08-08 07:39:05.997484 +0100][Debug  ][PostMaster        ] [se1.farm.particle.cz:1094] Stream 0 connected (IPv4).
[2025-08-08 07:39:05.997504 +0100][Debug  ][ExDbgMsg          ] [se1.farm.particle.cz:1094] Moving MsgHandler: 0x1f0cf750 (message: kXR_open (file: /dune/RSE/fardet-hd/5e/d0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6402377_697_20231202T111356Z_gen_g4_detsim_hitreco__20240507T211103Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) from out-queu to in-queue.
[2025-08-08 07:39:06.100441 +0100][Debug  ][ExDbgMsg          ] [msg: 0x8275570] Assigned MsgHandler: 0x1f0cf750.
[2025-08-08 07:39:06.100473 +0100][Debug  ][ExDbgMsg          ] [handler: 0x1f0cf750] Removed MsgHandler: 0x1f0cf750 from the in-queue.
[2025-08-08 07:39:06.100515 +0100][Debug  ][XRootD            ] [se1.farm.particle.cz:1094] Handling error while processing kXR_open (file: /dune/RSE/fardet-hd/5e/d0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6402377_697_20231202T111356Z_gen_g4_detsim_hitreco__20240507T211103Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [ERROR] Error response: no such file or directory.
[2025-08-08 07:39:06.100688 +0100][Debug  ][ExDbgMsg          ] [se1.farm.particle.cz:1094] Calling MsgHandler: 0x1f0cf750 (message: kXR_open (file: /dune/RSE/fardet-hd/5e/d0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6402377_697_20231202T111356Z_gen_g4_detsim_hitreco__20240507T211103Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) with status: [ERROR] Error response: no such file or directory.
[2025-08-08 07:39:06.100758 +0100][Debug  ][File              ] [0xd65f750@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/5e/d0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6402377_697_20231202T111356Z_gen_g4_detsim_hitreco__20240507T211103Z_reco2.root?xrdcl.requuid=3a544a09-8cd7-4042-85fd-47a8d59e063f] Open has returned with status [ERROR] Server responded with an error: [3011] No such file
[2025-08-08 07:39:06.100767 +0100][Debug  ][File              ] [0xd65f750@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/5e/d0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6402377_697_20231202T111356Z_gen_g4_detsim_hitreco__20240507T211103Z_reco2.root?xrdcl.requuid=3a544a09-8cd7-4042-85fd-47a8d59e063f] Error while opening at se1.farm.particle.cz:1094: [ERROR] Server responded with an error: [3011] No such file
[2025-08-08 07:39:06.100814 +0100][Debug  ][ExDbgMsg          ] [se1.farm.particle.cz:1094] Destroying MsgHandler: 0x1f0cf750.
[2025-08-08 07:39:06.312150 +0100][Debug  ][JobMgr            ] Stopping the job manager...
[2025-08-08 07:39:06.312438 +0100][Debug  ][JobMgr            ] Job manager stopped
[2025-08-08 07:39:06.312490 +0100][Debug  ][TaskMgr           ] Stopping the task manager...
[2025-08-08 07:39:06.312575 +0100][Debug  ][TaskMgr           ] Task manager stopped
[2025-08-08 07:39:06.312581 +0100][Debug  ][Poller            ] Stopping the poller...
[2025-08-08 07:39:06.312973 +0100][Debug  ][AsyncSock         ] [dune.dcache.nikhef.nl:1094.0] Closing the socket
[2025-08-08 07:39:06.313063 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.41]:56242><--><[::ffff:194.171.96.149]:1094> Removing socket from the poller
[2025-08-08 07:39:06.313168 +0100][Debug  ][PostMaster        ] [dune.dcache.nikhef.nl:1094] Destroying stream
[2025-08-08 07:39:06.313241 +0100][Debug  ][AsyncSock         ] [dune.dcache.nikhef.nl:1094.0] Closing the socket
[2025-08-08 07:39:06.313285 +0100][Debug  ][AsyncSock         ] [hooipolder-04.nikhef.nl:33207.0] Closing the socket
[2025-08-08 07:39:06.313300 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.41]:47070><--><[::ffff:194.171.96.152]:33207> Removing socket from the poller
[2025-08-08 07:39:06.313313 +0100][Debug  ][PostMaster        ] [hooipolder-04.nikhef.nl:33207] Destroying stream
[2025-08-08 07:39:06.313317 +0100][Debug  ][AsyncSock         ] [hooipolder-04.nikhef.nl:33207.0] Closing the socket
[2025-08-08 07:39:06.313326 +0100][Debug  ][AsyncSock         ] [se1.farm.particle.cz:1094.0] Closing the socket
[2025-08-08 07:39:06.313330 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.41]:36182><--><[::ffff:147.231.25.100]:1094> Removing socket from the poller
[2025-08-08 07:39:06.313341 +0100][Debug  ][PostMaster        ] [se1.farm.particle.cz:1094] Destroying stream
[2025-08-08 07:39:06.313346 +0100][Debug  ][AsyncSock         ] [se1.farm.particle.cz:1094.0] Closing the socket
lar exit code 1
=== Start last 100 lines of lar log file ===
tail: cannot open '_reco_20250808T063605Z.log' for reading: No such file or directory
=== End last 100 lines of lar log file ===
processed files
root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/15/73/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6404651_244_20231202T093135Z_gen_g4_detsim_hitreco__20240507T214220Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/5e/d0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6402377_697_20231202T111356Z_gen_g4_detsim_hitreco__20240507T211103Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/08/f4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_332_20231205T024539Z_gen_g4_detsim_hitreco__20240509T221331Z_reco2.root
root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/05/3f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66027687_480_20231207T003136Z_gen_g4_detsim_hitreco__20240510T061701Z_reco2.root
root://otter12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/68/4e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6403947_550_20231202T121116Z_gen_g4_detsim_hitreco__20240507T212505Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/29/59/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_389_20231202T180518Z_gen_g4_detsim_hitreco__20240508T063718Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c1/03/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50231162_142_20231118T173342Z_gen_g4_detsim_hitreco__20240503T051724Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b6/3d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_477_20231122T063338Z_gen_g4_detsim_hitreco__20240507T171325Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/d1/53/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6413290_239_20231202T202525Z_gen_g4_detsim_hitreco__20240508T061713Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/71/ea/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50261805_342_20231122T120832Z_gen_g4_detsim_hitreco__20240503T055655Z_reco2.root
.:
total 16828
-rw-r--r-- 1 gl05pi6 eddie_users 11005017 Aug  8 07:39 jobscript.log
-rw-r--r-- 1 gl05pi6 eddie_users  2069826 Aug  8 07:39 flatcaf.root
-rw-r--r-- 1 gl05pi6 eddie_users  1595928 Aug  8 07:39 caf.root
-rw-r--r-- 1 gl05pi6 eddie_users  1595928 Aug  8 07:39 caf_fd_hd_atmo_650_20250808T063605Z.root
-rw-r--r-- 1 gl05pi6 eddie_users   916965 Aug  8 07:39 caf_20250808T063605Z.log
-rw-r--r-- 1 gl05pi6 eddie_users     1948 Aug  8 07:39 caf_20250808T063605Z.file
-rw-r--r-- 1 gl05pi6 eddie_users     1948 Aug  8 07:39 caf_20250808T063605Z.pfns
-rw-r--r-- 1 gl05pi6 eddie_users     1948 Aug  8 07:36 file.list
-rw-r--r-- 1 gl05pi6 eddie_users     1948 Aug  8 07:39 justin-processed-pfns.txt
-rw-r--r-- 1 gl05pi6 eddie_users     1376 Aug  8 07:36 all-input-dids.txt
-rw-r--r-- 1 gl05pi6 eddie_users     1376 Aug  8 07:39 caf_20250808T063605Z.did
-rw-r--r-- 1 gl05pi6 eddie_users     1376 Aug  8 07:36 did.list