Jobsub ID 21919.1@dunegpschedd01.fnal.gov

Jobsub ID	21919.1@dunegpschedd01.fnal.gov
Workflow ID	649
Stage ID	1
User name	pgranger@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	18000 (5 hours)
Submitted time	2025-08-08 03:54:28
Site	UK_Edinburgh
Entry	DUNE_UK_SGridECDF_ce1
Last heartbeat	2025-08-08 04:00:01
From worker node	Hostname	node2b08.ecdf.ed.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-08-08 03:55:44
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_545_20231203T045642Z_gen_g4_detsim_hitreco__20240508T050403Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_780_20231205T054444Z_gen_g4_detsim_hitreco__20240510T022735Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74540126_733_20231207T064706Z_gen_g4_detsim_hitreco__20240510T055848Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50253389_5_20231120T123600Z_gen_g4_detsim_hitreco__20240503T062514Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_947_20231203T124140Z_gen_g4_detsim_hitreco__20240509T201428Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_690_20231205T110537Z_gen_g4_detsim_hitreco__20240510T042727Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_671_20231204T100452Z_gen_g4_detsim_hitreco__20240509T194758Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_626_20231204T074711Z_gen_g4_detsim_hitreco__20240509T215221Z_reco2.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-08-08 04:00:01
Saved logs	justin-logs:21919.1-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

] MsgHandler created: 0xaedf3a0 (message: kXR_open (file: /dune/RSE/fardet-hd/37/73/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_780_20231205T054444Z_gen_g4_detsim_hitreco__20240510T022735Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ).
[2025-08-08 04:59:45.936648 +0100][Debug  ][PostMaster        ] Creating new channel to: root://se1.farm.particle.cz:1094/
[2025-08-08 04:59:45.936823 +0100][Debug  ][PostMaster        ] [se1.farm.particle.cz:1094] Stream parameters: Network Stack: IPAuto, Connection Window: 30, ConnectionRetry: 5, Stream Error Window: 1800
[2025-08-08 04:59:45.936834 +0100][Debug  ][TaskMgr           ] Registering task: "TickGeneratorTask for: root://se1.farm.particle.cz:1094/" to be run at: [2025-08-08 05:00:00 +0100]
[2025-08-08 04:59:45.937356 +0100][Debug  ][PostMaster        ] [se1.farm.particle.cz:1094] Found 1 address(es): [::ffff:147.231.25.100]:1094
[2025-08-08 04:59:45.937372 +0100][Debug  ][AsyncSock         ] [se1.farm.particle.cz:1094.0] Attempting connection to [::ffff:147.231.25.100]:1094
[2025-08-08 04:59:45.937395 +0100][Debug  ][Poller            ] Adding socket 0x6ee98e0 to the poller
[2025-08-08 04:59:45.969032 +0100][Debug  ][AsyncSock         ] [se1.farm.particle.cz:1094.0] Async connection call returned
[2025-08-08 04:59:45.969047 +0100][Debug  ][XRootDTransport   ] [se1.farm.particle.cz:1094.0] Sending out the initial hand shake + kXR_protocol
[2025-08-08 04:59:46.000815 +0100][Debug  ][XRootDTransport   ] [se1.farm.particle.cz:1094.0] Got the server hand shake response (type: manager [], protocol version 500)
[2025-08-08 04:59:46.000951 +0100][Debug  ][XRootDTransport   ] [se1.farm.particle.cz:1094.0] kXR_protocol successful (type: manager [], protocol version 500)
[2025-08-08 04:59:46.001702 +0100][Debug  ][XRootDTransport   ] [se1.farm.particle.cz:1094.0] Sending out kXR_login request, username: gl05pi6, cgi: xrd.cc=uk&xrd.tz=0&xrd.appname=lar&xrd.info=&xrd.hostname=node2b08.ecdf.ed.ac.uk&xrd.rn=v5.5.5, dual-stack: false, private IPv4: false, private IPv6: false
[2025-08-08 04:59:46.035401 +0100][Debug  ][XRootDTransport   ] [se1.farm.particle.cz:1094.0] Logged in, session: 3d98dffe6220fe9c87865d5045058f68
[2025-08-08 04:59:46.035409 +0100][Debug  ][XRootDTransport   ] [se1.farm.particle.cz:1094.0] Authentication is required: &P=gsi,v:10400,c:ssl,ca:9c979c2b&P=unix
[2025-08-08 04:59:46.035413 +0100][Debug  ][XRootDTransport   ] [se1.farm.particle.cz:1094.0] Sending authentication data
[2025-08-08 04:59:46.035451 +0100][Debug  ][XRootDTransport   ] [se1.farm.particle.cz:1094.0] Trying to authenticate using gsi
[2025-08-08 04:59:46.087401 +0100][Debug  ][XRootDTransport   ] [se1.farm.particle.cz:1094.0] Sending more authentication data for gsi
[2025-08-08 04:59:46.135368 +0100][Debug  ][XRootDTransport   ] [se1.farm.particle.cz:1094.0] Authenticated with gsi.
[2025-08-08 04:59:46.135402 +0100][Debug  ][PostMaster        ] [se1.farm.particle.cz:1094] Stream 0 connected (IPv4).
[2025-08-08 04:59:46.135425 +0100][Debug  ][ExDbgMsg          ] [se1.farm.particle.cz:1094] Moving MsgHandler: 0xaedf3a0 (message: kXR_open (file: /dune/RSE/fardet-hd/37/73/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_780_20231205T054444Z_gen_g4_detsim_hitreco__20240510T022735Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) from out-queu to in-queue.
[2025-08-08 04:59:46.273604 +0100][Debug  ][ExDbgMsg          ] [msg: 0x16fcba50] Assigned MsgHandler: 0xaedf3a0.
[2025-08-08 04:59:46.273655 +0100][Debug  ][ExDbgMsg          ] [handler: 0xaedf3a0] Removed MsgHandler: 0xaedf3a0 from the in-queue.
[2025-08-08 04:59:46.273703 +0100][Debug  ][XRootD            ] [se1.farm.particle.cz:1094] Handling error while processing kXR_open (file: /dune/RSE/fardet-hd/37/73/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_780_20231205T054444Z_gen_g4_detsim_hitreco__20240510T022735Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [ERROR] Error response: no such file or directory.
[2025-08-08 04:59:46.273921 +0100][Debug  ][ExDbgMsg          ] [se1.farm.particle.cz:1094] Calling MsgHandler: 0xaedf3a0 (message: kXR_open (file: /dune/RSE/fardet-hd/37/73/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_780_20231205T054444Z_gen_g4_detsim_hitreco__20240510T022735Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) with status: [ERROR] Error response: no such file or directory.
[2025-08-08 04:59:46.274012 +0100][Debug  ][File              ] [0xb5af410@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/37/73/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_780_20231205T054444Z_gen_g4_detsim_hitreco__20240510T022735Z_reco2.root?xrdcl.requuid=abb6a971-3de8-42c0-b4a3-5335770fd084] Open has returned with status [ERROR] Server responded with an error: [3011] No such file
[2025-08-08 04:59:46.274035 +0100][Debug  ][File              ] [0xb5af410@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/37/73/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_780_20231205T054444Z_gen_g4_detsim_hitreco__20240510T022735Z_reco2.root?xrdcl.requuid=abb6a971-3de8-42c0-b4a3-5335770fd084] Error while opening at se1.farm.particle.cz:1094: [ERROR] Server responded with an error: [3011] No such file
[2025-08-08 04:59:46.274098 +0100][Debug  ][ExDbgMsg          ] [se1.farm.particle.cz:1094] Destroying MsgHandler: 0xaedf3a0.
[2025-08-08 04:59:46.484258 +0100][Debug  ][JobMgr            ] Stopping the job manager...
[2025-08-08 04:59:46.484580 +0100][Debug  ][JobMgr            ] Job manager stopped
[2025-08-08 04:59:46.484644 +0100][Debug  ][TaskMgr           ] Stopping the task manager...
[2025-08-08 04:59:46.484732 +0100][Debug  ][TaskMgr           ] Task manager stopped
[2025-08-08 04:59:46.484757 +0100][Debug  ][Poller            ] Stopping the poller...
[2025-08-08 04:59:46.485253 +0100][Debug  ][AsyncSock         ] [ceph-gw16.gridpp.rl.ac.uk:1094.0] Closing the socket
[2025-08-08 04:59:46.485355 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.41]:37906><--><[::ffff:130.246.178.214]:1094> Removing socket from the poller
[2025-08-08 04:59:46.485675 +0100][Debug  ][PostMaster        ] [ceph-gw16.gridpp.rl.ac.uk:1094] Destroying stream
[2025-08-08 04:59:46.485771 +0100][Debug  ][AsyncSock         ] [ceph-gw16.gridpp.rl.ac.uk:1094.0] Closing the socket
[2025-08-08 04:59:46.485867 +0100][Debug  ][AsyncSock         ] [se1.farm.particle.cz:1094.0] Closing the socket
[2025-08-08 04:59:46.485877 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.41]:35972><--><[::ffff:147.231.25.100]:1094> Removing socket from the poller
[2025-08-08 04:59:46.485892 +0100][Debug  ][PostMaster        ] [se1.farm.particle.cz:1094] Destroying stream
[2025-08-08 04:59:46.485896 +0100][Debug  ][AsyncSock         ] [se1.farm.particle.cz:1094.0] Closing the socket
[2025-08-08 04:59:46.485907 +0100][Debug  ][AsyncSock         ] [xrootd.echo.stfc.ac.uk:1094.0] Closing the socket
[2025-08-08 04:59:46.485931 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.41]:35116><--><[::ffff:130.246.217.9]:1094> Removing socket from the poller
[2025-08-08 04:59:46.486009 +0100][Debug  ][PostMaster        ] [xrootd.echo.stfc.ac.uk:1094] Destroying stream
[2025-08-08 04:59:46.486022 +0100][Debug  ][AsyncSock         ] [xrootd.echo.stfc.ac.uk:1094.0] Closing the socket
lar exit code 1
=== Start last 100 lines of lar log file ===
tail: cannot open '_reco_20250808T035551Z.log' for reading: No such file or directory
=== End last 100 lines of lar log file ===
processed files
root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/c4/8e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_545_20231203T045642Z_gen_g4_detsim_hitreco__20240508T050403Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/37/73/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_780_20231205T054444Z_gen_g4_detsim_hitreco__20240510T022735Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/a6/0b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74540126_733_20231207T064706Z_gen_g4_detsim_hitreco__20240510T055848Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c5/04/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50253389_5_20231120T123600Z_gen_g4_detsim_hitreco__20240503T062514Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/2d/bc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_947_20231203T124140Z_gen_g4_detsim_hitreco__20240509T201428Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/75/ef/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_690_20231205T110537Z_gen_g4_detsim_hitreco__20240510T042727Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/02/3f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517726_671_20231204T100452Z_gen_g4_detsim_hitreco__20240509T194758Z_reco2.root
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/57/c5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_626_20231204T074711Z_gen_g4_detsim_hitreco__20240509T215221Z_reco2.root
.:
total 14744
-rw-r--r-- 1 gl05pi6 eddie_users 10784981 Aug  8 04:59 jobscript.log
-rw-r--r-- 1 gl05pi6 eddie_users  1365631 Aug  8 04:59 flatcaf.root
-rw-r--r-- 1 gl05pi6 eddie_users   983137 Aug  8 04:59 caf.root
-rw-r--r-- 1 gl05pi6 eddie_users   983137 Aug  8 04:59 caf_fd_hd_atmo_649_20250808T035551Z.root
-rw-r--r-- 1 gl05pi6 eddie_users   922512 Aug  8 04:59 caf_20250808T035551Z.log
-rw-r--r-- 1 gl05pi6 eddie_users     1504 Aug  8 04:59 caf_20250808T035551Z.file
-rw-r--r-- 1 gl05pi6 eddie_users     1504 Aug  8 04:59 caf_20250808T035551Z.pfns
-rw-r--r-- 1 gl05pi6 eddie_users     1504 Aug  8 04:55 file.list
-rw-r--r-- 1 gl05pi6 eddie_users     1504 Aug  8 04:59 justin-processed-pfns.txt
-rw-r--r-- 1 gl05pi6 eddie_users     1100 Aug  8 04:55 all-input-dids.txt
-rw-r--r-- 1 gl05pi6 eddie_users     1100 Aug  8 04:59 caf_20250808T035551Z.did
-rw-r--r-- 1 gl05pi6 eddie_users     1100 Aug  8 04:55 did.list