Jobsub ID 15203.4@dunegpschedd02.fnal.gov
| Jobsub ID | 15203.4@dunegpschedd02.fnal.gov | |
| Workflow ID | 627 | |
| Stage ID | 1 | |
| User name | pgranger@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 18000 (5 hours) | |
| Submitted time | 2025-08-07 16:59:56 | |
| Site | UK_Edinburgh | |
| Entry | DUNE_UK_SGridECDF_ce1 | |
| Last heartbeat | 2025-08-07 17:01:00 | |
| From worker node | Hostname | node2b06.ecdf.ed.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 171000 (47 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Started | 2025-08-07 17:00:19 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_382_20231207T144404Z_gen_g4_detsim_hitreco__20240510T060953Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571580_410_20231203T022120Z_gen_g4_detsim_hitreco__20240508T071718Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6410210_1037_20231203T090813Z_gen_g4_detsim_hitreco__20240508T052722Z_reco2.root | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-08-07 17:01:00 | |
| Saved logs | justin-logs:15203.4-dunegpschedd02.fnal.gov.logs.tgz | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
18:00:42.985016 +0100][Debug ][Utility ] Env: overriding entry: requesttimeout=4096 with 14400 [2025-08-07 18:00:42.985032 +0100][Debug ][Utility ] Env: overriding entry: redirectlimit=16 with 64 [2025-08-07 18:00:42.985048 +0100][Debug ][Utility ] Env: overriding entry: multiprotocol=0 with 1 [2025-08-07 18:00:42.985476 +0100][Debug ][File ] [0x9db6800@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/31/b1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_382_20231207T144404Z_gen_g4_detsim_hitreco__20240510T060953Z_reco2.root?xrdcl.requuid=9d0b8085-da2b-4b51-bc2f-568d63d8e0be] Sending an open command [2025-08-07 18:00:42.986083 +0100][Debug ][Utility ] Env: trying to get a non-existent string entry: pollerpreference [2025-08-07 18:00:42.986190 +0100][Debug ][Poller ] Available pollers: built-in [2025-08-07 18:00:42.986194 +0100][Debug ][Poller ] Attempting to create a poller according to preference: built-in [2025-08-07 18:00:42.986199 +0100][Debug ][Poller ] Creating poller: built-in [2025-08-07 18:00:42.986294 +0100][Debug ][Poller ] Creating and starting the built-in poller... [2025-08-07 18:00:42.986825 +0100][Debug ][Poller ] Using 1 poller threads [2025-08-07 18:00:42.986872 +0100][Debug ][TaskMgr ] Starting the task manager... [2025-08-07 18:00:42.986916 +0100][Debug ][TaskMgr ] Task manager started [2025-08-07 18:00:42.986956 +0100][Debug ][JobMgr ] Starting the job manager... [2025-08-07 18:00:42.987069 +0100][Debug ][JobMgr ] Job manager started, 3 workers [2025-08-07 18:00:42.987170 +0100][Debug ][TaskMgr ] Registering task: "FileTimer task" to be run at: [2025-08-07 18:00:42 +0100] [2025-08-07 18:00:42.987521 +0100][Debug ][ExDbgMsg ] [se1.farm.particle.cz:1094] MsgHandler created: 0x9cb4900 (message: kXR_open (file: /dune/RSE/fardet-hd/31/b1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_382_20231207T144404Z_gen_g4_detsim_hitreco__20240510T060953Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ). [2025-08-07 18:00:42.988012 +0100][Debug ][PostMaster ] Creating new channel to: root://se1.farm.particle.cz:1094/ [2025-08-07 18:00:42.988335 +0100][Debug ][PostMaster ] [se1.farm.particle.cz:1094] Stream parameters: Network Stack: IPAuto, Connection Window: 30, ConnectionRetry: 5, Stream Error Window: 1800 [2025-08-07 18:00:42.988494 +0100][Debug ][TaskMgr ] Registering task: "TickGeneratorTask for: root://se1.farm.particle.cz:1094/" to be run at: [2025-08-07 18:00:57 +0100] [2025-08-07 18:00:42.989576 +0100][Debug ][PostMaster ] [se1.farm.particle.cz:1094] Found 1 address(es): [::ffff:147.231.25.100]:1094 [2025-08-07 18:00:42.989664 +0100][Debug ][AsyncSock ] [se1.farm.particle.cz:1094.0] Attempting connection to [::ffff:147.231.25.100]:1094 [2025-08-07 18:00:42.989744 +0100][Debug ][Poller ] Adding socket 0x9db5900 to the poller [2025-08-07 18:00:43.021579 +0100][Debug ][AsyncSock ] [se1.farm.particle.cz:1094.0] Async connection call returned [2025-08-07 18:00:43.021884 +0100][Debug ][XRootDTransport ] [se1.farm.particle.cz:1094.0] Sending out the initial hand shake + kXR_protocol [2025-08-07 18:00:43.054086 +0100][Debug ][XRootDTransport ] [se1.farm.particle.cz:1094.0] Got the server hand shake response (type: manager [], protocol version 500) [2025-08-07 18:00:43.054246 +0100][Debug ][XRootDTransport ] [se1.farm.particle.cz:1094.0] kXR_protocol successful (type: manager [], protocol version 500) [2025-08-07 18:00:43.055216 +0100][Debug ][XRootDTransport ] [se1.farm.particle.cz:1094.0] Sending out kXR_login request, username: gl05pi6, cgi: xrd.cc=uk&xrd.tz=0&xrd.appname=lar&xrd.info=&xrd.hostname=node2b06.ecdf.ed.ac.uk&xrd.rn=v5.5.5, dual-stack: false, private IPv4: false, private IPv6: false [2025-08-07 18:00:43.088169 +0100][Debug ][XRootDTransport ] [se1.farm.particle.cz:1094.0] Logged in, session: a86fe5777f0cee47ec9b59453ec32666 [2025-08-07 18:00:43.088184 +0100][Debug ][XRootDTransport ] [se1.farm.particle.cz:1094.0] Authentication is required: &P=gsi,v:10400,c:ssl,ca:9c979c2b&P=unix [2025-08-07 18:00:43.088224 +0100][Debug ][XRootDTransport ] [se1.farm.particle.cz:1094.0] Sending authentication data [2025-08-07 18:00:43.113401 +0100][Debug ][XRootDTransport ] [se1.farm.particle.cz:1094.0] Trying to authenticate using gsi [2025-08-07 18:00:43.410204 +0100][Debug ][XRootDTransport ] [se1.farm.particle.cz:1094.0] Sending more authentication data for gsi [2025-08-07 18:00:43.465846 +0100][Debug ][XRootDTransport ] [se1.farm.particle.cz:1094.0] Authenticated with gsi. [2025-08-07 18:00:43.465960 +0100][Debug ][PostMaster ] [se1.farm.particle.cz:1094] Stream 0 connected (IPv4). [2025-08-07 18:00:43.466001 +0100][Debug ][Utility ] Monitor library name not set. No monitoring [2025-08-07 18:00:43.466278 +0100][Debug ][ExDbgMsg ] [se1.farm.particle.cz:1094] Moving MsgHandler: 0x9cb4900 (message: kXR_open (file: /dune/RSE/fardet-hd/31/b1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_382_20231207T144404Z_gen_g4_detsim_hitreco__20240510T060953Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) from out-queu to in-queue. [2025-08-07 18:00:43.607361 +0100][Debug ][ExDbgMsg ] [msg: 0x71bcb40] Assigned MsgHandler: 0x9cb4900. [2025-08-07 18:00:43.607402 +0100][Debug ][ExDbgMsg ] [handler: 0x9cb4900] Removed MsgHandler: 0x9cb4900 from the in-queue. [2025-08-07 18:00:43.607698 +0100][Debug ][XRootD ] [se1.farm.particle.cz:1094] Handling error while processing kXR_open (file: /dune/RSE/fardet-hd/31/b1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_382_20231207T144404Z_gen_g4_detsim_hitreco__20240510T060953Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [ERROR] Error response: no such file or directory. [2025-08-07 18:00:43.607899 +0100][Debug ][ExDbgMsg ] [se1.farm.particle.cz:1094] Calling MsgHandler: 0x9cb4900 (message: kXR_open (file: /dune/RSE/fardet-hd/31/b1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_382_20231207T144404Z_gen_g4_detsim_hitreco__20240510T060953Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) with status: [ERROR] Error response: no such file or directory. [2025-08-07 18:00:43.608085 +0100][Debug ][File ] [0x9db6800@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/31/b1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_382_20231207T144404Z_gen_g4_detsim_hitreco__20240510T060953Z_reco2.root?xrdcl.requuid=9d0b8085-da2b-4b51-bc2f-568d63d8e0be] Open has returned with status [ERROR] Server responded with an error: [3011] No such file [2025-08-07 18:00:43.608135 +0100][Debug ][File ] [0x9db6800@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/31/b1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_382_20231207T144404Z_gen_g4_detsim_hitreco__20240510T060953Z_reco2.root?xrdcl.requuid=9d0b8085-da2b-4b51-bc2f-568d63d8e0be] Error while opening at se1.farm.particle.cz:1094: [ERROR] Server responded with an error: [3011] No such file [2025-08-07 18:00:43.608268 +0100][Debug ][ExDbgMsg ] [se1.farm.particle.cz:1094] Destroying MsgHandler: 0x9cb4900. [2025-08-07 18:00:43.690153 +0100][Debug ][JobMgr ] Stopping the job manager... [2025-08-07 18:00:43.690436 +0100][Debug ][JobMgr ] Job manager stopped [2025-08-07 18:00:43.690619 +0100][Debug ][TaskMgr ] Stopping the task manager... [2025-08-07 18:00:43.690706 +0100][Debug ][TaskMgr ] Task manager stopped [2025-08-07 18:00:43.690713 +0100][Debug ][Poller ] Stopping the poller... [2025-08-07 18:00:43.691118 +0100][Debug ][AsyncSock ] [se1.farm.particle.cz:1094.0] Closing the socket [2025-08-07 18:00:43.691196 +0100][Debug ][Poller ] <[::ffff:192.41.105.39]:53994><--><[::ffff:147.231.25.100]:1094> Removing socket from the poller [2025-08-07 18:00:43.691292 +0100][Debug ][PostMaster ] [se1.farm.particle.cz:1094] Destroying stream [2025-08-07 18:00:43.691364 +0100][Debug ][AsyncSock ] [se1.farm.particle.cz:1094.0] Closing the socket lar exit code 20 === Start last 100 lines of lar log file === tail: cannot open '_reco_20250807T170028Z.log' for reading: No such file or directory === End last 100 lines of lar log file === processed files root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/31/b1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_382_20231207T144404Z_gen_g4_detsim_hitreco__20240510T060953Z_reco2.root root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/08/bd/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50571580_410_20231203T022120Z_gen_g4_detsim_hitreco__20240508T071718Z_reco2.root root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/1f/b4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6410210_1037_20231203T090813Z_gen_g4_detsim_hitreco__20240508T052722Z_reco2.root .: total 72 -rw-r--r-- 1 gl05pi6 eddie_users 15182 Aug 7 18:00 caf_20250807T170028Z.log -rw-r--r-- 1 gl05pi6 eddie_users 14758 Aug 7 18:00 jobscript.log -rw-r--r-- 1 gl05pi6 eddie_users 560 Aug 7 18:00 caf_20250807T170028Z.file -rw-r--r-- 1 gl05pi6 eddie_users 560 Aug 7 18:00 caf_20250807T170028Z.pfns -rw-r--r-- 1 gl05pi6 eddie_users 560 Aug 7 18:00 file.list -rw-r--r-- 1 gl05pi6 eddie_users 560 Aug 7 18:00 justin-processed-pfns.txt -rw-r--r-- 1 gl05pi6 eddie_users 519 Aug 7 18:00 caf.root -rw-r--r-- 1 gl05pi6 eddie_users 519 Aug 7 18:00 caf_fd_hd_atmo_627_20250807T170028Z.root -rw-r--r-- 1 gl05pi6 eddie_users 413 Aug 7 18:00 all-input-dids.txt -rw-r--r-- 1 gl05pi6 eddie_users 413 Aug 7 18:00 caf_20250807T170028Z.did -rw-r--r-- 1 gl05pi6 eddie_users 413 Aug 7 18:00 did.list -rw-r--r-- 1 gl05pi6 eddie_users 411 Aug 7 18:00 flatcaf.root