Jobsub ID 15363.3@dunegpschedd02.fnal.gov
| Jobsub ID | 15363.3@dunegpschedd02.fnal.gov | |
| Workflow ID | 648 | |
| Stage ID | 1 | |
| User name | pgranger@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 18000 (5 hours) | |
| Submitted time | 2025-08-08 01:32:21 | |
| Site | UK_Edinburgh | |
| Entry | DUNE_UK_SGridECDF_ce1 | |
| Last heartbeat | 2025-08-08 01:34:34 | |
| From worker node | Hostname | node2b03.ecdf.ed.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 171000 (47 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Started | 2025-08-08 01:33:53 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6396715_774_20231202T065334Z_gen_g4_detsim_hitreco__20240507T202637Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50264716_267_20231123T005421Z_gen_g4_detsim_hitreco__20240507T175713Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_421_20231201T105712Z_gen_g4_detsim_hitreco__20240507T193255Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_223_20231205T020756Z_gen_g4_detsim_hitreco__20240509T222750Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6390196_337_20231201T205615Z_gen_g4_detsim_hitreco__20240507T220413Z_reco2.root | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-08-08 01:34:34 | |
| Saved logs | justin-logs:15363.3-dunegpschedd02.fnal.gov.logs.tgz | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/24/c6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6396715_774_20231202T065334Z_gen_g4_detsim_hitreco__20240507T202637Z_reco2.root?xrdcl.requuid=9004a5e8-8cdb-429e-9c60-f270f159e129] Sending an open command [2025-08-08 02:34:17.018667 +0100][Debug ][Utility ] Env: trying to get a non-existent string entry: pollerpreference [2025-08-08 02:34:17.018760 +0100][Debug ][Poller ] Available pollers: built-in [2025-08-08 02:34:17.018765 +0100][Debug ][Poller ] Attempting to create a poller according to preference: built-in [2025-08-08 02:34:17.018769 +0100][Debug ][Poller ] Creating poller: built-in [2025-08-08 02:34:17.018845 +0100][Debug ][Poller ] Creating and starting the built-in poller... [2025-08-08 02:34:17.019292 +0100][Debug ][Poller ] Using 1 poller threads [2025-08-08 02:34:17.019330 +0100][Debug ][TaskMgr ] Starting the task manager... [2025-08-08 02:34:17.019369 +0100][Debug ][TaskMgr ] Task manager started [2025-08-08 02:34:17.019403 +0100][Debug ][JobMgr ] Starting the job manager... [2025-08-08 02:34:17.019501 +0100][Debug ][JobMgr ] Job manager started, 3 workers [2025-08-08 02:34:17.019584 +0100][Debug ][TaskMgr ] Registering task: "FileTimer task" to be run at: [2025-08-08 02:34:17 +0100] [2025-08-08 02:34:17.019872 +0100][Debug ][ExDbgMsg ] [se1.farm.particle.cz:1094] MsgHandler created: 0x9acb3b0 (message: kXR_open (file: /dune/RSE/fardet-hd/24/c6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6396715_774_20231202T065334Z_gen_g4_detsim_hitreco__20240507T202637Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ). [2025-08-08 02:34:17.020303 +0100][Debug ][PostMaster ] Creating new channel to: root://se1.farm.particle.cz:1094/ [2025-08-08 02:34:17.020578 +0100][Debug ][PostMaster ] [se1.farm.particle.cz:1094] Stream parameters: Network Stack: IPAuto, Connection Window: 30, ConnectionRetry: 5, Stream Error Window: 1800 [2025-08-08 02:34:17.020717 +0100][Debug ][TaskMgr ] Registering task: "TickGeneratorTask for: root://se1.farm.particle.cz:1094/" to be run at: [2025-08-08 02:34:32 +0100] [2025-08-08 02:34:17.021684 +0100][Debug ][PostMaster ] [se1.farm.particle.cz:1094] Found 1 address(es): [::ffff:147.231.25.100]:1094 [2025-08-08 02:34:17.021768 +0100][Debug ][AsyncSock ] [se1.farm.particle.cz:1094.0] Attempting connection to [::ffff:147.231.25.100]:1094 [2025-08-08 02:34:17.021839 +0100][Debug ][Poller ] Adding socket 0x90d0650 to the poller [2025-08-08 02:34:17.053352 +0100][Debug ][AsyncSock ] [se1.farm.particle.cz:1094.0] Async connection call returned [2025-08-08 02:34:17.053546 +0100][Debug ][XRootDTransport ] [se1.farm.particle.cz:1094.0] Sending out the initial hand shake + kXR_protocol [2025-08-08 02:34:17.085257 +0100][Debug ][XRootDTransport ] [se1.farm.particle.cz:1094.0] Got the server hand shake response (type: manager [], protocol version 500) [2025-08-08 02:34:17.085369 +0100][Debug ][XRootDTransport ] [se1.farm.particle.cz:1094.0] kXR_protocol successful (type: manager [], protocol version 500) [2025-08-08 02:34:17.086053 +0100][Debug ][XRootDTransport ] [se1.farm.particle.cz:1094.0] Sending out kXR_login request, username: gl05pi6, cgi: xrd.cc=uk&xrd.tz=0&xrd.appname=lar&xrd.info=&xrd.hostname=node2b03.ecdf.ed.ac.uk&xrd.rn=v5.5.5, dual-stack: false, private IPv4: false, private IPv6: false [2025-08-08 02:34:17.117579 +0100][Debug ][XRootDTransport ] [se1.farm.particle.cz:1094.0] Logged in, session: dc82c5970741e12519c0109b1a81dbe6 [2025-08-08 02:34:17.117587 +0100][Debug ][XRootDTransport ] [se1.farm.particle.cz:1094.0] Authentication is required: &P=gsi,v:10400,c:ssl,ca:9c979c2b&P=unix [2025-08-08 02:34:17.117613 +0100][Debug ][XRootDTransport ] [se1.farm.particle.cz:1094.0] Sending authentication data [2025-08-08 02:34:17.139634 +0100][Debug ][XRootDTransport ] [se1.farm.particle.cz:1094.0] Trying to authenticate using gsi [2025-08-08 02:34:18.082335 +0100][Debug ][XRootDTransport ] [se1.farm.particle.cz:1094.0] Sending more authentication data for gsi [2025-08-08 02:34:18.128291 +0100][Debug ][XRootDTransport ] [se1.farm.particle.cz:1094.0] Authenticated with gsi. [2025-08-08 02:34:18.128373 +0100][Debug ][PostMaster ] [se1.farm.particle.cz:1094] Stream 0 connected (IPv4). [2025-08-08 02:34:18.128408 +0100][Debug ][Utility ] Monitor library name not set. No monitoring [2025-08-08 02:34:18.128634 +0100][Debug ][ExDbgMsg ] [se1.farm.particle.cz:1094] Moving MsgHandler: 0x9acb3b0 (message: kXR_open (file: /dune/RSE/fardet-hd/24/c6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6396715_774_20231202T065334Z_gen_g4_detsim_hitreco__20240507T202637Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) from out-queu to in-queue. [2025-08-08 02:34:18.262844 +0100][Debug ][ExDbgMsg ] [msg: 0xa042a90] Assigned MsgHandler: 0x9acb3b0. [2025-08-08 02:34:18.262865 +0100][Debug ][ExDbgMsg ] [handler: 0x9acb3b0] Removed MsgHandler: 0x9acb3b0 from the in-queue. [2025-08-08 02:34:18.263105 +0100][Debug ][XRootD ] [se1.farm.particle.cz:1094] Handling error while processing kXR_open (file: /dune/RSE/fardet-hd/24/c6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6396715_774_20231202T065334Z_gen_g4_detsim_hitreco__20240507T202637Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [ERROR] Error response: no such file or directory. [2025-08-08 02:34:18.263278 +0100][Debug ][ExDbgMsg ] [se1.farm.particle.cz:1094] Calling MsgHandler: 0x9acb3b0 (message: kXR_open (file: /dune/RSE/fardet-hd/24/c6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6396715_774_20231202T065334Z_gen_g4_detsim_hitreco__20240507T202637Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) with status: [ERROR] Error response: no such file or directory. [2025-08-08 02:34:18.263448 +0100][Debug ][File ] [0xa043b30@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/24/c6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6396715_774_20231202T065334Z_gen_g4_detsim_hitreco__20240507T202637Z_reco2.root?xrdcl.requuid=9004a5e8-8cdb-429e-9c60-f270f159e129] Open has returned with status [ERROR] Server responded with an error: [3011] No such file [2025-08-08 02:34:18.263483 +0100][Debug ][File ] [0xa043b30@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/24/c6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6396715_774_20231202T065334Z_gen_g4_detsim_hitreco__20240507T202637Z_reco2.root?xrdcl.requuid=9004a5e8-8cdb-429e-9c60-f270f159e129] Error while opening at se1.farm.particle.cz:1094: [ERROR] Server responded with an error: [3011] No such file [2025-08-08 02:34:18.263598 +0100][Debug ][ExDbgMsg ] [se1.farm.particle.cz:1094] Destroying MsgHandler: 0x9acb3b0. [2025-08-08 02:34:18.366104 +0100][Debug ][JobMgr ] Stopping the job manager... [2025-08-08 02:34:18.366455 +0100][Debug ][JobMgr ] Job manager stopped [2025-08-08 02:34:18.366504 +0100][Debug ][TaskMgr ] Stopping the task manager... [2025-08-08 02:34:18.366588 +0100][Debug ][TaskMgr ] Task manager stopped [2025-08-08 02:34:18.366594 +0100][Debug ][Poller ] Stopping the poller... [2025-08-08 02:34:18.366956 +0100][Debug ][AsyncSock ] [se1.farm.particle.cz:1094.0] Closing the socket [2025-08-08 02:34:18.367026 +0100][Debug ][Poller ] <[::ffff:192.41.105.36]:33598><--><[::ffff:147.231.25.100]:1094> Removing socket from the poller [2025-08-08 02:34:18.367118 +0100][Debug ][PostMaster ] [se1.farm.particle.cz:1094] Destroying stream [2025-08-08 02:34:18.367196 +0100][Debug ][AsyncSock ] [se1.farm.particle.cz:1094.0] Closing the socket lar exit code 20 === Start last 100 lines of lar log file === tail: cannot open '_reco_20250808T013401Z.log' for reading: No such file or directory === End last 100 lines of lar log file === processed files root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/24/c6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6396715_774_20231202T065334Z_gen_g4_detsim_hitreco__20240507T202637Z_reco2.root root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/91/b9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50264716_267_20231123T005421Z_gen_g4_detsim_hitreco__20240507T175713Z_reco2.root root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b0/37/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_421_20231201T105712Z_gen_g4_detsim_hitreco__20240507T193255Z_reco2.root root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b4/be/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_223_20231205T020756Z_gen_g4_detsim_hitreco__20240509T222750Z_reco2.root root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/9f/05/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6390196_337_20231201T205615Z_gen_g4_detsim_hitreco__20240507T220413Z_reco2.root .: total 72 -rw-r--r-- 1 gl05pi6 eddie_users 15731 Aug 8 02:34 jobscript.log -rw-r--r-- 1 gl05pi6 eddie_users 15182 Aug 8 02:34 caf_20250808T013401Z.log -rw-r--r-- 1 gl05pi6 eddie_users 963 Aug 8 02:34 caf_20250808T013401Z.file -rw-r--r-- 1 gl05pi6 eddie_users 963 Aug 8 02:34 caf_20250808T013401Z.pfns -rw-r--r-- 1 gl05pi6 eddie_users 963 Aug 8 02:33 file.list -rw-r--r-- 1 gl05pi6 eddie_users 963 Aug 8 02:34 justin-processed-pfns.txt -rw-r--r-- 1 gl05pi6 eddie_users 688 Aug 8 02:33 all-input-dids.txt -rw-r--r-- 1 gl05pi6 eddie_users 688 Aug 8 02:34 caf_20250808T013401Z.did -rw-r--r-- 1 gl05pi6 eddie_users 688 Aug 8 02:33 did.list -rw-r--r-- 1 gl05pi6 eddie_users 519 Aug 8 02:34 caf.root -rw-r--r-- 1 gl05pi6 eddie_users 519 Aug 8 02:34 caf_fd_hd_atmo_648_20250808T013401Z.root -rw-r--r-- 1 gl05pi6 eddie_users 411 Aug 8 02:34 flatcaf.root