Jobsub ID 15496.1@dunegpschedd02.fnal.gov
| Jobsub ID | 15496.1@dunegpschedd02.fnal.gov | |
| Workflow ID | 648 | |
| Stage ID | 1 | |
| User name | pgranger@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 18000 (5 hours) | |
| Submitted time | 2025-08-08 06:36:36 | |
| Site | UK_Edinburgh | |
| Entry | DUNE_UK_SGridECDF_ce1 | |
| Last heartbeat | 2025-08-08 06:42:37 | |
| From worker node | Hostname | node2b01.ecdf.ed.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 171000 (47 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Started | 2025-08-08 06:37:09 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_597_20231207T112658Z_gen_g4_detsim_hitreco__20240510T053540Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_186_20231203T180154Z_gen_g4_detsim_hitreco__20240509T203907Z_reco2.root | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-08-08 06:42:37 | |
| Saved logs | justin-logs:15496.1-dunegpschedd02.fnal.gov.logs.tgz | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
29.farm.particle.cz:23709] Calling MsgHandler: 0xdae31b0 (message: kXR_read (handle: 0x00000000, offset: 1252084798, size: 147) ) with status: [SUCCESS] . [2025-08-08 07:42:19.104964 +0100][Debug ][ExDbgMsg ] [ds29.farm.particle.cz:23709] Destroying MsgHandler: 0xdae31b0. [2025-08-08 07:42:19.105006 +0100][Debug ][File ] [0x9704f70@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/81/e3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_597_20231207T112658Z_gen_g4_detsim_hitreco__20240510T053540Z_reco2.root?xrdcl.requuid=dc33f6d7-cc22-45fc-86de-4775bc7169c3] Sending a read command for handle 0x0 to ds29.farm.particle.cz:23709 [2025-08-08 07:42:19.105028 +0100][Debug ][ExDbgMsg ] [ds29.farm.particle.cz:23709] MsgHandler created: 0xdae31b0 (message: kXR_read (handle: 0x00000000, offset: 1252084945, size: 512) ). [2025-08-08 07:42:19.105082 +0100][Debug ][ExDbgMsg ] [ds29.farm.particle.cz:23709] Moving MsgHandler: 0xdae31b0 (message: kXR_read (handle: 0x00000000, offset: 1252084945, size: 512) ) from out-queu to in-queue. [2025-08-08 07:42:19.136730 +0100][Debug ][ExDbgMsg ] [msg: 0xc8b1ee0] Assigned MsgHandler: 0xdae31b0. [2025-08-08 07:42:19.136774 +0100][Debug ][ExDbgMsg ] [handler: 0xdae31b0] Removed MsgHandler: 0xdae31b0 from the in-queue. [2025-08-08 07:42:19.136809 +0100][Debug ][ExDbgMsg ] [ds29.farm.particle.cz:23709] Calling MsgHandler: 0xdae31b0 (message: kXR_read (handle: 0x00000000, offset: 1252084945, size: 512) ) with status: [SUCCESS] . [2025-08-08 07:42:19.136825 +0100][Debug ][ExDbgMsg ] [ds29.farm.particle.cz:23709] Destroying MsgHandler: 0xdae31b0. [2025-08-08 07:42:19.141543 +0100][Debug ][File ] [0x9704f70@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/81/e3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_597_20231207T112658Z_gen_g4_detsim_hitreco__20240510T053540Z_reco2.root?xrdcl.requuid=dc33f6d7-cc22-45fc-86de-4775bc7169c3] Sending a close command for handle 0x0 to ds29.farm.particle.cz:23709 [2025-08-08 07:42:19.141654 +0100][Debug ][ExDbgMsg ] [ds29.farm.particle.cz:23709] MsgHandler created: 0xd06fbb0 (message: kXR_close (handle: 0x00000000) ). [2025-08-08 07:42:19.141719 +0100][Debug ][ExDbgMsg ] [ds29.farm.particle.cz:23709] Moving MsgHandler: 0xd06fbb0 (message: kXR_close (handle: 0x00000000) ) from out-queu to in-queue. [2025-08-08 07:42:19.173362 +0100][Debug ][ExDbgMsg ] [msg: 0xc8e40a0] Assigned MsgHandler: 0xd06fbb0. [2025-08-08 07:42:19.173369 +0100][Debug ][ExDbgMsg ] [handler: 0xd06fbb0] Removed MsgHandler: 0xd06fbb0 from the in-queue. [2025-08-08 07:42:19.173388 +0100][Debug ][ExDbgMsg ] [ds29.farm.particle.cz:23709] Calling MsgHandler: 0xd06fbb0 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] . [2025-08-08 07:42:19.173455 +0100][Debug ][File ] [0x9704f70@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/81/e3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_597_20231207T112658Z_gen_g4_detsim_hitreco__20240510T053540Z_reco2.root?xrdcl.requuid=dc33f6d7-cc22-45fc-86de-4775bc7169c3] Close returned from ds29.farm.particle.cz:23709 with: [SUCCESS] [2025-08-08 07:42:19.173500 +0100][Debug ][ExDbgMsg ] [ds29.farm.particle.cz:23709] Destroying MsgHandler: 0xd06fbb0. [2025-08-08 07:42:19.175815 +0100][Debug ][Utility ] Env: overriding entry: connectionwindow=30 with 30 [2025-08-08 07:42:19.175841 +0100][Debug ][Utility ] Env: overriding entry: requesttimeout=14400 with 4096 [2025-08-08 07:42:19.175848 +0100][Debug ][Utility ] Env: overriding entry: requesttimeout=4096 with 14400 [2025-08-08 07:42:19.175866 +0100][Debug ][Utility ] Env: overriding entry: redirectlimit=64 with 64 [2025-08-08 07:42:19.175881 +0100][Debug ][Utility ] Env: overriding entry: multiprotocol=1 with 1 [2025-08-08 07:42:19.175965 +0100][Debug ][File ] [0xc7c2d10@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/df/1a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_186_20231203T180154Z_gen_g4_detsim_hitreco__20240509T203907Z_reco2.root?xrdcl.requuid=66d362af-b4f4-4afe-b872-6dfdcbe1807b] Sending an open command [2025-08-08 07:42:19.175982 +0100][Debug ][ExDbgMsg ] [se1.farm.particle.cz:1094] MsgHandler created: 0xd06fbb0 (message: kXR_open (file: /dune/RSE/fardet-hd/df/1a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_186_20231203T180154Z_gen_g4_detsim_hitreco__20240509T203907Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ). [2025-08-08 07:42:19.176022 +0100][Debug ][ExDbgMsg ] [se1.farm.particle.cz:1094] Moving MsgHandler: 0xd06fbb0 (message: kXR_open (file: /dune/RSE/fardet-hd/df/1a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_186_20231203T180154Z_gen_g4_detsim_hitreco__20240509T203907Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) from out-queu to in-queue. [2025-08-08 07:42:19.214517 +0100][Debug ][ExDbgMsg ] [msg: 0xc986dc0] Assigned MsgHandler: 0xd06fbb0. [2025-08-08 07:42:19.214528 +0100][Debug ][ExDbgMsg ] [handler: 0xd06fbb0] Removed MsgHandler: 0xd06fbb0 from the in-queue. [2025-08-08 07:42:19.214553 +0100][Debug ][XRootD ] [se1.farm.particle.cz:1094] Handling error while processing kXR_open (file: /dune/RSE/fardet-hd/df/1a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_186_20231203T180154Z_gen_g4_detsim_hitreco__20240509T203907Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [ERROR] Error response: no such file or directory. [2025-08-08 07:42:19.214723 +0100][Debug ][ExDbgMsg ] [se1.farm.particle.cz:1094] Calling MsgHandler: 0xd06fbb0 (message: kXR_open (file: /dune/RSE/fardet-hd/df/1a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_186_20231203T180154Z_gen_g4_detsim_hitreco__20240509T203907Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) with status: [ERROR] Error response: no such file or directory. [2025-08-08 07:42:19.214785 +0100][Debug ][File ] [0xc7c2d10@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/df/1a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_186_20231203T180154Z_gen_g4_detsim_hitreco__20240509T203907Z_reco2.root?xrdcl.requuid=66d362af-b4f4-4afe-b872-6dfdcbe1807b] Open has returned with status [ERROR] Server responded with an error: [3011] No such file [2025-08-08 07:42:19.214794 +0100][Debug ][File ] [0xc7c2d10@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/df/1a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_186_20231203T180154Z_gen_g4_detsim_hitreco__20240509T203907Z_reco2.root?xrdcl.requuid=66d362af-b4f4-4afe-b872-6dfdcbe1807b] Error while opening at se1.farm.particle.cz:1094: [ERROR] Server responded with an error: [3011] No such file [2025-08-08 07:42:19.214838 +0100][Debug ][ExDbgMsg ] [se1.farm.particle.cz:1094] Destroying MsgHandler: 0xd06fbb0. [2025-08-08 07:42:19.411203 +0100][Debug ][JobMgr ] Stopping the job manager... [2025-08-08 07:42:19.411469 +0100][Debug ][JobMgr ] Job manager stopped [2025-08-08 07:42:19.411522 +0100][Debug ][TaskMgr ] Stopping the task manager... [2025-08-08 07:42:19.411605 +0100][Debug ][TaskMgr ] Task manager stopped [2025-08-08 07:42:19.411632 +0100][Debug ][Poller ] Stopping the poller... [2025-08-08 07:42:19.412042 +0100][Debug ][AsyncSock ] [ds29.farm.particle.cz:23709.0] Closing the socket [2025-08-08 07:42:19.412131 +0100][Debug ][Poller ] <[::ffff:192.41.105.34]:48350><--><[::ffff:147.231.25.167]:23709> Removing socket from the poller [2025-08-08 07:42:19.412220 +0100][Debug ][PostMaster ] [ds29.farm.particle.cz:23709] Destroying stream [2025-08-08 07:42:19.412290 +0100][Debug ][AsyncSock ] [ds29.farm.particle.cz:23709.0] Closing the socket [2025-08-08 07:42:19.412335 +0100][Debug ][AsyncSock ] [se1.farm.particle.cz:1094.0] Closing the socket [2025-08-08 07:42:19.412343 +0100][Debug ][Poller ] <[::ffff:192.41.105.34]:46744><--><[::ffff:147.231.25.100]:1094> Removing socket from the poller [2025-08-08 07:42:19.412355 +0100][Debug ][PostMaster ] [se1.farm.particle.cz:1094] Destroying stream [2025-08-08 07:42:19.412359 +0100][Debug ][AsyncSock ] [se1.farm.particle.cz:1094.0] Closing the socket lar exit code 1 === Start last 100 lines of lar log file === tail: cannot open '_reco_20250808T063717Z.log' for reading: No such file or directory === End last 100 lines of lar log file === processed files root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/81/e3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_597_20231207T112658Z_gen_g4_detsim_hitreco__20240510T053540Z_reco2.root root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/df/1a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_186_20231203T180154Z_gen_g4_detsim_hitreco__20240509T203907Z_reco2.root .: total 17372 -rw-r--r-- 1 gl05pi6 eddie_users 10815024 Aug 8 07:42 jobscript.log -rw-r--r-- 1 gl05pi6 eddie_users 2331983 Aug 8 07:42 flatcaf.root -rw-r--r-- 1 gl05pi6 eddie_users 1822023 Aug 8 07:42 caf.root -rw-r--r-- 1 gl05pi6 eddie_users 1822023 Aug 8 07:42 caf_fd_hd_atmo_648_20250808T063717Z.root -rw-r--r-- 1 gl05pi6 eddie_users 942519 Aug 8 07:42 caf_20250808T063717Z.log -rw-r--r-- 1 gl05pi6 eddie_users 374 Aug 8 07:42 caf_20250808T063717Z.file -rw-r--r-- 1 gl05pi6 eddie_users 374 Aug 8 07:42 caf_20250808T063717Z.pfns -rw-r--r-- 1 gl05pi6 eddie_users 374 Aug 8 07:37 file.list -rw-r--r-- 1 gl05pi6 eddie_users 374 Aug 8 07:42 justin-processed-pfns.txt -rw-r--r-- 1 gl05pi6 eddie_users 276 Aug 8 07:37 all-input-dids.txt -rw-r--r-- 1 gl05pi6 eddie_users 276 Aug 8 07:42 caf_20250808T063717Z.did -rw-r--r-- 1 gl05pi6 eddie_users 276 Aug 8 07:37 did.list