Jobsub ID 21964.4@dunegpschedd01.fnal.gov
| Jobsub ID | 21964.4@dunegpschedd01.fnal.gov | |
| Workflow ID | 648 | |
| Stage ID | 1 | |
| User name | pgranger@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 18000 (5 hours) | |
| Submitted time | 2025-08-08 06:20:38 | |
| Site | UK_Edinburgh | |
| Entry | DUNE_UK_SGridECDF_ce1 | |
| Last heartbeat | 2025-08-08 06:27:20 | |
| From worker node | Hostname | node2b08.ecdf.ed.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 171000 (47 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Started | 2025-08-08 06:21:58 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_597_20231207T112658Z_gen_g4_detsim_hitreco__20240510T053540Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_186_20231203T180154Z_gen_g4_detsim_hitreco__20240509T203907Z_reco2.root | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-08-08 06:27:20 | |
| Saved logs | justin-logs:21964.4-dunegpschedd01.fnal.gov.logs.tgz | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
.farm.particle.cz:23709] Calling MsgHandler: 0x21c42950 (message: kXR_read (handle: 0x00000000, offset: 1252084798, size: 147) ) with status: [SUCCESS] . [2025-08-08 07:27:05.100294 +0100][Debug ][ExDbgMsg ] [ds29.farm.particle.cz:23709] Destroying MsgHandler: 0x21c42950. [2025-08-08 07:27:05.100330 +0100][Debug ][File ] [0x965b6e0@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/81/e3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_597_20231207T112658Z_gen_g4_detsim_hitreco__20240510T053540Z_reco2.root?xrdcl.requuid=32dcb8fd-8258-4a8d-bd0d-2a67a6c18609] Sending a read command for handle 0x0 to ds29.farm.particle.cz:23709 [2025-08-08 07:27:05.100350 +0100][Debug ][ExDbgMsg ] [ds29.farm.particle.cz:23709] MsgHandler created: 0xeafaf70 (message: kXR_read (handle: 0x00000000, offset: 1252084945, size: 512) ). [2025-08-08 07:27:05.100401 +0100][Debug ][ExDbgMsg ] [ds29.farm.particle.cz:23709] Moving MsgHandler: 0xeafaf70 (message: kXR_read (handle: 0x00000000, offset: 1252084945, size: 512) ) from out-queu to in-queue. [2025-08-08 07:27:05.131913 +0100][Debug ][ExDbgMsg ] [msg: 0xf2b8970] Assigned MsgHandler: 0xeafaf70. [2025-08-08 07:27:05.131951 +0100][Debug ][ExDbgMsg ] [handler: 0xeafaf70] Removed MsgHandler: 0xeafaf70 from the in-queue. [2025-08-08 07:27:05.131983 +0100][Debug ][ExDbgMsg ] [ds29.farm.particle.cz:23709] Calling MsgHandler: 0xeafaf70 (message: kXR_read (handle: 0x00000000, offset: 1252084945, size: 512) ) with status: [SUCCESS] . [2025-08-08 07:27:05.132011 +0100][Debug ][ExDbgMsg ] [ds29.farm.particle.cz:23709] Destroying MsgHandler: 0xeafaf70. [2025-08-08 07:27:05.136864 +0100][Debug ][File ] [0x965b6e0@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/81/e3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_597_20231207T112658Z_gen_g4_detsim_hitreco__20240510T053540Z_reco2.root?xrdcl.requuid=32dcb8fd-8258-4a8d-bd0d-2a67a6c18609] Sending a close command for handle 0x0 to ds29.farm.particle.cz:23709 [2025-08-08 07:27:05.136980 +0100][Debug ][ExDbgMsg ] [ds29.farm.particle.cz:23709] MsgHandler created: 0xdfc8ae0 (message: kXR_close (handle: 0x00000000) ). [2025-08-08 07:27:05.137057 +0100][Debug ][ExDbgMsg ] [ds29.farm.particle.cz:23709] Moving MsgHandler: 0xdfc8ae0 (message: kXR_close (handle: 0x00000000) ) from out-queu to in-queue. [2025-08-08 07:27:05.169032 +0100][Debug ][ExDbgMsg ] [msg: 0xda9d3e0] Assigned MsgHandler: 0xdfc8ae0. [2025-08-08 07:27:05.169041 +0100][Debug ][ExDbgMsg ] [handler: 0xdfc8ae0] Removed MsgHandler: 0xdfc8ae0 from the in-queue. [2025-08-08 07:27:05.169058 +0100][Debug ][ExDbgMsg ] [ds29.farm.particle.cz:23709] Calling MsgHandler: 0xdfc8ae0 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] . [2025-08-08 07:27:05.169122 +0100][Debug ][File ] [0x965b6e0@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/81/e3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_597_20231207T112658Z_gen_g4_detsim_hitreco__20240510T053540Z_reco2.root?xrdcl.requuid=32dcb8fd-8258-4a8d-bd0d-2a67a6c18609] Close returned from ds29.farm.particle.cz:23709 with: [SUCCESS] [2025-08-08 07:27:05.169171 +0100][Debug ][ExDbgMsg ] [ds29.farm.particle.cz:23709] Destroying MsgHandler: 0xdfc8ae0. [2025-08-08 07:27:05.171125 +0100][Debug ][Utility ] Env: overriding entry: connectionwindow=30 with 30 [2025-08-08 07:27:05.171142 +0100][Debug ][Utility ] Env: overriding entry: requesttimeout=14400 with 4096 [2025-08-08 07:27:05.171149 +0100][Debug ][Utility ] Env: overriding entry: requesttimeout=4096 with 14400 [2025-08-08 07:27:05.171165 +0100][Debug ][Utility ] Env: overriding entry: redirectlimit=64 with 64 [2025-08-08 07:27:05.171181 +0100][Debug ][Utility ] Env: overriding entry: multiprotocol=1 with 1 [2025-08-08 07:27:05.171266 +0100][Debug ][File ] [0xd982780@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/df/1a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_186_20231203T180154Z_gen_g4_detsim_hitreco__20240509T203907Z_reco2.root?xrdcl.requuid=a5b39b8b-dff4-4477-8196-13133953c908] Sending an open command [2025-08-08 07:27:05.171282 +0100][Debug ][ExDbgMsg ] [se1.farm.particle.cz:1094] MsgHandler created: 0xdfc8ae0 (message: kXR_open (file: /dune/RSE/fardet-hd/df/1a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_186_20231203T180154Z_gen_g4_detsim_hitreco__20240509T203907Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ). [2025-08-08 07:27:05.171322 +0100][Debug ][ExDbgMsg ] [se1.farm.particle.cz:1094] Moving MsgHandler: 0xdfc8ae0 (message: kXR_open (file: /dune/RSE/fardet-hd/df/1a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_186_20231203T180154Z_gen_g4_detsim_hitreco__20240509T203907Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) from out-queu to in-queue. [2025-08-08 07:27:05.209541 +0100][Debug ][ExDbgMsg ] [msg: 0xdaa9160] Assigned MsgHandler: 0xdfc8ae0. [2025-08-08 07:27:05.209550 +0100][Debug ][ExDbgMsg ] [handler: 0xdfc8ae0] Removed MsgHandler: 0xdfc8ae0 from the in-queue. [2025-08-08 07:27:05.209574 +0100][Debug ][XRootD ] [se1.farm.particle.cz:1094] Handling error while processing kXR_open (file: /dune/RSE/fardet-hd/df/1a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_186_20231203T180154Z_gen_g4_detsim_hitreco__20240509T203907Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [ERROR] Error response: no such file or directory. [2025-08-08 07:27:05.209739 +0100][Debug ][ExDbgMsg ] [se1.farm.particle.cz:1094] Calling MsgHandler: 0xdfc8ae0 (message: kXR_open (file: /dune/RSE/fardet-hd/df/1a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_186_20231203T180154Z_gen_g4_detsim_hitreco__20240509T203907Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) with status: [ERROR] Error response: no such file or directory. [2025-08-08 07:27:05.209799 +0100][Debug ][File ] [0xd982780@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/df/1a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_186_20231203T180154Z_gen_g4_detsim_hitreco__20240509T203907Z_reco2.root?xrdcl.requuid=a5b39b8b-dff4-4477-8196-13133953c908] Open has returned with status [ERROR] Server responded with an error: [3011] No such file [2025-08-08 07:27:05.209807 +0100][Debug ][File ] [0xd982780@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/df/1a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_186_20231203T180154Z_gen_g4_detsim_hitreco__20240509T203907Z_reco2.root?xrdcl.requuid=a5b39b8b-dff4-4477-8196-13133953c908] Error while opening at se1.farm.particle.cz:1094: [ERROR] Server responded with an error: [3011] No such file [2025-08-08 07:27:05.209851 +0100][Debug ][ExDbgMsg ] [se1.farm.particle.cz:1094] Destroying MsgHandler: 0xdfc8ae0. [2025-08-08 07:27:05.410453 +0100][Debug ][JobMgr ] Stopping the job manager... [2025-08-08 07:27:05.410729 +0100][Debug ][JobMgr ] Job manager stopped [2025-08-08 07:27:05.410778 +0100][Debug ][TaskMgr ] Stopping the task manager... [2025-08-08 07:27:05.410848 +0100][Debug ][TaskMgr ] Task manager stopped [2025-08-08 07:27:05.410868 +0100][Debug ][Poller ] Stopping the poller... [2025-08-08 07:27:05.411265 +0100][Debug ][AsyncSock ] [ds29.farm.particle.cz:23709.0] Closing the socket [2025-08-08 07:27:05.411338 +0100][Debug ][Poller ] <[::ffff:192.41.105.41]:32886><--><[::ffff:147.231.25.167]:23709> Removing socket from the poller [2025-08-08 07:27:05.411450 +0100][Debug ][PostMaster ] [ds29.farm.particle.cz:23709] Destroying stream [2025-08-08 07:27:05.411524 +0100][Debug ][AsyncSock ] [ds29.farm.particle.cz:23709.0] Closing the socket [2025-08-08 07:27:05.411571 +0100][Debug ][AsyncSock ] [se1.farm.particle.cz:1094.0] Closing the socket [2025-08-08 07:27:05.411578 +0100][Debug ][Poller ] <[::ffff:192.41.105.41]:33220><--><[::ffff:147.231.25.100]:1094> Removing socket from the poller [2025-08-08 07:27:05.411590 +0100][Debug ][PostMaster ] [se1.farm.particle.cz:1094] Destroying stream [2025-08-08 07:27:05.411593 +0100][Debug ][AsyncSock ] [se1.farm.particle.cz:1094.0] Closing the socket lar exit code 1 === Start last 100 lines of lar log file === tail: cannot open '_reco_20250808T062206Z.log' for reading: No such file or directory === End last 100 lines of lar log file === processed files root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/81/e3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_597_20231207T112658Z_gen_g4_detsim_hitreco__20240510T053540Z_reco2.root root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/df/1a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_186_20231203T180154Z_gen_g4_detsim_hitreco__20240509T203907Z_reco2.root .: total 17360 -rw-r--r-- 1 gl05pi6 eddie_users 10808786 Aug 8 07:27 jobscript.log -rw-r--r-- 1 gl05pi6 eddie_users 2331983 Aug 8 07:27 flatcaf.root -rw-r--r-- 1 gl05pi6 eddie_users 1822017 Aug 8 07:27 caf.root -rw-r--r-- 1 gl05pi6 eddie_users 1822017 Aug 8 07:27 caf_fd_hd_atmo_648_20250808T062206Z.root -rw-r--r-- 1 gl05pi6 eddie_users 942519 Aug 8 07:27 caf_20250808T062206Z.log -rw-r--r-- 1 gl05pi6 eddie_users 374 Aug 8 07:27 caf_20250808T062206Z.file -rw-r--r-- 1 gl05pi6 eddie_users 374 Aug 8 07:27 caf_20250808T062206Z.pfns -rw-r--r-- 1 gl05pi6 eddie_users 374 Aug 8 07:22 file.list -rw-r--r-- 1 gl05pi6 eddie_users 374 Aug 8 07:27 justin-processed-pfns.txt -rw-r--r-- 1 gl05pi6 eddie_users 276 Aug 8 07:22 all-input-dids.txt -rw-r--r-- 1 gl05pi6 eddie_users 276 Aug 8 07:27 caf_20250808T062206Z.did -rw-r--r-- 1 gl05pi6 eddie_users 276 Aug 8 07:22 did.list