Jobsub ID 15400.5@dunegpschedd02.fnal.gov
| Jobsub ID | 15400.5@dunegpschedd02.fnal.gov | |
| Workflow ID | 643 | |
| Stage ID | 1 | |
| User name | pgranger@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 18000 (5 hours) | |
| Submitted time | 2025-08-08 03:24:27 | |
| Site | UK_Edinburgh | |
| Entry | DUNE_UK_SGridECDF_ce1 | |
| Last heartbeat | 2025-08-08 03:30:11 | |
| From worker node | Hostname | node2b08.ecdf.ed.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 171000 (47 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Started | 2025-08-08 03:24:52 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_226_20231207T143234Z_gen_g4_detsim_hitreco__20240510T060915Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_1008_20231203T134835Z_gen_g4_detsim_hitreco__20240508T042957Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_1068_20231202T164130Z_gen_g4_detsim_hitreco__20240508T030215Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50546594_474_20231202T073012Z_gen_g4_detsim_hitreco__20240508T063850Z_reco2.root | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-08-08 03:30:11 | |
| Saved logs | justin-logs:15400.5-dunegpschedd02.fnal.gov.logs.tgz | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
det-hd/85/bd/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_226_20231207T143234Z_gen_g4_detsim_hitreco__20240510T060915Z_reco2.root?xrdcl.requuid=9885cede-88b5-4869-9d41-08599acc52ec] Sending a read command for handle 0x0 to ds27.farm.particle.cz:20995 [2025-08-08 04:29:55.610972 +0100][Debug ][ExDbgMsg ] [ds27.farm.particle.cz:20995] MsgHandler created: 0x1ad40d00 (message: kXR_read (handle: 0x00000000, offset: 1003028105, size: 788) ). [2025-08-08 04:29:55.611010 +0100][Debug ][ExDbgMsg ] [ds27.farm.particle.cz:20995] Moving MsgHandler: 0x1ad40d00 (message: kXR_read (handle: 0x00000000, offset: 1003028105, size: 788) ) from out-queu to in-queue. [2025-08-08 04:29:55.642440 +0100][Debug ][ExDbgMsg ] [msg: 0xc9f4a80] Assigned MsgHandler: 0x1ad40d00. [2025-08-08 04:29:55.642470 +0100][Debug ][ExDbgMsg ] [handler: 0x1ad40d00] Removed MsgHandler: 0x1ad40d00 from the in-queue. [2025-08-08 04:29:55.642496 +0100][Debug ][ExDbgMsg ] [ds27.farm.particle.cz:20995] Calling MsgHandler: 0x1ad40d00 (message: kXR_read (handle: 0x00000000, offset: 1003028105, size: 788) ) with status: [SUCCESS] . [2025-08-08 04:29:55.642508 +0100][Debug ][ExDbgMsg ] [ds27.farm.particle.cz:20995] Destroying MsgHandler: 0x1ad40d00. [2025-08-08 04:29:55.649128 +0100][Debug ][File ] [0x9693a60@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/85/bd/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_226_20231207T143234Z_gen_g4_detsim_hitreco__20240510T060915Z_reco2.root?xrdcl.requuid=9885cede-88b5-4869-9d41-08599acc52ec] Sending a close command for handle 0x0 to ds27.farm.particle.cz:20995 [2025-08-08 04:29:55.649210 +0100][Debug ][ExDbgMsg ] [ds27.farm.particle.cz:20995] MsgHandler created: 0x15a76900 (message: kXR_close (handle: 0x00000000) ). [2025-08-08 04:29:55.649265 +0100][Debug ][ExDbgMsg ] [ds27.farm.particle.cz:20995] Moving MsgHandler: 0x15a76900 (message: kXR_close (handle: 0x00000000) ) from out-queu to in-queue. [2025-08-08 04:29:55.680721 +0100][Debug ][ExDbgMsg ] [msg: 0x1a0a2420] Assigned MsgHandler: 0x15a76900. [2025-08-08 04:29:55.680735 +0100][Debug ][ExDbgMsg ] [handler: 0x15a76900] Removed MsgHandler: 0x15a76900 from the in-queue. [2025-08-08 04:29:55.680754 +0100][Debug ][ExDbgMsg ] [ds27.farm.particle.cz:20995] Calling MsgHandler: 0x15a76900 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] . [2025-08-08 04:29:55.680820 +0100][Debug ][File ] [0x9693a60@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/85/bd/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_226_20231207T143234Z_gen_g4_detsim_hitreco__20240510T060915Z_reco2.root?xrdcl.requuid=9885cede-88b5-4869-9d41-08599acc52ec] Close returned from ds27.farm.particle.cz:20995 with: [SUCCESS] [2025-08-08 04:29:55.680866 +0100][Debug ][ExDbgMsg ] [ds27.farm.particle.cz:20995] Destroying MsgHandler: 0x15a76900. [2025-08-08 04:29:55.683139 +0100][Debug ][Utility ] Env: overriding entry: connectionwindow=30 with 30 [2025-08-08 04:29:55.683155 +0100][Debug ][Utility ] Env: overriding entry: requesttimeout=14400 with 4096 [2025-08-08 04:29:55.683162 +0100][Debug ][Utility ] Env: overriding entry: requesttimeout=4096 with 14400 [2025-08-08 04:29:55.683181 +0100][Debug ][Utility ] Env: overriding entry: redirectlimit=64 with 64 [2025-08-08 04:29:55.683195 +0100][Debug ][Utility ] Env: overriding entry: multiprotocol=1 with 1 [2025-08-08 04:29:55.683279 +0100][Debug ][File ] [0x15981150@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b3/ec/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_1008_20231203T134835Z_gen_g4_detsim_hitreco__20240508T042957Z_reco2.root?xrdcl.requuid=eab6c2ac-044b-424f-bcd1-cbb0377b2ad6] Sending an open command [2025-08-08 04:29:55.683294 +0100][Debug ][ExDbgMsg ] [se1.farm.particle.cz:1094] MsgHandler created: 0x15a76900 (message: kXR_open (file: /dune/RSE/fardet-hd/b3/ec/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_1008_20231203T134835Z_gen_g4_detsim_hitreco__20240508T042957Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ). [2025-08-08 04:29:55.683339 +0100][Debug ][ExDbgMsg ] [se1.farm.particle.cz:1094] Moving MsgHandler: 0x15a76900 (message: kXR_open (file: /dune/RSE/fardet-hd/b3/ec/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_1008_20231203T134835Z_gen_g4_detsim_hitreco__20240508T042957Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) from out-queu to in-queue. [2025-08-08 04:29:55.719331 +0100][Debug ][ExDbgMsg ] [msg: 0xc2728c0] Assigned MsgHandler: 0x15a76900. [2025-08-08 04:29:55.719346 +0100][Debug ][ExDbgMsg ] [handler: 0x15a76900] Removed MsgHandler: 0x15a76900 from the in-queue. [2025-08-08 04:29:55.719372 +0100][Debug ][XRootD ] [se1.farm.particle.cz:1094] Handling error while processing kXR_open (file: /dune/RSE/fardet-hd/b3/ec/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_1008_20231203T134835Z_gen_g4_detsim_hitreco__20240508T042957Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [ERROR] Error response: no such file or directory. [2025-08-08 04:29:55.719523 +0100][Debug ][ExDbgMsg ] [se1.farm.particle.cz:1094] Calling MsgHandler: 0x15a76900 (message: kXR_open (file: /dune/RSE/fardet-hd/b3/ec/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_1008_20231203T134835Z_gen_g4_detsim_hitreco__20240508T042957Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) with status: [ERROR] Error response: no such file or directory. [2025-08-08 04:29:55.719585 +0100][Debug ][File ] [0x15981150@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b3/ec/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_1008_20231203T134835Z_gen_g4_detsim_hitreco__20240508T042957Z_reco2.root?xrdcl.requuid=eab6c2ac-044b-424f-bcd1-cbb0377b2ad6] Open has returned with status [ERROR] Server responded with an error: [3011] No such file [2025-08-08 04:29:55.719593 +0100][Debug ][File ] [0x15981150@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b3/ec/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_1008_20231203T134835Z_gen_g4_detsim_hitreco__20240508T042957Z_reco2.root?xrdcl.requuid=eab6c2ac-044b-424f-bcd1-cbb0377b2ad6] Error while opening at se1.farm.particle.cz:1094: [ERROR] Server responded with an error: [3011] No such file [2025-08-08 04:29:55.719635 +0100][Debug ][ExDbgMsg ] [se1.farm.particle.cz:1094] Destroying MsgHandler: 0x15a76900. [2025-08-08 04:29:55.930952 +0100][Debug ][JobMgr ] Stopping the job manager... [2025-08-08 04:29:55.931225 +0100][Debug ][JobMgr ] Job manager stopped [2025-08-08 04:29:55.931276 +0100][Debug ][TaskMgr ] Stopping the task manager... [2025-08-08 04:29:55.931361 +0100][Debug ][TaskMgr ] Task manager stopped [2025-08-08 04:29:55.931367 +0100][Debug ][Poller ] Stopping the poller... [2025-08-08 04:29:55.931774 +0100][Debug ][AsyncSock ] [ds27.farm.particle.cz:20995.0] Closing the socket [2025-08-08 04:29:55.931870 +0100][Debug ][Poller ] <[::ffff:192.41.105.41]:45972><--><[::ffff:147.231.25.165]:20995> Removing socket from the poller [2025-08-08 04:29:55.931966 +0100][Debug ][PostMaster ] [ds27.farm.particle.cz:20995] Destroying stream [2025-08-08 04:29:55.932039 +0100][Debug ][AsyncSock ] [ds27.farm.particle.cz:20995.0] Closing the socket [2025-08-08 04:29:55.932088 +0100][Debug ][AsyncSock ] [se1.farm.particle.cz:1094.0] Closing the socket [2025-08-08 04:29:55.932093 +0100][Debug ][Poller ] <[::ffff:192.41.105.41]:45548><--><[::ffff:147.231.25.100]:1094> Removing socket from the poller [2025-08-08 04:29:55.932104 +0100][Debug ][PostMaster ] [se1.farm.particle.cz:1094] Destroying stream [2025-08-08 04:29:55.932108 +0100][Debug ][AsyncSock ] [se1.farm.particle.cz:1094.0] Closing the socket lar exit code 1 === Start last 100 lines of lar log file === tail: cannot open '_reco_20250808T032459Z.log' for reading: No such file or directory === End last 100 lines of lar log file === processed files root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/85/bd/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6457551_226_20231207T143234Z_gen_g4_detsim_hitreco__20240510T060915Z_reco2.root root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/b3/ec/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574832_1008_20231203T134835Z_gen_g4_detsim_hitreco__20240508T042957Z_reco2.root root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/02/a5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_1068_20231202T164130Z_gen_g4_detsim_hitreco__20240508T030215Z_reco2.root root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/3c/5c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50546594_474_20231202T073012Z_gen_g4_detsim_hitreco__20240508T063850Z_reco2.root .: total 16116 -rw-r--r-- 1 gl05pi6 eddie_users 10855235 Aug 8 04:29 jobscript.log -rw-r--r-- 1 gl05pi6 eddie_users 1865157 Aug 8 04:29 flatcaf.root -rw-r--r-- 1 gl05pi6 eddie_users 1400753 Aug 8 04:29 caf.root -rw-r--r-- 1 gl05pi6 eddie_users 1400753 Aug 8 04:29 caf_fd_hd_atmo_643_20250808T032459Z.root -rw-r--r-- 1 gl05pi6 eddie_users 935555 Aug 8 04:29 caf_20250808T032459Z.log -rw-r--r-- 1 gl05pi6 eddie_users 749 Aug 8 04:29 caf_20250808T032459Z.file -rw-r--r-- 1 gl05pi6 eddie_users 749 Aug 8 04:29 caf_20250808T032459Z.pfns -rw-r--r-- 1 gl05pi6 eddie_users 749 Aug 8 04:24 file.list -rw-r--r-- 1 gl05pi6 eddie_users 749 Aug 8 04:29 justin-processed-pfns.txt -rw-r--r-- 1 gl05pi6 eddie_users 553 Aug 8 04:24 all-input-dids.txt -rw-r--r-- 1 gl05pi6 eddie_users 553 Aug 8 04:29 caf_20250808T032459Z.did -rw-r--r-- 1 gl05pi6 eddie_users 553 Aug 8 04:24 did.list