Jobsub ID 15379.1@dunegpschedd02.fnal.gov
| Jobsub ID | 15379.1@dunegpschedd02.fnal.gov | |
| Workflow ID | 648 | |
| Stage ID | 1 | |
| User name | pgranger@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 18000 (5 hours) | |
| Submitted time | 2025-08-08 02:20:23 | |
| Site | UK_Edinburgh | |
| Entry | DUNE_UK_SGridECDF_ce1 | |
| Last heartbeat | 2025-08-08 02:33:16 | |
| From worker node | Hostname | node2b08.ecdf.ed.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 171000 (47 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Started | 2025-08-08 02:20:53 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_608_20231207T120529Z_gen_g4_detsim_hitreco__20240510T055437Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_1111_20231203T125003Z_gen_g4_detsim_hitreco__20240509T192323Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74581430_697_20231208T082350Z_gen_g4_detsim_hitreco__20240510T062742Z_reco2.root fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_730_20231207T031744Z_gen_g4_detsim_hitreco__20240510T050721Z_reco2.root | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-08-08 02:33:16 | |
| Saved logs | justin-logs:15379.1-dunegpschedd02.fnal.gov.logs.tgz | |
| List job events Cached HTCondor job logs | ||
Jobscript log (last 10,000 characters)
a6a-9696-46c8-bae3-1f72f20c6c95] Sending a close command for handle 0x0 to ccdcacli530.in2p3.fr:30402 [2025-08-08 03:32:47.060102 +0100][Debug ][ExDbgMsg ] [ccdcacli530.in2p3.fr:30402] MsgHandler created: 0x1b404600 (message: kXR_close (handle: 0x00000000) ). [2025-08-08 03:32:47.060178 +0100][Debug ][ExDbgMsg ] [ccdcacli530.in2p3.fr:30402] Moving MsgHandler: 0x1b404600 (message: kXR_close (handle: 0x00000000) ) from out-queu to in-queue. [2025-08-08 03:32:47.088162 +0100][Debug ][ExDbgMsg ] [msg: 0xcb06000] Assigned MsgHandler: 0x1b404600. [2025-08-08 03:32:47.088446 +0100][Debug ][ExDbgMsg ] [handler: 0x1b404600] Removed MsgHandler: 0x1b404600 from the in-queue. [2025-08-08 03:32:47.088485 +0100][Debug ][ExDbgMsg ] [ccdcacli530.in2p3.fr:30402] Calling MsgHandler: 0x1b404600 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] . [2025-08-08 03:32:47.088496 +0100][Debug ][File ] [0xac0f230@root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/6e/d0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74581430_697_20231208T082350Z_gen_g4_detsim_hitreco__20240510T062742Z_reco2.root?xrdcl.requuid=154f2a6a-9696-46c8-bae3-1f72f20c6c95] Close returned from ccdcacli530.in2p3.fr:30402 with: [SUCCESS] [2025-08-08 03:32:47.088510 +0100][Debug ][ExDbgMsg ] [ccdcacli530.in2p3.fr:30402] Destroying MsgHandler: 0x1b404600. [2025-08-08 03:32:47.090357 +0100][Debug ][Utility ] Env: overriding entry: connectionwindow=30 with 30 [2025-08-08 03:32:47.090372 +0100][Debug ][Utility ] Env: overriding entry: requesttimeout=14400 with 4096 [2025-08-08 03:32:47.090380 +0100][Debug ][Utility ] Env: overriding entry: requesttimeout=4096 with 14400 [2025-08-08 03:32:47.090396 +0100][Debug ][Utility ] Env: overriding entry: redirectlimit=64 with 64 [2025-08-08 03:32:47.090409 +0100][Debug ][Utility ] Env: overriding entry: multiprotocol=1 with 1 [2025-08-08 03:32:47.090505 +0100][Debug ][File ] [0x1c616e90@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/09/dc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_730_20231207T031744Z_gen_g4_detsim_hitreco__20240510T050721Z_reco2.root?xrdcl.requuid=6f2e2986-f0b1-4612-a123-3012c9633f57] Sending an open command [2025-08-08 03:32:47.090526 +0100][Debug ][ExDbgMsg ] [se1.farm.particle.cz:1094] MsgHandler created: 0x1b404600 (message: kXR_open (file: /dune/RSE/fardet-hd/09/dc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_730_20231207T031744Z_gen_g4_detsim_hitreco__20240510T050721Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ). [2025-08-08 03:32:47.090577 +0100][Debug ][ExDbgMsg ] [se1.farm.particle.cz:1094] Moving MsgHandler: 0x1b404600 (message: kXR_open (file: /dune/RSE/fardet-hd/09/dc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_730_20231207T031744Z_gen_g4_detsim_hitreco__20240510T050721Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) from out-queu to in-queue. [2025-08-08 03:32:47.126632 +0100][Debug ][ExDbgMsg ] [msg: 0x1a740620] Assigned MsgHandler: 0x1b404600. [2025-08-08 03:32:47.126640 +0100][Debug ][ExDbgMsg ] [handler: 0x1b404600] Removed MsgHandler: 0x1b404600 from the in-queue. [2025-08-08 03:32:47.126671 +0100][Debug ][XRootD ] [se1.farm.particle.cz:1094] Handling error while processing kXR_open (file: /dune/RSE/fardet-hd/09/dc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_730_20231207T031744Z_gen_g4_detsim_hitreco__20240510T050721Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [ERROR] Error response: no such file or directory. [2025-08-08 03:32:47.126897 +0100][Debug ][ExDbgMsg ] [se1.farm.particle.cz:1094] Calling MsgHandler: 0x1b404600 (message: kXR_open (file: /dune/RSE/fardet-hd/09/dc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_730_20231207T031744Z_gen_g4_detsim_hitreco__20240510T050721Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) with status: [ERROR] Error response: no such file or directory. [2025-08-08 03:32:47.126980 +0100][Debug ][File ] [0x1c616e90@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/09/dc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_730_20231207T031744Z_gen_g4_detsim_hitreco__20240510T050721Z_reco2.root?xrdcl.requuid=6f2e2986-f0b1-4612-a123-3012c9633f57] Open has returned with status [ERROR] Server responded with an error: [3011] No such file [2025-08-08 03:32:47.126989 +0100][Debug ][File ] [0x1c616e90@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/09/dc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_730_20231207T031744Z_gen_g4_detsim_hitreco__20240510T050721Z_reco2.root?xrdcl.requuid=6f2e2986-f0b1-4612-a123-3012c9633f57] Error while opening at se1.farm.particle.cz:1094: [ERROR] Server responded with an error: [3011] No such file [2025-08-08 03:32:47.127046 +0100][Debug ][ExDbgMsg ] [se1.farm.particle.cz:1094] Destroying MsgHandler: 0x1b404600. [2025-08-08 03:32:47.360224 +0100][Debug ][JobMgr ] Stopping the job manager... [2025-08-08 03:32:47.360553 +0100][Debug ][JobMgr ] Job manager stopped [2025-08-08 03:32:47.360613 +0100][Debug ][TaskMgr ] Stopping the task manager... [2025-08-08 03:32:47.360699 +0100][Debug ][TaskMgr ] Task manager stopped [2025-08-08 03:32:47.360727 +0100][Debug ][Poller ] Stopping the poller... [2025-08-08 03:32:47.360990 +0100][Debug ][AsyncSock ] [ccdcacli530.in2p3.fr:30402.0] Closing the socket [2025-08-08 03:32:47.361007 +0100][Debug ][Poller ] <[::ffff:192.41.105.41]:54148><--><[::ffff:134.158.209.140]:30402> Removing socket from the poller [2025-08-08 03:32:47.361043 +0100][Debug ][PostMaster ] [ccdcacli530.in2p3.fr:30402] Destroying stream [2025-08-08 03:32:47.361049 +0100][Debug ][AsyncSock ] [ccdcacli530.in2p3.fr:30402.0] Closing the socket [2025-08-08 03:32:47.361064 +0100][Debug ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] Closing the socket [2025-08-08 03:32:47.361072 +0100][Debug ][Poller ] <[::ffff:192.41.105.41]:44382><--><[::ffff:134.158.209.218]:1094> Removing socket from the poller [2025-08-08 03:32:47.361395 +0100][Debug ][PostMaster ] [ccxrootdegee.in2p3.fr:1094] Destroying stream [2025-08-08 03:32:47.361443 +0100][Debug ][AsyncSock ] [ccxrootdegee.in2p3.fr:1094.0] Closing the socket [2025-08-08 03:32:47.361503 +0100][Debug ][AsyncSock ] [dune.dcache.nikhef.nl:1094.0] Closing the socket [2025-08-08 03:32:47.361540 +0100][Debug ][Poller ] <[::ffff:192.41.105.41]:45192><--><[::ffff:194.171.96.151]:1094> Removing socket from the poller [2025-08-08 03:32:47.361552 +0100][Debug ][PostMaster ] [dune.dcache.nikhef.nl:1094] Destroying stream [2025-08-08 03:32:47.361557 +0100][Debug ][AsyncSock ] [dune.dcache.nikhef.nl:1094.0] Closing the socket [2025-08-08 03:32:47.361565 +0100][Debug ][AsyncSock ] [hooipolder-01.nikhef.nl:33230.0] Closing the socket [2025-08-08 03:32:47.361570 +0100][Debug ][Poller ] <[::ffff:192.41.105.41]:34982><--><[::ffff:194.171.96.149]:33230> Removing socket from the poller [2025-08-08 03:32:47.361579 +0100][Debug ][PostMaster ] [hooipolder-01.nikhef.nl:33230] Destroying stream [2025-08-08 03:32:47.361586 +0100][Debug ][AsyncSock ] [hooipolder-01.nikhef.nl:33230.0] Closing the socket [2025-08-08 03:32:47.361596 +0100][Debug ][AsyncSock ] [se1.farm.particle.cz:1094.0] Closing the socket [2025-08-08 03:32:47.361603 +0100][Debug ][Poller ] <[::ffff:192.41.105.41]:46770><--><[::ffff:147.231.25.100]:1094> Removing socket from the poller [2025-08-08 03:32:47.361625 +0100][Debug ][PostMaster ] [se1.farm.particle.cz:1094] Destroying stream [2025-08-08 03:32:47.361630 +0100][Debug ][AsyncSock ] [se1.farm.particle.cz:1094.0] Closing the socket lar exit code 1 === Start last 100 lines of lar log file === tail: cannot open '_reco_20250808T022101Z.log' for reading: No such file or directory === End last 100 lines of lar log file === processed files root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/9e/f5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_608_20231207T120529Z_gen_g4_detsim_hitreco__20240510T055437Z_reco2.root root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/8f/b4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_1111_20231203T125003Z_gen_g4_detsim_hitreco__20240509T192323Z_reco2.root root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/6e/d0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74581430_697_20231208T082350Z_gen_g4_detsim_hitreco__20240510T062742Z_reco2.root root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/09/dc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_730_20231207T031744Z_gen_g4_detsim_hitreco__20240510T050721Z_reco2.root .: total 52924 -rw-r--r-- 1 gl05pi6 eddie_users 32568552 Aug 8 03:32 jobscript.log -rw-r--r-- 1 gl05pi6 eddie_users 7074030 Aug 8 03:32 flatcaf.root -rw-r--r-- 1 gl05pi6 eddie_users 5850259 Aug 8 03:32 caf.root -rw-r--r-- 1 gl05pi6 eddie_users 5850259 Aug 8 03:32 caf_fd_hd_atmo_648_20250808T022101Z.root -rw-r--r-- 1 gl05pi6 eddie_users 2793273 Aug 8 03:32 caf_20250808T022101Z.log -rw-r--r-- 1 gl05pi6 eddie_users 798 Aug 8 03:32 caf_20250808T022101Z.file -rw-r--r-- 1 gl05pi6 eddie_users 798 Aug 8 03:32 caf_20250808T022101Z.pfns -rw-r--r-- 1 gl05pi6 eddie_users 798 Aug 8 03:20 file.list -rw-r--r-- 1 gl05pi6 eddie_users 798 Aug 8 03:32 justin-processed-pfns.txt -rw-r--r-- 1 gl05pi6 eddie_users 552 Aug 8 03:20 all-input-dids.txt -rw-r--r-- 1 gl05pi6 eddie_users 552 Aug 8 03:32 caf_20250808T022101Z.did -rw-r--r-- 1 gl05pi6 eddie_users 552 Aug 8 03:20 did.list