justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 21967.20@dunegpschedd02.fnal.gov

Jobsub ID21967.20@dunegpschedd02.fnal.gov
Workflow ID1591
Stage ID1
User namelwhite86@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit3600 (1 hours)
Submitted time2025-08-19 12:09:56
SiteUK_Edinburgh
EntryDUNE_UK_SGridECDF_ce1_multicore
Last heartbeat2025-08-19 12:20:40
From worker nodeHostnamenode2b04.ecdf.ed.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-08-19 12:11:40
Input filesfardet-hd:nu_dune10kt_1x2x6_1429_879_20230828T004509Z_gen_g4_detsim_hitreco__20240405T212243Z_reco2.root
JobscriptExit code0
Real time8m (520s)
CPU time7m (461s = 88%)
Max RSS bytes1318891520 (1257 MiB)
Outputting started2025-08-19 12:20:22
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/lwhite86/fnal/01591/1/001/trainingFile_nu_dune10kt_1x2x6_1429_879_20230828T004509Z_gen_g4_detsim_hitreco__20240405T212243Z_reco2.root
Finished2025-08-19 12:20:40
Saved logsjustin-logs:21967.20-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ac.uk:50829] MsgHandler created: 0x1ccced20 (message: kXR_read (handle: 0x00000000, offset: 169535, size: 132) ).
[2025-08-19 13:20:18.673500 +0100][Debug  ][ExDbgMsg          ][ 1146] [heplns148.pp.rl.ac.uk:50829] Moving MsgHandler: 0x1ccced20 (message: kXR_read (handle: 0x00000000, offset: 169535, size: 132) ) from out-queu to in-queue.
[2025-08-19 13:20:18.687534 +0100][Debug  ][ExDbgMsg          ][ 1146] [msg: 0x1cccfcf0] Assigned MsgHandler: 0x1ccced20.
[2025-08-19 13:20:18.687554 +0100][Debug  ][ExDbgMsg          ][ 1146] [handler: 0x1ccced20] Removed MsgHandler: 0x1ccced20 from the in-queue.
[2025-08-19 13:20:18.687678 +0100][Debug  ][ExDbgMsg          ][ 1146] [heplns148.pp.rl.ac.uk:50829] Calling MsgHandler: 0x1ccced20 (message: kXR_read (handle: 0x00000000, offset: 169535, size: 132) ) with status: [SUCCESS] .
[2025-08-19 13:20:18.687711 +0100][Debug  ][ExDbgMsg          ][ 1146] [heplns148.pp.rl.ac.uk:50829] Destroying MsgHandler: 0x1ccced20.
[2025-08-19 13:20:18.687833 +0100][Debug  ][File              ][ 1146] [0xcc85cc0@root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/ff/92/nu_dune10kt_1x2x6_1429_879_20230828T004509Z_gen_g4_detsim_hitreco__20240405T212243Z_reco2.root?xrdcl.requuid=e257f296-1516-4dec-853a-5058bb6bdce0] Sending a read command for handle 0x0 to heplns148.pp.rl.ac.uk:50829
[2025-08-19 13:20:18.687869 +0100][Debug  ][ExDbgMsg          ][ 1146] [heplns148.pp.rl.ac.uk:50829] MsgHandler created: 0x1ccced00 (message: kXR_read (handle: 0x00000000, offset: 169667, size: 138) ).
[2025-08-19 13:20:18.687908 +0100][Debug  ][ExDbgMsg          ][ 1146] [heplns148.pp.rl.ac.uk:50829] Moving MsgHandler: 0x1ccced00 (message: kXR_read (handle: 0x00000000, offset: 169667, size: 138) ) from out-queu to in-queue.
[2025-08-19 13:20:18.701932 +0100][Debug  ][ExDbgMsg          ][ 1146] [msg: 0x1cccfa90] Assigned MsgHandler: 0x1ccced00.
[2025-08-19 13:20:18.701952 +0100][Debug  ][ExDbgMsg          ][ 1146] [handler: 0x1ccced00] Removed MsgHandler: 0x1ccced00 from the in-queue.
[2025-08-19 13:20:18.702061 +0100][Debug  ][ExDbgMsg          ][ 1146] [heplns148.pp.rl.ac.uk:50829] Calling MsgHandler: 0x1ccced00 (message: kXR_read (handle: 0x00000000, offset: 169667, size: 138) ) with status: [SUCCESS] .
[2025-08-19 13:20:18.702086 +0100][Debug  ][ExDbgMsg          ][ 1146] [heplns148.pp.rl.ac.uk:50829] Destroying MsgHandler: 0x1ccced00.
[2025-08-19 13:20:18.702193 +0100][Debug  ][File              ][ 1146] [0xcc85cc0@root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/ff/92/nu_dune10kt_1x2x6_1429_879_20230828T004509Z_gen_g4_detsim_hitreco__20240405T212243Z_reco2.root?xrdcl.requuid=e257f296-1516-4dec-853a-5058bb6bdce0] Sending a read command for handle 0x0 to heplns148.pp.rl.ac.uk:50829
[2025-08-19 13:20:18.702224 +0100][Debug  ][ExDbgMsg          ][ 1146] [heplns148.pp.rl.ac.uk:50829] MsgHandler created: 0x1ccd8150 (message: kXR_read (handle: 0x00000000, offset: 170550, size: 1939) ).
[2025-08-19 13:20:18.702260 +0100][Debug  ][ExDbgMsg          ][ 1146] [heplns148.pp.rl.ac.uk:50829] Moving MsgHandler: 0x1ccd8150 (message: kXR_read (handle: 0x00000000, offset: 170550, size: 1939) ) from out-queu to in-queue.
[2025-08-19 13:20:18.716302 +0100][Debug  ][ExDbgMsg          ][ 1146] [msg: 0x1ccd9060] Assigned MsgHandler: 0x1ccd8150.
[2025-08-19 13:20:18.716321 +0100][Debug  ][ExDbgMsg          ][ 1146] [handler: 0x1ccd8150] Removed MsgHandler: 0x1ccd8150 from the in-queue.
[2025-08-19 13:20:18.716451 +0100][Debug  ][ExDbgMsg          ][ 1146] [heplns148.pp.rl.ac.uk:50829] Calling MsgHandler: 0x1ccd8150 (message: kXR_read (handle: 0x00000000, offset: 170550, size: 1939) ) with status: [SUCCESS] .
[2025-08-19 13:20:18.716474 +0100][Debug  ][ExDbgMsg          ][ 1146] [heplns148.pp.rl.ac.uk:50829] Destroying MsgHandler: 0x1ccd8150.
19-Aug-2025 13:20:20 BST  Closed output file "nu_dune10kt_1x2x6_1429_879_20230828T004509Z_gen_g4_detsim_hitreco__20240405T212243Z_reco2_reco2.root"
[2025-08-19 13:20:20.816879 +0100][Debug  ][File              ][ 1146] [0xcc85cc0@root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/ff/92/nu_dune10kt_1x2x6_1429_879_20230828T004509Z_gen_g4_detsim_hitreco__20240405T212243Z_reco2.root?xrdcl.requuid=e257f296-1516-4dec-853a-5058bb6bdce0] Sending a close command for handle 0x0 to heplns148.pp.rl.ac.uk:50829
[2025-08-19 13:20:20.817058 +0100][Debug  ][ExDbgMsg          ][ 1146] [heplns148.pp.rl.ac.uk:50829] MsgHandler created: 0xb438c20 (message: kXR_close (handle: 0x00000000) ).
[2025-08-19 13:20:20.817221 +0100][Debug  ][ExDbgMsg          ][ 1146] [heplns148.pp.rl.ac.uk:50829] Moving MsgHandler: 0xb438c20 (message: kXR_close (handle: 0x00000000) ) from out-queu to in-queue.
[2025-08-19 13:20:20.831939 +0100][Debug  ][ExDbgMsg          ][ 1146] [msg: 0xda071e0] Assigned MsgHandler: 0xb438c20.
[2025-08-19 13:20:20.831976 +0100][Debug  ][ExDbgMsg          ][ 1146] [handler: 0xb438c20] Removed MsgHandler: 0xb438c20 from the in-queue.
[2025-08-19 13:20:20.832022 +0100][Debug  ][ExDbgMsg          ][ 1146] [heplns148.pp.rl.ac.uk:50829] Calling MsgHandler: 0xb438c20 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] .
[2025-08-19 13:20:20.832054 +0100][Debug  ][File              ][ 1146] [0xcc85cc0@root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/ff/92/nu_dune10kt_1x2x6_1429_879_20230828T004509Z_gen_g4_detsim_hitreco__20240405T212243Z_reco2.root?xrdcl.requuid=e257f296-1516-4dec-853a-5058bb6bdce0] Close returned from heplns148.pp.rl.ac.uk:50829 with: [SUCCESS] 
[2025-08-19 13:20:20.832073 +0100][Debug  ][ExDbgMsg          ][ 1146] [heplns148.pp.rl.ac.uk:50829] Destroying MsgHandler: 0xb438c20.
19-Aug-2025 13:20:20 BST  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/ff/92/nu_dune10kt_1x2x6_1429_879_20230828T004509Z_gen_g4_detsim_hitreco__20240405T212243Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.121121       4.48038       223.152       1.20693       22.2779        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0283953     0.0446407     0.141472      0.0470109     0.0145706       100    
reco:pandora2:StandardPandora                   0.09114       4.43513       223.103       1.1696        22.2771        100    
[art]:TriggerResults:TriggerResultInserter    1.0253e-05    1.87641e-05   0.000213802   1.36715e-05   2.19629e-05      100    
end_path:out1:RootOutput                       2.097e-06    2.57825e-06   1.0993e-05    2.4685e-06    8.89799e-07      100    
end_path:out1:RootOutput(write)               0.000133403   0.00044642    0.00248129    0.000241534   0.000475381      100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2303.39 MB
  Peak resident set size usage (VmHWM): 1318.89 MB
====================================================================================================
Art has completed and will exit with status 0.
[2025-08-19 13:20:21.219252 +0100][Debug  ][JobMgr            ][ 1146] Stopping the job manager...
[2025-08-19 13:20:21.219554 +0100][Debug  ][JobMgr            ][ 1146] Job manager stopped
[2025-08-19 13:20:21.219666 +0100][Debug  ][TaskMgr           ][ 1146] Stopping the task manager...
[2025-08-19 13:20:21.219763 +0100][Debug  ][TaskMgr           ][ 1146] Task manager stopped
[2025-08-19 13:20:21.219776 +0100][Debug  ][Poller            ][ 1146] Stopping the poller...
[2025-08-19 13:20:21.219886 +0100][Debug  ][AsyncSock         ][ 1146] [heplns148.pp.rl.ac.uk:50829.0] Closing the socket
[2025-08-19 13:20:21.219981 +0100][Debug  ][Poller            ][ 1146] <[::ffff:192.41.105.37]:45932><--><[::ffff:130.246.47.148]:50829> Removing socket from the poller
[2025-08-19 13:20:21.220055 +0100][Debug  ][PostMaster        ][ 1146] [heplns148.pp.rl.ac.uk:50829] Destroying stream
[2025-08-19 13:20:21.220075 +0100][Debug  ][AsyncSock         ][ 1146] [heplns148.pp.rl.ac.uk:50829.0] Closing the socket
[2025-08-19 13:20:21.220170 +0100][Debug  ][AsyncSock         ][ 1146] [mover.pp.rl.ac.uk:1094.0] Closing the socket
[2025-08-19 13:20:21.220201 +0100][Debug  ][Poller            ][ 1146] <[::ffff:192.41.105.37]:41964><--><[::ffff:130.246.47.231]:1094> Removing socket from the poller
[2025-08-19 13:20:21.220350 +0100][Debug  ][PostMaster        ][ 1146] [mover.pp.rl.ac.uk:1094] Destroying stream
[2025-08-19 13:20:21.220574 +0100][Debug  ][AsyncSock         ][ 1146] [mover.pp.rl.ac.uk:1094.0] Closing the socket
lar exit code 0
total 4708
-rw-r--r-- 1 gl05pi6 eddie_users     210 Aug 19 13:11 all-input-dids.txt
-rw-r--r-- 1 gl05pi6 eddie_users       0 Aug 19 13:11 debugprod.log
-rw-r--r-- 1 gl05pi6 eddie_users 3031909 Aug 19 13:20 jobscript.log
-rw-r--r-- 1 gl05pi6 eddie_users     168 Aug 19 13:20 justin-processed-pfns.txt
drwxr-xr-x 4 gl05pi6 eddie_users    4096 Aug 19 13:11 larpandoracontent
-rw-r--r-- 1 gl05pi6 eddie_users  354842 Aug 19 13:20 nu_dune10kt_1x2x6_1429_879_20230828T004509Z_gen_g4_detsim_hitreco__20240405T212243Z_reco2_reco2.root
-rw-r--r-- 1 gl05pi6 eddie_users     519 Aug 19 13:20 reco2_hist.root
-rw-r--r-- 1 gl05pi6 eddie_users 1400898 Aug 19 13:20 trainingFile_nu_dune10kt_1x2x6_1429_879_20230828T004509Z_gen_g4_detsim_hitreco__20240405T212243Z_reco2.root
justIN time: 2025-11-05 09:03:33 UTC       justIN version: 01.05.01