justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 272704.117@dunegpschedd01.fnal.gov

Jobsub ID272704.117@dunegpschedd01.fnal.gov
Workflow ID11624
Stage ID1
User nameimawby@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes1048576000 (1000 MiB)
Wall seconds limit7200 (2 hours)
Submitted time2026-01-08 15:57:13
SiteUK_Edinburgh
EntryDUNE_UK_SGridECDF_ce1_multicore
Last heartbeat2026-01-08 16:11:12
From worker nodeHostnamenode2b03.ecdf.ed.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes1048576000 (1000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2026-01-08 15:57:43
Input filesfardet-hd:nu_dune10kt_1x2x6_1098_779_20230826T013915Z_gen_g4_detsim_hitreco__20240229T184842Z_reco2.root
JobscriptExit code0
Real time6m (405s)
CPU time4m (276s = 68%)
Max RSS bytes1282490368 (1223 MiB)
Outputting started2026-01-08 16:04:28
Output files
Finished2026-01-08 16:11:12
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ac.uk:51861] Calling MsgHandler: 0x1a88b750 (message: kXR_read (handle: 0x00000000, offset: 1260329900, size: 7948) ) with status: [SUCCESS] .
[2026-01-08 16:04:26.357277 +0000][Debug  ][ExDbgMsg          ][ 1306] [heplns140.pp.rl.ac.uk:51861] Destroying MsgHandler: 0x1a88b750.
[2026-01-08 16:04:26.363136 +0000][Debug  ][File              ][ 1306] [0xd54a620@root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/d0/5d/nu_dune10kt_1x2x6_1098_779_20230826T013915Z_gen_g4_detsim_hitreco__20240229T184842Z_reco2.root?xrdcl.requuid=95804399-c5bc-41ff-ad1e-c8a78662788e] Sending a read command for handle 0x0 to heplns140.pp.rl.ac.uk:51861
[2026-01-08 16:04:26.363174 +0000][Debug  ][ExDbgMsg          ][ 1306] [heplns140.pp.rl.ac.uk:51861] MsgHandler created: 0xce20b70 (message: kXR_read (handle: 0x00000000, offset: 176266, size: 132) ).
[2026-01-08 16:04:26.363242 +0000][Debug  ][ExDbgMsg          ][ 1306] [heplns140.pp.rl.ac.uk:51861] Moving MsgHandler: 0xce20b70 (message: kXR_read (handle: 0x00000000, offset: 176266, size: 132) ) from out-queu to in-queue.
[2026-01-08 16:04:26.377179 +0000][Debug  ][ExDbgMsg          ][ 1306] [msg: 0x171ef300] Assigned MsgHandler: 0xce20b70.
[2026-01-08 16:04:26.377209 +0000][Debug  ][ExDbgMsg          ][ 1306] [handler: 0xce20b70] Removed MsgHandler: 0xce20b70 from the in-queue.
[2026-01-08 16:04:26.377240 +0000][Debug  ][ExDbgMsg          ][ 1306] [heplns140.pp.rl.ac.uk:51861] Calling MsgHandler: 0xce20b70 (message: kXR_read (handle: 0x00000000, offset: 176266, size: 132) ) with status: [SUCCESS] .
[2026-01-08 16:04:26.377254 +0000][Debug  ][ExDbgMsg          ][ 1306] [heplns140.pp.rl.ac.uk:51861] Destroying MsgHandler: 0xce20b70.
[2026-01-08 16:04:26.377298 +0000][Debug  ][File              ][ 1306] [0xd54a620@root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/d0/5d/nu_dune10kt_1x2x6_1098_779_20230826T013915Z_gen_g4_detsim_hitreco__20240229T184842Z_reco2.root?xrdcl.requuid=95804399-c5bc-41ff-ad1e-c8a78662788e] Sending a read command for handle 0x0 to heplns140.pp.rl.ac.uk:51861
[2026-01-08 16:04:26.377315 +0000][Debug  ][ExDbgMsg          ][ 1306] [heplns140.pp.rl.ac.uk:51861] MsgHandler created: 0xce20b70 (message: kXR_read (handle: 0x00000000, offset: 176398, size: 138) ).
[2026-01-08 16:04:26.377362 +0000][Debug  ][ExDbgMsg          ][ 1306] [heplns140.pp.rl.ac.uk:51861] Moving MsgHandler: 0xce20b70 (message: kXR_read (handle: 0x00000000, offset: 176398, size: 138) ) from out-queu to in-queue.
[2026-01-08 16:04:26.391327 +0000][Debug  ][ExDbgMsg          ][ 1306] [msg: 0x171eeea0] Assigned MsgHandler: 0xce20b70.
[2026-01-08 16:04:26.391364 +0000][Debug  ][ExDbgMsg          ][ 1306] [handler: 0xce20b70] Removed MsgHandler: 0xce20b70 from the in-queue.
[2026-01-08 16:04:26.391394 +0000][Debug  ][ExDbgMsg          ][ 1306] [heplns140.pp.rl.ac.uk:51861] Calling MsgHandler: 0xce20b70 (message: kXR_read (handle: 0x00000000, offset: 176398, size: 138) ) with status: [SUCCESS] .
[2026-01-08 16:04:26.391410 +0000][Debug  ][ExDbgMsg          ][ 1306] [heplns140.pp.rl.ac.uk:51861] Destroying MsgHandler: 0xce20b70.
[2026-01-08 16:04:26.391456 +0000][Debug  ][File              ][ 1306] [0xd54a620@root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/d0/5d/nu_dune10kt_1x2x6_1098_779_20230826T013915Z_gen_g4_detsim_hitreco__20240229T184842Z_reco2.root?xrdcl.requuid=95804399-c5bc-41ff-ad1e-c8a78662788e] Sending a read command for handle 0x0 to heplns140.pp.rl.ac.uk:51861
[2026-01-08 16:04:26.391490 +0000][Debug  ][ExDbgMsg          ][ 1306] [heplns140.pp.rl.ac.uk:51861] MsgHandler created: 0x171ee910 (message: kXR_read (handle: 0x00000000, offset: 166356, size: 3327) ).
[2026-01-08 16:04:26.391541 +0000][Debug  ][ExDbgMsg          ][ 1306] [heplns140.pp.rl.ac.uk:51861] Moving MsgHandler: 0x171ee910 (message: kXR_read (handle: 0x00000000, offset: 166356, size: 3327) ) from out-queu to in-queue.
[2026-01-08 16:04:26.405503 +0000][Debug  ][ExDbgMsg          ][ 1306] [msg: 0x171ef240] Assigned MsgHandler: 0x171ee910.
[2026-01-08 16:04:26.405562 +0000][Debug  ][ExDbgMsg          ][ 1306] [handler: 0x171ee910] Removed MsgHandler: 0x171ee910 from the in-queue.
[2026-01-08 16:04:26.405601 +0000][Debug  ][ExDbgMsg          ][ 1306] [heplns140.pp.rl.ac.uk:51861] Calling MsgHandler: 0x171ee910 (message: kXR_read (handle: 0x00000000, offset: 166356, size: 3327) ) with status: [SUCCESS] .
[2026-01-08 16:04:26.405620 +0000][Debug  ][ExDbgMsg          ][ 1306] [heplns140.pp.rl.ac.uk:51861] Destroying MsgHandler: 0x171ee910.
08-Jan-2026 16:04:27 GMT  Closed output file "nu_dune10kt_1x2x6_1098_779_20230826T013915Z_gen_g4_detsim_hitreco__20240229T184842Z_reco2_reco2.root"
[2026-01-08 16:04:27.702732 +0000][Debug  ][File              ][ 1306] [0xd54a620@root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/d0/5d/nu_dune10kt_1x2x6_1098_779_20230826T013915Z_gen_g4_detsim_hitreco__20240229T184842Z_reco2.root?xrdcl.requuid=95804399-c5bc-41ff-ad1e-c8a78662788e] Sending a close command for handle 0x0 to heplns140.pp.rl.ac.uk:51861
[2026-01-08 16:04:27.702870 +0000][Debug  ][ExDbgMsg          ][ 1306] [heplns140.pp.rl.ac.uk:51861] MsgHandler created: 0x11ae4e40 (message: kXR_close (handle: 0x00000000) ).
[2026-01-08 16:04:27.702970 +0000][Debug  ][ExDbgMsg          ][ 1306] [heplns140.pp.rl.ac.uk:51861] Moving MsgHandler: 0x11ae4e40 (message: kXR_close (handle: 0x00000000) ) from out-queu to in-queue.
[2026-01-08 16:04:27.717533 +0000][Debug  ][ExDbgMsg          ][ 1306] [msg: 0x135a1260] Assigned MsgHandler: 0x11ae4e40.
[2026-01-08 16:04:27.717542 +0000][Debug  ][ExDbgMsg          ][ 1306] [handler: 0x11ae4e40] Removed MsgHandler: 0x11ae4e40 from the in-queue.
[2026-01-08 16:04:27.717564 +0000][Debug  ][ExDbgMsg          ][ 1306] [heplns140.pp.rl.ac.uk:51861] Calling MsgHandler: 0x11ae4e40 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] .
[2026-01-08 16:04:27.717587 +0000][Debug  ][File              ][ 1306] [0xd54a620@root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/d0/5d/nu_dune10kt_1x2x6_1098_779_20230826T013915Z_gen_g4_detsim_hitreco__20240229T184842Z_reco2.root?xrdcl.requuid=95804399-c5bc-41ff-ad1e-c8a78662788e] Close returned from heplns140.pp.rl.ac.uk:51861 with: [SUCCESS] 
[2026-01-08 16:04:27.717606 +0000][Debug  ][ExDbgMsg          ][ 1306] [heplns140.pp.rl.ac.uk:51861] Destroying MsgHandler: 0x11ae4e40.
08-Jan-2026 16:04:27 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/d0/5d/nu_dune10kt_1x2x6_1098_779_20230826T013915Z_gen_g4_detsim_hitreco__20240229T184842Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.123535       2.48569       76.9253       1.36401       7.58639        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0289495     0.0458603     0.0904338     0.0478984    0.00977061       100    
reco:pandora:StandardPandora                   0.0936487      2.43077       76.7912       1.3098        7.57718        100    
[art]:TriggerResults:TriggerResultInserter    1.1305e-05    2.08547e-05   0.000119346   1.68485e-05   1.29855e-05      100    
end_path:out1:RootOutput                       2.238e-06    2.94798e-06   1.2602e-05    2.6485e-06    1.34252e-06      100    
end_path:out1:RootOutput(write)               0.000799442   0.00888285     0.0830656    0.00602032     0.0101234       100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2270.36 MB
  Peak resident set size usage (VmHWM): 1282.49 MB
====================================================================================================
Art has completed and will exit with status 0.
[2026-01-08 16:04:28.229251 +0000][Debug  ][JobMgr            ][ 1306] Stopping the job manager...
[2026-01-08 16:04:28.230208 +0000][Debug  ][JobMgr            ][ 1306] Job manager stopped
[2026-01-08 16:04:28.230716 +0000][Debug  ][TaskMgr           ][ 1306] Stopping the task manager...
[2026-01-08 16:04:28.230929 +0000][Debug  ][TaskMgr           ][ 1306] Task manager stopped
[2026-01-08 16:04:28.230946 +0000][Debug  ][Poller            ][ 1306] Stopping the poller...
[2026-01-08 16:04:28.231269 +0000][Debug  ][AsyncSock         ][ 1306] [heplns140.pp.rl.ac.uk:51861.0] Closing the socket
[2026-01-08 16:04:28.231339 +0000][Debug  ][Poller            ][ 1306] <[::ffff:192.41.105.36]:59892><--><[::ffff:130.246.47.140]:51861> Removing socket from the poller
[2026-01-08 16:04:28.231462 +0000][Debug  ][PostMaster        ][ 1306] [heplns140.pp.rl.ac.uk:51861] Destroying stream
[2026-01-08 16:04:28.231482 +0000][Debug  ][AsyncSock         ][ 1306] [heplns140.pp.rl.ac.uk:51861.0] Closing the socket
[2026-01-08 16:04:28.231507 +0000][Debug  ][AsyncSock         ][ 1306] [mover.pp.rl.ac.uk:1094.0] Closing the socket
[2026-01-08 16:04:28.231526 +0000][Debug  ][Poller            ][ 1306] <[::ffff:192.41.105.36]:53938><--><[::ffff:130.246.47.231]:1094> Removing socket from the poller
[2026-01-08 16:04:28.232026 +0000][Debug  ][PostMaster        ][ 1306] [mover.pp.rl.ac.uk:1094] Destroying stream
[2026-01-08 16:04:28.232066 +0000][Debug  ][AsyncSock         ][ 1306] [mover.pp.rl.ac.uk:1094.0] Closing the socket
=== End last 100 lines of lar log file ===
justIN time: 2026-02-04 07:05:01 UTC       justIN version: 01.06.00