justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 272704.97@dunegpschedd01.fnal.gov

Jobsub ID272704.97@dunegpschedd01.fnal.gov
Workflow ID11624
Stage ID1
User nameimawby@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes1048576000 (1000 MiB)
Wall seconds limit7200 (2 hours)
Submitted time2026-01-08 15:57:13
SiteUK_Edinburgh
EntryDUNE_UK_SGridECDF_ce1_multicore
Last heartbeat2026-01-08 16:10:02
From worker nodeHostnamenode2b05.ecdf.ed.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes1048576000 (1000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2026-01-08 15:57:55
Input filesfardet-hd:nu_dune10kt_1x2x6_1408_143_20230826T093314Z_gen_g4_detsim_hitreco__20240227T211803Z_reco2.root
JobscriptExit code0
Real time5m (320s)
CPU time3m (190s = 59%)
Max RSS bytes1113600000 (1062 MiB)
Outputting started2026-01-08 16:03:16
Output files
Finished2026-01-08 16:10:02
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

1129] Calling MsgHandler: 0x1873bd10 (message: kXR_read (handle: 0x00000000, offset: 1090516026, size: 43332) ) with status: [SUCCESS] .
[2026-01-08 16:03:13.560357 +0000][Debug  ][ExDbgMsg          ][ 1307] [heplns132.pp.rl.ac.uk:51129] Destroying MsgHandler: 0x1873bd10.
[2026-01-08 16:03:13.568239 +0000][Debug  ][File              ][ 1307] [0xb4f0540@root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/44/0e/nu_dune10kt_1x2x6_1408_143_20230826T093314Z_gen_g4_detsim_hitreco__20240227T211803Z_reco2.root?xrdcl.requuid=a3f7f07b-136c-40cb-8acd-f64790dcf89e] Sending a read command for handle 0x0 to heplns132.pp.rl.ac.uk:51129
[2026-01-08 16:03:13.568274 +0000][Debug  ][ExDbgMsg          ][ 1307] [heplns132.pp.rl.ac.uk:51129] MsgHandler created: 0x18f46ab0 (message: kXR_read (handle: 0x00000000, offset: 168835, size: 132) ).
[2026-01-08 16:03:13.568338 +0000][Debug  ][ExDbgMsg          ][ 1307] [heplns132.pp.rl.ac.uk:51129] Moving MsgHandler: 0x18f46ab0 (message: kXR_read (handle: 0x00000000, offset: 168835, size: 132) ) from out-queu to in-queue.
[2026-01-08 16:03:13.581993 +0000][Debug  ][ExDbgMsg          ][ 1307] [msg: 0x18966120] Assigned MsgHandler: 0x18f46ab0.
[2026-01-08 16:03:13.582000 +0000][Debug  ][ExDbgMsg          ][ 1307] [handler: 0x18f46ab0] Removed MsgHandler: 0x18f46ab0 from the in-queue.
[2026-01-08 16:03:13.582021 +0000][Debug  ][ExDbgMsg          ][ 1307] [heplns132.pp.rl.ac.uk:51129] Calling MsgHandler: 0x18f46ab0 (message: kXR_read (handle: 0x00000000, offset: 168835, size: 132) ) with status: [SUCCESS] .
[2026-01-08 16:03:13.582032 +0000][Debug  ][ExDbgMsg          ][ 1307] [heplns132.pp.rl.ac.uk:51129] Destroying MsgHandler: 0x18f46ab0.
[2026-01-08 16:03:13.582067 +0000][Debug  ][File              ][ 1307] [0xb4f0540@root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/44/0e/nu_dune10kt_1x2x6_1408_143_20230826T093314Z_gen_g4_detsim_hitreco__20240227T211803Z_reco2.root?xrdcl.requuid=a3f7f07b-136c-40cb-8acd-f64790dcf89e] Sending a read command for handle 0x0 to heplns132.pp.rl.ac.uk:51129
[2026-01-08 16:03:13.582231 +0000][Debug  ][ExDbgMsg          ][ 1307] [heplns132.pp.rl.ac.uk:51129] MsgHandler created: 0x18f46ab0 (message: kXR_read (handle: 0x00000000, offset: 168967, size: 138) ).
[2026-01-08 16:03:13.582267 +0000][Debug  ][ExDbgMsg          ][ 1307] [heplns132.pp.rl.ac.uk:51129] Moving MsgHandler: 0x18f46ab0 (message: kXR_read (handle: 0x00000000, offset: 168967, size: 138) ) from out-queu to in-queue.
[2026-01-08 16:03:13.595956 +0000][Debug  ][ExDbgMsg          ][ 1307] [msg: 0x18966120] Assigned MsgHandler: 0x18f46ab0.
[2026-01-08 16:03:13.595963 +0000][Debug  ][ExDbgMsg          ][ 1307] [handler: 0x18f46ab0] Removed MsgHandler: 0x18f46ab0 from the in-queue.
[2026-01-08 16:03:13.595981 +0000][Debug  ][ExDbgMsg          ][ 1307] [heplns132.pp.rl.ac.uk:51129] Calling MsgHandler: 0x18f46ab0 (message: kXR_read (handle: 0x00000000, offset: 168967, size: 138) ) with status: [SUCCESS] .
[2026-01-08 16:03:13.595992 +0000][Debug  ][ExDbgMsg          ][ 1307] [heplns132.pp.rl.ac.uk:51129] Destroying MsgHandler: 0x18f46ab0.
[2026-01-08 16:03:13.596023 +0000][Debug  ][File              ][ 1307] [0xb4f0540@root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/44/0e/nu_dune10kt_1x2x6_1408_143_20230826T093314Z_gen_g4_detsim_hitreco__20240227T211803Z_reco2.root?xrdcl.requuid=a3f7f07b-136c-40cb-8acd-f64790dcf89e] Sending a read command for handle 0x0 to heplns132.pp.rl.ac.uk:51129
[2026-01-08 16:03:13.596036 +0000][Debug  ][ExDbgMsg          ][ 1307] [heplns132.pp.rl.ac.uk:51129] MsgHandler created: 0x19005e20 (message: kXR_read (handle: 0x00000000, offset: 169693, size: 1157) ).
[2026-01-08 16:03:13.596062 +0000][Debug  ][ExDbgMsg          ][ 1307] [heplns132.pp.rl.ac.uk:51129] Moving MsgHandler: 0x19005e20 (message: kXR_read (handle: 0x00000000, offset: 169693, size: 1157) ) from out-queu to in-queue.
[2026-01-08 16:03:13.609837 +0000][Debug  ][ExDbgMsg          ][ 1307] [msg: 0x158d3be0] Assigned MsgHandler: 0x19005e20.
[2026-01-08 16:03:13.609879 +0000][Debug  ][ExDbgMsg          ][ 1307] [handler: 0x19005e20] Removed MsgHandler: 0x19005e20 from the in-queue.
[2026-01-08 16:03:13.609922 +0000][Debug  ][ExDbgMsg          ][ 1307] [heplns132.pp.rl.ac.uk:51129] Calling MsgHandler: 0x19005e20 (message: kXR_read (handle: 0x00000000, offset: 169693, size: 1157) ) with status: [SUCCESS] .
[2026-01-08 16:03:13.609941 +0000][Debug  ][ExDbgMsg          ][ 1307] [heplns132.pp.rl.ac.uk:51129] Destroying MsgHandler: 0x19005e20.
08-Jan-2026 16:03:15 GMT  Closed output file "nu_dune10kt_1x2x6_1408_143_20230826T093314Z_gen_g4_detsim_hitreco__20240227T211803Z_reco2_reco2.root"
[2026-01-08 16:03:15.082213 +0000][Debug  ][File              ][ 1307] [0xb4f0540@root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/44/0e/nu_dune10kt_1x2x6_1408_143_20230826T093314Z_gen_g4_detsim_hitreco__20240227T211803Z_reco2.root?xrdcl.requuid=a3f7f07b-136c-40cb-8acd-f64790dcf89e] Sending a close command for handle 0x0 to heplns132.pp.rl.ac.uk:51129
[2026-01-08 16:03:15.082254 +0000][Debug  ][ExDbgMsg          ][ 1307] [heplns132.pp.rl.ac.uk:51129] MsgHandler created: 0xab2dbe0 (message: kXR_close (handle: 0x00000000) ).
[2026-01-08 16:03:15.082333 +0000][Debug  ][ExDbgMsg          ][ 1307] [heplns132.pp.rl.ac.uk:51129] Moving MsgHandler: 0xab2dbe0 (message: kXR_close (handle: 0x00000000) ) from out-queu to in-queue.
[2026-01-08 16:03:15.096479 +0000][Debug  ][ExDbgMsg          ][ 1307] [msg: 0x12259190] Assigned MsgHandler: 0xab2dbe0.
[2026-01-08 16:03:15.096487 +0000][Debug  ][ExDbgMsg          ][ 1307] [handler: 0xab2dbe0] Removed MsgHandler: 0xab2dbe0 from the in-queue.
[2026-01-08 16:03:15.096510 +0000][Debug  ][ExDbgMsg          ][ 1307] [heplns132.pp.rl.ac.uk:51129] Calling MsgHandler: 0xab2dbe0 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] .
[2026-01-08 16:03:15.096544 +0000][Debug  ][File              ][ 1307] [0xb4f0540@root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/44/0e/nu_dune10kt_1x2x6_1408_143_20230826T093314Z_gen_g4_detsim_hitreco__20240227T211803Z_reco2.root?xrdcl.requuid=a3f7f07b-136c-40cb-8acd-f64790dcf89e] Close returned from heplns132.pp.rl.ac.uk:51129 with: [SUCCESS] 
[2026-01-08 16:03:15.096561 +0000][Debug  ][ExDbgMsg          ][ 1307] [heplns132.pp.rl.ac.uk:51129] Destroying MsgHandler: 0xab2dbe0.
08-Jan-2026 16:03:15 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/44/0e/nu_dune10kt_1x2x6_1408_143_20230826T093314Z_gen_g4_detsim_hitreco__20240227T211803Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      1.01807       1.69138       12.4749       1.35582       1.48403        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0283857     0.0732581     0.146613      0.0455955     0.0446181       100    
reco:pandora:StandardPandora                   0.987363       1.60911       12.4131       1.28804       1.47634        100    
[art]:TriggerResults:TriggerResultInserter    1.0875e-05    2.33919e-05   0.00032173    1.5748e-05    3.68055e-05      100    
end_path:out1:RootOutput                       2.067e-06    2.81865e-06   1.1463e-05     2.69e-06     1.00173e-06      100    
end_path:out1:RootOutput(write)               0.000918608   0.00885193     0.0569587    0.00682259    0.00829428       100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2099.33 MB
  Peak resident set size usage (VmHWM): 1113.6 MB
====================================================================================================
Art has completed and will exit with status 0.
[2026-01-08 16:03:15.684997 +0000][Debug  ][JobMgr            ][ 1307] Stopping the job manager...
[2026-01-08 16:03:15.685900 +0000][Debug  ][JobMgr            ][ 1307] Job manager stopped
[2026-01-08 16:03:15.686338 +0000][Debug  ][TaskMgr           ][ 1307] Stopping the task manager...
[2026-01-08 16:03:15.686532 +0000][Debug  ][TaskMgr           ][ 1307] Task manager stopped
[2026-01-08 16:03:15.686651 +0000][Debug  ][Poller            ][ 1307] Stopping the poller...
[2026-01-08 16:03:15.686871 +0000][Debug  ][AsyncSock         ][ 1307] [heplns132.pp.rl.ac.uk:51129.0] Closing the socket
[2026-01-08 16:03:15.686942 +0000][Debug  ][Poller            ][ 1307] <[::ffff:192.41.105.38]:39118><--><[::ffff:130.246.47.132]:51129> Removing socket from the poller
[2026-01-08 16:03:15.687015 +0000][Debug  ][PostMaster        ][ 1307] [heplns132.pp.rl.ac.uk:51129] Destroying stream
[2026-01-08 16:03:15.687044 +0000][Debug  ][AsyncSock         ][ 1307] [heplns132.pp.rl.ac.uk:51129.0] Closing the socket
[2026-01-08 16:03:15.687160 +0000][Debug  ][AsyncSock         ][ 1307] [mover.pp.rl.ac.uk:1094.0] Closing the socket
[2026-01-08 16:03:15.687173 +0000][Debug  ][Poller            ][ 1307] <[::ffff:192.41.105.38]:51376><--><[::ffff:130.246.47.230]:1094> Removing socket from the poller
[2026-01-08 16:03:15.687333 +0000][Debug  ][PostMaster        ][ 1307] [mover.pp.rl.ac.uk:1094] Destroying stream
[2026-01-08 16:03:15.687354 +0000][Debug  ][AsyncSock         ][ 1307] [mover.pp.rl.ac.uk:1094.0] Closing the socket
=== End last 100 lines of lar log file ===
justIN time: 2026-02-04 06:08:02 UTC       justIN version: 01.06.00