justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 272704.108@dunegpschedd01.fnal.gov

Jobsub ID272704.108@dunegpschedd01.fnal.gov
Workflow ID11624
Stage ID1
User nameimawby@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes1048576000 (1000 MiB)
Wall seconds limit7200 (2 hours)
Submitted time2026-01-08 15:57:13
SiteUK_Edinburgh
EntryDUNE_UK_SGridECDF_ce1_multicore
Last heartbeat2026-01-08 16:11:51
From worker nodeHostnamenode2b06.ecdf.ed.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes1048576000 (1000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2026-01-08 15:57:55
Input filesfardet-hd:nu_dune10kt_1x2x6_1412_161_20230826T153302Z_gen_g4_detsim_hitreco__20240227T210653Z_reco2.root
JobscriptExit code0
Real time7m (430s)
CPU time4m (296s = 68%)
Max RSS bytes1175040000 (1120 MiB)
Outputting started2026-01-08 16:05:06
Output files
Finished2026-01-08 16:11:51
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

c.uk:50199] Calling MsgHandler: 0x16ffa930 (message: kXR_read (handle: 0x00000000, offset: 1264682284, size: 36525) ) with status: [SUCCESS] .
[2026-01-08 16:05:03.154562 +0000][Debug  ][ExDbgMsg          ][ 1305] [heplns131.pp.rl.ac.uk:50199] Destroying MsgHandler: 0x16ffa930.
[2026-01-08 16:05:03.163589 +0000][Debug  ][File              ][ 1305] [0xe5f72f0@root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/c1/30/nu_dune10kt_1x2x6_1412_161_20230826T153302Z_gen_g4_detsim_hitreco__20240227T210653Z_reco2.root?xrdcl.requuid=68ff1ef2-c156-46f8-b6b2-895df1140f7b] Sending a read command for handle 0x0 to heplns131.pp.rl.ac.uk:50199
[2026-01-08 16:05:03.163650 +0000][Debug  ][ExDbgMsg          ][ 1305] [heplns131.pp.rl.ac.uk:50199] MsgHandler created: 0x190316c0 (message: kXR_read (handle: 0x00000000, offset: 168532, size: 132) ).
[2026-01-08 16:05:03.163742 +0000][Debug  ][ExDbgMsg          ][ 1305] [heplns131.pp.rl.ac.uk:50199] Moving MsgHandler: 0x190316c0 (message: kXR_read (handle: 0x00000000, offset: 168532, size: 132) ) from out-queu to in-queue.
[2026-01-08 16:05:03.177611 +0000][Debug  ][ExDbgMsg          ][ 1305] [msg: 0x19032520] Assigned MsgHandler: 0x190316c0.
[2026-01-08 16:05:03.177673 +0000][Debug  ][ExDbgMsg          ][ 1305] [handler: 0x190316c0] Removed MsgHandler: 0x190316c0 from the in-queue.
[2026-01-08 16:05:03.177720 +0000][Debug  ][ExDbgMsg          ][ 1305] [heplns131.pp.rl.ac.uk:50199] Calling MsgHandler: 0x190316c0 (message: kXR_read (handle: 0x00000000, offset: 168532, size: 132) ) with status: [SUCCESS] .
[2026-01-08 16:05:03.177742 +0000][Debug  ][ExDbgMsg          ][ 1305] [heplns131.pp.rl.ac.uk:50199] Destroying MsgHandler: 0x190316c0.
[2026-01-08 16:05:03.177816 +0000][Debug  ][File              ][ 1305] [0xe5f72f0@root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/c1/30/nu_dune10kt_1x2x6_1412_161_20230826T153302Z_gen_g4_detsim_hitreco__20240227T210653Z_reco2.root?xrdcl.requuid=68ff1ef2-c156-46f8-b6b2-895df1140f7b] Sending a read command for handle 0x0 to heplns131.pp.rl.ac.uk:50199
[2026-01-08 16:05:03.177843 +0000][Debug  ][ExDbgMsg          ][ 1305] [heplns131.pp.rl.ac.uk:50199] MsgHandler created: 0x190316c0 (message: kXR_read (handle: 0x00000000, offset: 168664, size: 138) ).
[2026-01-08 16:05:03.177927 +0000][Debug  ][ExDbgMsg          ][ 1305] [heplns131.pp.rl.ac.uk:50199] Moving MsgHandler: 0x190316c0 (message: kXR_read (handle: 0x00000000, offset: 168664, size: 138) ) from out-queu to in-queue.
[2026-01-08 16:05:03.191804 +0000][Debug  ][ExDbgMsg          ][ 1305] [msg: 0x19031ce0] Assigned MsgHandler: 0x190316c0.
[2026-01-08 16:05:03.191860 +0000][Debug  ][ExDbgMsg          ][ 1305] [handler: 0x190316c0] Removed MsgHandler: 0x190316c0 from the in-queue.
[2026-01-08 16:05:03.191907 +0000][Debug  ][ExDbgMsg          ][ 1305] [heplns131.pp.rl.ac.uk:50199] Calling MsgHandler: 0x190316c0 (message: kXR_read (handle: 0x00000000, offset: 168664, size: 138) ) with status: [SUCCESS] .
[2026-01-08 16:05:03.191928 +0000][Debug  ][ExDbgMsg          ][ 1305] [heplns131.pp.rl.ac.uk:50199] Destroying MsgHandler: 0x190316c0.
[2026-01-08 16:05:03.192003 +0000][Debug  ][File              ][ 1305] [0xe5f72f0@root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/c1/30/nu_dune10kt_1x2x6_1412_161_20230826T153302Z_gen_g4_detsim_hitreco__20240227T210653Z_reco2.root?xrdcl.requuid=68ff1ef2-c156-46f8-b6b2-895df1140f7b] Sending a read command for handle 0x0 to heplns131.pp.rl.ac.uk:50199
[2026-01-08 16:05:03.192036 +0000][Debug  ][ExDbgMsg          ][ 1305] [heplns131.pp.rl.ac.uk:50199] MsgHandler created: 0xda7c110 (message: kXR_read (handle: 0x00000000, offset: 159645, size: 2405) ).
[2026-01-08 16:05:03.192116 +0000][Debug  ][ExDbgMsg          ][ 1305] [heplns131.pp.rl.ac.uk:50199] Moving MsgHandler: 0xda7c110 (message: kXR_read (handle: 0x00000000, offset: 159645, size: 2405) ) from out-queu to in-queue.
[2026-01-08 16:05:03.205943 +0000][Debug  ][ExDbgMsg          ][ 1305] [msg: 0x1e1db350] Assigned MsgHandler: 0xda7c110.
[2026-01-08 16:05:03.205987 +0000][Debug  ][ExDbgMsg          ][ 1305] [handler: 0xda7c110] Removed MsgHandler: 0xda7c110 from the in-queue.
[2026-01-08 16:05:03.206031 +0000][Debug  ][ExDbgMsg          ][ 1305] [heplns131.pp.rl.ac.uk:50199] Calling MsgHandler: 0xda7c110 (message: kXR_read (handle: 0x00000000, offset: 159645, size: 2405) ) with status: [SUCCESS] .
[2026-01-08 16:05:03.206066 +0000][Debug  ][ExDbgMsg          ][ 1305] [heplns131.pp.rl.ac.uk:50199] Destroying MsgHandler: 0xda7c110.
08-Jan-2026 16:05:04 GMT  Closed output file "nu_dune10kt_1x2x6_1412_161_20230826T153302Z_gen_g4_detsim_hitreco__20240227T210653Z_reco2_reco2.root"
[2026-01-08 16:05:04.772869 +0000][Debug  ][File              ][ 1305] [0xe5f72f0@root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/c1/30/nu_dune10kt_1x2x6_1412_161_20230826T153302Z_gen_g4_detsim_hitreco__20240227T210653Z_reco2.root?xrdcl.requuid=68ff1ef2-c156-46f8-b6b2-895df1140f7b] Sending a close command for handle 0x0 to heplns131.pp.rl.ac.uk:50199
[2026-01-08 16:05:04.772969 +0000][Debug  ][ExDbgMsg          ][ 1305] [heplns131.pp.rl.ac.uk:50199] MsgHandler created: 0xd32e520 (message: kXR_close (handle: 0x00000000) ).
[2026-01-08 16:05:04.773079 +0000][Debug  ][ExDbgMsg          ][ 1305] [heplns131.pp.rl.ac.uk:50199] Moving MsgHandler: 0xd32e520 (message: kXR_close (handle: 0x00000000) ) from out-queu to in-queue.
[2026-01-08 16:05:04.787037 +0000][Debug  ][ExDbgMsg          ][ 1305] [msg: 0x11ebf2c0] Assigned MsgHandler: 0xd32e520.
[2026-01-08 16:05:04.787046 +0000][Debug  ][ExDbgMsg          ][ 1305] [handler: 0xd32e520] Removed MsgHandler: 0xd32e520 from the in-queue.
[2026-01-08 16:05:04.787069 +0000][Debug  ][ExDbgMsg          ][ 1305] [heplns131.pp.rl.ac.uk:50199] Calling MsgHandler: 0xd32e520 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] .
[2026-01-08 16:05:04.787094 +0000][Debug  ][File              ][ 1305] [0xe5f72f0@root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/c1/30/nu_dune10kt_1x2x6_1412_161_20230826T153302Z_gen_g4_detsim_hitreco__20240227T210653Z_reco2.root?xrdcl.requuid=68ff1ef2-c156-46f8-b6b2-895df1140f7b] Close returned from heplns131.pp.rl.ac.uk:50199 with: [SUCCESS] 
[2026-01-08 16:05:04.787113 +0000][Debug  ][ExDbgMsg          ][ 1305] [heplns131.pp.rl.ac.uk:50199] Destroying MsgHandler: 0xd32e520.
08-Jan-2026 16:05:04 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/c1/30/nu_dune10kt_1x2x6_1412_161_20230826T153302Z_gen_g4_detsim_hitreco__20240227T210653Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      0.1256        2.70606       90.6757       1.4408        8.96515        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0290636     0.0616593     0.181259      0.0430896     0.0419929       100    
reco:pandora:StandardPandora                   0.0950514      2.63327       90.5274       1.37326       8.95542        100    
[art]:TriggerResults:TriggerResultInserter    1.2945e-05    2.23527e-05    4.841e-05     2.07e-05     7.75999e-06      100    
end_path:out1:RootOutput                       2.758e-06    3.57097e-06   1.3307e-05    3.2895e-06    1.17165e-06      100    
end_path:out1:RootOutput(write)               0.000869852    0.0109283     0.0819761    0.00744444     0.0130327       100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2162.36 MB
  Peak resident set size usage (VmHWM): 1175.04 MB
====================================================================================================
Art has completed and will exit with status 0.
[2026-01-08 16:05:05.473693 +0000][Debug  ][JobMgr            ][ 1305] Stopping the job manager...
[2026-01-08 16:05:05.474758 +0000][Debug  ][JobMgr            ][ 1305] Job manager stopped
[2026-01-08 16:05:05.475221 +0000][Debug  ][TaskMgr           ][ 1305] Stopping the task manager...
[2026-01-08 16:05:05.475433 +0000][Debug  ][TaskMgr           ][ 1305] Task manager stopped
[2026-01-08 16:05:05.475560 +0000][Debug  ][Poller            ][ 1305] Stopping the poller...
[2026-01-08 16:05:05.475911 +0000][Debug  ][AsyncSock         ][ 1305] [heplns131.pp.rl.ac.uk:50199.0] Closing the socket
[2026-01-08 16:05:05.475966 +0000][Debug  ][Poller            ][ 1305] <[::ffff:192.41.105.39]:56454><--><[::ffff:130.246.47.131]:50199> Removing socket from the poller
[2026-01-08 16:05:05.476056 +0000][Debug  ][PostMaster        ][ 1305] [heplns131.pp.rl.ac.uk:50199] Destroying stream
[2026-01-08 16:05:05.476128 +0000][Debug  ][AsyncSock         ][ 1305] [heplns131.pp.rl.ac.uk:50199.0] Closing the socket
[2026-01-08 16:05:05.476168 +0000][Debug  ][AsyncSock         ][ 1305] [mover.pp.rl.ac.uk:1094.0] Closing the socket
[2026-01-08 16:05:05.476176 +0000][Debug  ][Poller            ][ 1305] <[::ffff:192.41.105.39]:41658><--><[::ffff:130.246.47.231]:1094> Removing socket from the poller
[2026-01-08 16:05:05.476402 +0000][Debug  ][PostMaster        ][ 1305] [mover.pp.rl.ac.uk:1094] Destroying stream
[2026-01-08 16:05:05.476550 +0000][Debug  ][AsyncSock         ][ 1305] [mover.pp.rl.ac.uk:1094.0] Closing the socket
=== End last 100 lines of lar log file ===
justIN time: 2026-02-04 05:49:55 UTC       justIN version: 01.06.00