justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 14773.26@dunegpschedd02.fnal.gov

Jobsub ID14773.26@dunegpschedd02.fnal.gov
Workflow ID523
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod.mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-08-06 08:28:19
SiteUK_Edinburgh
EntryDUNE_UK_SGridECDF_ce1_multicore
Last heartbeat2025-08-06 09:52:01
From worker nodeHostnamenode2b04.ecdf.ed.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-08-06 08:41:41
Input filesfardet-vd:prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250728T130352Z_gen_003703_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-08-06 09:52:01
Saved logsjustin-logs:14773.26-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

age2_detsim_reco.root?xrdcl.requuid=157d4982-5426-4b11-8aff-49577f182c6e] Sending a read command for handle 0x0 to stkendca2020.fnal.gov:23365
[2025-08-06 10:51:38.678138 +0100][Debug  ][ExDbgMsg          ] [stkendca2020.fnal.gov:23365] MsgHandler created: 0x17fab260 (message: kXR_read (handle: 0x00000000, offset: 2949187202, size: 809690) ).
[2025-08-06 10:51:38.678339 +0100][Debug  ][ExDbgMsg          ] [stkendca2020.fnal.gov:23365] Moving MsgHandler: 0x17fab260 (message: kXR_read (handle: 0x00000000, offset: 2949187202, size: 809690) ) from out-queu to in-queue.
[2025-08-06 10:51:38.791076 +0100][Debug  ][ExDbgMsg          ] [msg: 0x17fac0c0] Assigned MsgHandler: 0x17fab260.
[2025-08-06 10:51:38.791124 +0100][Debug  ][ExDbgMsg          ] [handler: 0x17fab260] Removed MsgHandler: 0x17fab260 from the in-queue.
[2025-08-06 10:51:38.792519 +0100][Debug  ][ExDbgMsg          ] [stkendca2020.fnal.gov:23365] Calling MsgHandler: 0x17fab260 (message: kXR_read (handle: 0x00000000, offset: 2949187202, size: 809690) ) with status: [SUCCESS] .
[2025-08-06 10:51:38.792553 +0100][Debug  ][ExDbgMsg          ] [stkendca2020.fnal.gov:23365] Destroying MsgHandler: 0x17fab260.
[2025-08-06 10:51:38.840622 +0100][Debug  ][File              ] [0x99e9230@root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/02/45/prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250728T130352Z_gen_003703_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=157d4982-5426-4b11-8aff-49577f182c6e] Sending a read command for handle 0x0 to stkendca2020.fnal.gov:23365
[2025-08-06 10:51:38.840691 +0100][Debug  ][ExDbgMsg          ] [stkendca2020.fnal.gov:23365] MsgHandler created: 0x18342410 (message: kXR_read (handle: 0x00000000, offset: 2044292277, size: 20155290) ).
[2025-08-06 10:51:38.840854 +0100][Debug  ][ExDbgMsg          ] [stkendca2020.fnal.gov:23365] Moving MsgHandler: 0x18342410 (message: kXR_read (handle: 0x00000000, offset: 2044292277, size: 20155290) ) from out-queu to in-queue.
[2025-08-06 10:51:38.957341 +0100][Debug  ][ExDbgMsg          ] [msg: 0x183431a0] Assigned MsgHandler: 0x18342410.
[2025-08-06 10:51:39.079979 +0100][Debug  ][ExDbgMsg          ] [msg: 0x18343210] Assigned MsgHandler: 0x18342410.
[2025-08-06 10:51:39.297989 +0100][Debug  ][ExDbgMsg          ] [msg: 0x18343280] Assigned MsgHandler: 0x18342410.
[2025-08-06 10:51:39.298036 +0100][Debug  ][ExDbgMsg          ] [handler: 0x18342410] Removed MsgHandler: 0x18342410 from the in-queue.
[2025-08-06 10:51:39.304542 +0100][Debug  ][ExDbgMsg          ] [stkendca2020.fnal.gov:23365] Calling MsgHandler: 0x18342410 (message: kXR_read (handle: 0x00000000, offset: 2044292277, size: 20155290) ) with status: [SUCCESS] .
[2025-08-06 10:51:39.304600 +0100][Debug  ][ExDbgMsg          ] [stkendca2020.fnal.gov:23365] Destroying MsgHandler: 0x18342410.
R__unzip: error -3 in inflate (zlib)
[2025-08-06 10:51:39.454878 +0100][Debug  ][File              ] [0x99e9230@root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/02/45/prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250728T130352Z_gen_003703_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=157d4982-5426-4b11-8aff-49577f182c6e] Sending a close command for handle 0x0 to stkendca2020.fnal.gov:23365
[2025-08-06 10:51:39.454943 +0100][Debug  ][ExDbgMsg          ] [stkendca2020.fnal.gov:23365] MsgHandler created: 0xf8d3430 (message: kXR_close (handle: 0x00000000) ).
[2025-08-06 10:51:39.455118 +0100][Debug  ][ExDbgMsg          ] [stkendca2020.fnal.gov:23365] Moving MsgHandler: 0xf8d3430 (message: kXR_close (handle: 0x00000000) ) from out-queu to in-queue.
[2025-08-06 10:51:39.568028 +0100][Debug  ][ExDbgMsg          ] [msg: 0xe587920] Assigned MsgHandler: 0xf8d3430.
[2025-08-06 10:51:39.568078 +0100][Debug  ][ExDbgMsg          ] [handler: 0xf8d3430] Removed MsgHandler: 0xf8d3430 from the in-queue.
[2025-08-06 10:51:39.568132 +0100][Debug  ][ExDbgMsg          ] [stkendca2020.fnal.gov:23365] Calling MsgHandler: 0xf8d3430 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] .
[2025-08-06 10:51:39.568273 +0100][Debug  ][File              ] [0x99e9230@root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/02/45/prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250728T130352Z_gen_003703_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=157d4982-5426-4b11-8aff-49577f182c6e] Close returned from stkendca2020.fnal.gov:23365 with: [SUCCESS] 
[2025-08-06 10:51:39.568330 +0100][Debug  ][ExDbgMsg          ] [stkendca2020.fnal.gov:23365] Destroying MsgHandler: 0xf8d3430.
06-Aug-2025 10:51:39 BST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/02/45/prodbackground_radiological_decay0_dunevd10kt_1x8x14_20250728T130352Z_gen_003703_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                            0.00434177      1337.93       2052.74       1961.03       946.794         3     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00276677     0.0402706     0.113703     0.00434177     0.0519287        3     
reco:pandora:StandardPandora                            1799.52       1841.1        1882.67       1841.1        41.5735         2     
reco:pandoraTrack:LArPandoraTrackCreation             0.000195286     0.02086      0.0415247      0.02086      0.0206647        2     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00021985    0.00142997    0.00264009    0.00142997    0.00121012        2     
reco:pandoracalo:Calorimetry                          0.000734073    0.0021188    0.00350353     0.0021188    0.00138473        2     
reco:pandorapid:Chi2ParticleID                        4.9062e-05    0.000194423   0.000339785   0.000194423   0.000145362       2     
reco:linecluster:LineCluster                            1.01579       1.27392       1.53206       1.27392      0.258136         2     
reco:trajcluster:TrajCluster                            0.23102      0.241875      0.252731      0.241875      0.0108555        2     
reco:pmtracktc:PMAlgTrackMaker                          4.10689       4.12659       4.14629       4.12659      0.0197007        2     
reco:emtrkmichelid:EmTrackMichelId                      154.804       158.627       162.45        158.627       3.82278         2     
reco:solarflash:SolarOpFlash                            1.13057       1.24852       1.36647       1.24852      0.117951         2     
[art]:TriggerResults:TriggerResultInserter            2.1606e-05    3.02815e-05   3.8957e-05    3.02815e-05   8.6755e-06        2     
end_path:out1:RootOutput                               5.073e-06      9.8e-06     1.4527e-05      9.8e-06      4.727e-06        2     
end_path:out1:RootOutput(write)                        0.174952      0.187761       0.20057      0.187761      0.0128088        2     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3911.97 MB
  Peak resident set size usage (VmHWM): 2439.54 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 06-Aug-2025 10:51:39 BST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- FileReadError BEGIN
      ---- FatalRootError BEGIN
        Fatal Root Error: TBasket::ReadBasketBuffers
        fNbytes = 20155290, fKeylen = 119, fObjlen = 42177718, noutot = 16777215, nout=0, nin=8038200, nbuf=16777215
        ROOT severity: 3000
      ---- FatalRootError END
      
      The above exception was thrown while processing module StandardPandora/pandora run: 87 subRun: 0 event: 37023
    ---- FileReadError END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 2 entries while art::TriggerResults_TriggerResults__RadBkgGen. has 10 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
[2025-08-06 10:51:40.051816 +0100][Debug  ][JobMgr            ] Stopping the job manager...
[2025-08-06 10:51:40.052286 +0100][Debug  ][JobMgr            ] Job manager stopped
[2025-08-06 10:51:40.052361 +0100][Debug  ][TaskMgr           ] Stopping the task manager...
[2025-08-06 10:51:40.052484 +0100][Debug  ][TaskMgr           ] Task manager stopped
[2025-08-06 10:51:40.052489 +0100][Debug  ][Poller            ] Stopping the poller...
[2025-08-06 10:51:40.052685 +0100][Debug  ][AsyncSock         ] [stkendca2020.fnal.gov:23365.0] Closing the socket
[2025-08-06 10:51:40.052704 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.37]:49772><--><[::ffff:131.225.69.144]:23365> Removing socket from the poller
[2025-08-06 10:51:40.052744 +0100][Debug  ][PostMaster        ] [stkendca2020.fnal.gov:23365] Destroying stream
[2025-08-06 10:51:40.052749 +0100][Debug  ][AsyncSock         ] [stkendca2020.fnal.gov:23365.0] Closing the socket
Art has completed and will exit with status 1.
justIN time: 2025-09-19 01:00:16 UTC       justIN version: 01.05.00