justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 11495.167@dunegpschedd02.fnal.gov

Jobsub ID11495.167@dunegpschedd02.fnal.gov
Workflow ID77
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod.mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-07-26 15:55:43
SiteUK_Edinburgh
EntryDUNE_UK_SGridECDF_ce1_multicore
Last heartbeat2025-07-26 18:04:14
From worker nodeHostnamenode2b01.ecdf.ed.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-07-26 18:02:26
Input filesfardet-vd:prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241024T113206Z_gen_002395_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-07-26 18:04:14
Saved logsjustin-logs:11495.167-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

T113206Z_gen_002395_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=e442e960-15d1-4a6a-aec5-0296bd5e73e8] Sending a read command for handle 0x0 to dtn02.tier2.hep.manchester.ac.uk:1095
[2025-07-26 19:03:33.788346 +0100][Debug  ][ExDbgMsg          ] [dtn02.tier2.hep.manchester.ac.uk:1095] MsgHandler created: 0x2888f990 (message: kXR_read (handle: 0x00000000, offset: 2962702629, size: 151) ).
[2025-07-26 19:03:33.788372 +0100][Debug  ][ExDbgMsg          ] [dtn02.tier2.hep.manchester.ac.uk:1095] Moving MsgHandler: 0x2888f990 (message: kXR_read (handle: 0x00000000, offset: 2962702629, size: 151) ) from out-queu to in-queue.
[2025-07-26 19:03:33.795615 +0100][Debug  ][ExDbgMsg          ] [msg: 0x2889c5c0] Assigned MsgHandler: 0x2888f990.
[2025-07-26 19:03:33.795622 +0100][Debug  ][ExDbgMsg          ] [handler: 0x2888f990] Removed MsgHandler: 0x2888f990 from the in-queue.
[2025-07-26 19:03:33.795651 +0100][Debug  ][ExDbgMsg          ] [dtn02.tier2.hep.manchester.ac.uk:1095] Calling MsgHandler: 0x2888f990 (message: kXR_read (handle: 0x00000000, offset: 2962702629, size: 151) ) with status: [SUCCESS] .
[2025-07-26 19:03:33.795659 +0100][Debug  ][ExDbgMsg          ] [dtn02.tier2.hep.manchester.ac.uk:1095] Destroying MsgHandler: 0x2888f990.
[2025-07-26 19:03:33.795676 +0100][Debug  ][File              ] [0xf458730@root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/fardet-vd/47/ed/prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241024T113206Z_gen_002395_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=e442e960-15d1-4a6a-aec5-0296bd5e73e8] Sending a read command for handle 0x0 to dtn02.tier2.hep.manchester.ac.uk:1095
[2025-07-26 19:03:33.795687 +0100][Debug  ][ExDbgMsg          ] [dtn02.tier2.hep.manchester.ac.uk:1095] MsgHandler created: 0x2888f990 (message: kXR_read (handle: 0x00000000, offset: 2962702780, size: 113) ).
[2025-07-26 19:03:33.795713 +0100][Debug  ][ExDbgMsg          ] [dtn02.tier2.hep.manchester.ac.uk:1095] Moving MsgHandler: 0x2888f990 (message: kXR_read (handle: 0x00000000, offset: 2962702780, size: 113) ) from out-queu to in-queue.
[2025-07-26 19:03:33.802048 +0100][Debug  ][ExDbgMsg          ] [msg: 0x2889c5d0] Assigned MsgHandler: 0x2888f990.
[2025-07-26 19:03:33.802055 +0100][Debug  ][ExDbgMsg          ] [handler: 0x2888f990] Removed MsgHandler: 0x2888f990 from the in-queue.
[2025-07-26 19:03:33.802077 +0100][Debug  ][ExDbgMsg          ] [dtn02.tier2.hep.manchester.ac.uk:1095] Calling MsgHandler: 0x2888f990 (message: kXR_read (handle: 0x00000000, offset: 2962702780, size: 113) ) with status: [SUCCESS] .
[2025-07-26 19:03:33.802085 +0100][Debug  ][ExDbgMsg          ] [dtn02.tier2.hep.manchester.ac.uk:1095] Destroying MsgHandler: 0x2888f990.
[2025-07-26 19:03:33.802110 +0100][Debug  ][File              ] [0xf458730@root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/fardet-vd/47/ed/prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241024T113206Z_gen_002395_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=e442e960-15d1-4a6a-aec5-0296bd5e73e8] Sending a read command for handle 0x0 to dtn02.tier2.hep.manchester.ac.uk:1095
[2025-07-26 19:03:33.802121 +0100][Debug  ][ExDbgMsg          ] [dtn02.tier2.hep.manchester.ac.uk:1095] MsgHandler created: 0x2888f990 (message: kXR_read (handle: 0x00000000, offset: 2962702893, size: 127) ).
[2025-07-26 19:03:33.802148 +0100][Debug  ][ExDbgMsg          ] [dtn02.tier2.hep.manchester.ac.uk:1095] Moving MsgHandler: 0x2888f990 (message: kXR_read (handle: 0x00000000, offset: 2962702893, size: 127) ) from out-queu to in-queue.
[2025-07-26 19:03:33.808477 +0100][Debug  ][ExDbgMsg          ] [msg: 0x2889c760] Assigned MsgHandler: 0x2888f990.
[2025-07-26 19:03:33.808485 +0100][Debug  ][ExDbgMsg          ] [handler: 0x2888f990] Removed MsgHandler: 0x2888f990 from the in-queue.
[2025-07-26 19:03:33.808506 +0100][Debug  ][ExDbgMsg          ] [dtn02.tier2.hep.manchester.ac.uk:1095] Calling MsgHandler: 0x2888f990 (message: kXR_read (handle: 0x00000000, offset: 2962702893, size: 127) ) with status: [SUCCESS] .
[2025-07-26 19:03:33.808515 +0100][Debug  ][ExDbgMsg          ] [dtn02.tier2.hep.manchester.ac.uk:1095] Destroying MsgHandler: 0x2888f990.
[2025-07-26 19:03:33.817816 +0100][Debug  ][File              ] [0xf458730@root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/fardet-vd/47/ed/prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241024T113206Z_gen_002395_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=e442e960-15d1-4a6a-aec5-0296bd5e73e8] Sending a close command for handle 0x0 to dtn02.tier2.hep.manchester.ac.uk:1095
[2025-07-26 19:03:33.817897 +0100][Debug  ][ExDbgMsg          ] [dtn02.tier2.hep.manchester.ac.uk:1095] MsgHandler created: 0xb475500 (message: kXR_close (handle: 0x00000000) ).
[2025-07-26 19:03:33.818018 +0100][Debug  ][ExDbgMsg          ] [dtn02.tier2.hep.manchester.ac.uk:1095] Moving MsgHandler: 0xb475500 (message: kXR_close (handle: 0x00000000) ) from out-queu to in-queue.
[2025-07-26 19:03:33.824352 +0100][Debug  ][ExDbgMsg          ] [msg: 0xf2f5670] Assigned MsgHandler: 0xb475500.
[2025-07-26 19:03:33.824413 +0100][Debug  ][ExDbgMsg          ] [handler: 0xb475500] Removed MsgHandler: 0xb475500 from the in-queue.
[2025-07-26 19:03:33.824461 +0100][Debug  ][ExDbgMsg          ] [dtn02.tier2.hep.manchester.ac.uk:1095] Calling MsgHandler: 0xb475500 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] .
[2025-07-26 19:03:33.824597 +0100][Debug  ][File              ] [0xf458730@root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/fardet-vd/47/ed/prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241024T113206Z_gen_002395_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=e442e960-15d1-4a6a-aec5-0296bd5e73e8] Close returned from dtn02.tier2.hep.manchester.ac.uk:1095 with: [SUCCESS] 
[2025-07-26 19:03:33.824682 +0100][Debug  ][ExDbgMsg          ] [dtn02.tier2.hep.manchester.ac.uk:1095] Destroying MsgHandler: 0xb475500.
26-Jul-2025 19:03:33 BST  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/fardet-vd/47/ed/prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241024T113206Z_gen_002395_supernova_g4stage1_g4stage2_detsim_reco.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
[ No processed events ]
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3416.03 MB
  Peak resident set size usage (VmHWM): 1937.25 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 26-Jul-2025 19:03:52 BST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- LogicError BEGIN
      Invalid FindOneP
      ---- ProductNotFound BEGIN
        Found zero products matching all selection criteria
          C++ type: art::Assns<simb::MCParticle,simb::MCTruth,void>
          Module label: 'largeant'
          Product instance name: ''
          Process name: (empty)
      ---- ProductNotFound END
      Attempt to use a FindOneP where the underlying art::Assns product was not found.The above exception was thrown while processing module StandardPandora/pandora run: 3808 subRun: 0 event: 23941
    ---- LogicError END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 0 entries while sim::OpDetDivRecs_sipmAr10ppmExt__detsim. has 10 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
[2025-07-26 19:03:52.079200 +0100][Debug  ][JobMgr            ] Stopping the job manager...
[2025-07-26 19:03:52.079567 +0100][Debug  ][JobMgr            ] Job manager stopped
[2025-07-26 19:03:52.079630 +0100][Debug  ][TaskMgr           ] Stopping the task manager...
[2025-07-26 19:03:52.079780 +0100][Debug  ][TaskMgr           ] Task manager stopped
[2025-07-26 19:03:52.079787 +0100][Debug  ][Poller            ] Stopping the poller...
[2025-07-26 19:03:52.080265 +0100][Debug  ][AsyncSock         ] [dtn02.tier2.hep.manchester.ac.uk:1095.0] Closing the socket
[2025-07-26 19:03:52.080376 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.34]:47320><--><[::ffff:195.194.107.233]:1095> Removing socket from the poller
[2025-07-26 19:03:52.080764 +0100][Debug  ][PostMaster        ] [dtn02.tier2.hep.manchester.ac.uk:1095] Destroying stream
[2025-07-26 19:03:52.080886 +0100][Debug  ][AsyncSock         ] [dtn02.tier2.hep.manchester.ac.uk:1095.0] Closing the socket
[2025-07-26 19:03:52.081005 +0100][Debug  ][AsyncSock         ] [meitner.tier2.hep.manchester.ac.uk:1094.0] Closing the socket
[2025-07-26 19:03:52.081013 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.34]:39854><--><[::ffff:195.194.108.197]:1094> Removing socket from the poller
[2025-07-26 19:03:52.081113 +0100][Debug  ][PostMaster        ] [meitner.tier2.hep.manchester.ac.uk:1094] Destroying stream
[2025-07-26 19:03:52.081145 +0100][Debug  ][AsyncSock         ] [meitner.tier2.hep.manchester.ac.uk:1094.0] Closing the socket
Art has completed and will exit with status 1.
justIN time: 2025-08-04 14:18:21 UTC       justIN version: 01.04.00