justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 11495.6@dunegpschedd02.fnal.gov

Jobsub ID11495.6@dunegpschedd02.fnal.gov
Workflow ID77
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod.mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-07-26 15:55:43
SiteUK_Edinburgh
EntryDUNE_UK_SGridECDF_ce1_multicore
Last heartbeat2025-07-26 17:30:18
From worker nodeHostnamenode2b01.ecdf.ed.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Started2025-07-26 17:28:49
Input filesfardet-vd:prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241024T113206Z_gen_002395_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-07-26 17:30:18
Saved logsjustin-logs:11495.6-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

024T113206Z_gen_002395_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=a15c7392-5e49-4a36-a5b9-38306c3f9a19] Sending a read command for handle 0x0 to dtn04.tier2.hep.manchester.ac.uk:1095
[2025-07-26 18:29:50.120836 +0100][Debug  ][ExDbgMsg          ] [dtn04.tier2.hep.manchester.ac.uk:1095] MsgHandler created: 0x281ecb40 (message: kXR_read (handle: 0x00000000, offset: 2962702629, size: 151) ).
[2025-07-26 18:29:50.120898 +0100][Debug  ][ExDbgMsg          ] [dtn04.tier2.hep.manchester.ac.uk:1095] Moving MsgHandler: 0x281ecb40 (message: kXR_read (handle: 0x00000000, offset: 2962702629, size: 151) ) from out-queu to in-queue.
[2025-07-26 18:29:50.127093 +0100][Debug  ][ExDbgMsg          ] [msg: 0x282306a0] Assigned MsgHandler: 0x281ecb40.
[2025-07-26 18:29:50.127123 +0100][Debug  ][ExDbgMsg          ] [handler: 0x281ecb40] Removed MsgHandler: 0x281ecb40 from the in-queue.
[2025-07-26 18:29:50.127179 +0100][Debug  ][ExDbgMsg          ] [dtn04.tier2.hep.manchester.ac.uk:1095] Calling MsgHandler: 0x281ecb40 (message: kXR_read (handle: 0x00000000, offset: 2962702629, size: 151) ) with status: [SUCCESS] .
[2025-07-26 18:29:50.127201 +0100][Debug  ][ExDbgMsg          ] [dtn04.tier2.hep.manchester.ac.uk:1095] Destroying MsgHandler: 0x281ecb40.
[2025-07-26 18:29:50.127240 +0100][Debug  ][File              ] [0xed55730@root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/fardet-vd/47/ed/prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241024T113206Z_gen_002395_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=a15c7392-5e49-4a36-a5b9-38306c3f9a19] Sending a read command for handle 0x0 to dtn04.tier2.hep.manchester.ac.uk:1095
[2025-07-26 18:29:50.127265 +0100][Debug  ][ExDbgMsg          ] [dtn04.tier2.hep.manchester.ac.uk:1095] MsgHandler created: 0x281ecb40 (message: kXR_read (handle: 0x00000000, offset: 2962702780, size: 113) ).
[2025-07-26 18:29:50.127321 +0100][Debug  ][ExDbgMsg          ] [dtn04.tier2.hep.manchester.ac.uk:1095] Moving MsgHandler: 0x281ecb40 (message: kXR_read (handle: 0x00000000, offset: 2962702780, size: 113) ) from out-queu to in-queue.
[2025-07-26 18:29:50.133502 +0100][Debug  ][ExDbgMsg          ] [msg: 0x28230730] Assigned MsgHandler: 0x281ecb40.
[2025-07-26 18:29:50.133530 +0100][Debug  ][ExDbgMsg          ] [handler: 0x281ecb40] Removed MsgHandler: 0x281ecb40 from the in-queue.
[2025-07-26 18:29:50.133645 +0100][Debug  ][ExDbgMsg          ] [dtn04.tier2.hep.manchester.ac.uk:1095] Calling MsgHandler: 0x281ecb40 (message: kXR_read (handle: 0x00000000, offset: 2962702780, size: 113) ) with status: [SUCCESS] .
[2025-07-26 18:29:50.133667 +0100][Debug  ][ExDbgMsg          ] [dtn04.tier2.hep.manchester.ac.uk:1095] Destroying MsgHandler: 0x281ecb40.
[2025-07-26 18:29:50.133708 +0100][Debug  ][File              ] [0xed55730@root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/fardet-vd/47/ed/prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241024T113206Z_gen_002395_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=a15c7392-5e49-4a36-a5b9-38306c3f9a19] Sending a read command for handle 0x0 to dtn04.tier2.hep.manchester.ac.uk:1095
[2025-07-26 18:29:50.133729 +0100][Debug  ][ExDbgMsg          ] [dtn04.tier2.hep.manchester.ac.uk:1095] MsgHandler created: 0x281ecb40 (message: kXR_read (handle: 0x00000000, offset: 2962702893, size: 127) ).
[2025-07-26 18:29:50.133783 +0100][Debug  ][ExDbgMsg          ] [dtn04.tier2.hep.manchester.ac.uk:1095] Moving MsgHandler: 0x281ecb40 (message: kXR_read (handle: 0x00000000, offset: 2962702893, size: 127) ) from out-queu to in-queue.
[2025-07-26 18:29:50.139983 +0100][Debug  ][ExDbgMsg          ] [msg: 0x28230470] Assigned MsgHandler: 0x281ecb40.
[2025-07-26 18:29:50.140011 +0100][Debug  ][ExDbgMsg          ] [handler: 0x281ecb40] Removed MsgHandler: 0x281ecb40 from the in-queue.
[2025-07-26 18:29:50.140062 +0100][Debug  ][ExDbgMsg          ] [dtn04.tier2.hep.manchester.ac.uk:1095] Calling MsgHandler: 0x281ecb40 (message: kXR_read (handle: 0x00000000, offset: 2962702893, size: 127) ) with status: [SUCCESS] .
[2025-07-26 18:29:50.140082 +0100][Debug  ][ExDbgMsg          ] [dtn04.tier2.hep.manchester.ac.uk:1095] Destroying MsgHandler: 0x281ecb40.
[2025-07-26 18:29:50.149125 +0100][Debug  ][File              ] [0xed55730@root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/fardet-vd/47/ed/prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241024T113206Z_gen_002395_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=a15c7392-5e49-4a36-a5b9-38306c3f9a19] Sending a close command for handle 0x0 to dtn04.tier2.hep.manchester.ac.uk:1095
[2025-07-26 18:29:50.149407 +0100][Debug  ][ExDbgMsg          ] [dtn04.tier2.hep.manchester.ac.uk:1095] MsgHandler created: 0xb670e10 (message: kXR_close (handle: 0x00000000) ).
[2025-07-26 18:29:50.149515 +0100][Debug  ][ExDbgMsg          ] [dtn04.tier2.hep.manchester.ac.uk:1095] Moving MsgHandler: 0xb670e10 (message: kXR_close (handle: 0x00000000) ) from out-queu to in-queue.
[2025-07-26 18:29:50.155717 +0100][Debug  ][ExDbgMsg          ] [msg: 0xf474c10] Assigned MsgHandler: 0xb670e10.
[2025-07-26 18:29:50.155744 +0100][Debug  ][ExDbgMsg          ] [handler: 0xb670e10] Removed MsgHandler: 0xb670e10 from the in-queue.
[2025-07-26 18:29:50.155808 +0100][Debug  ][ExDbgMsg          ] [dtn04.tier2.hep.manchester.ac.uk:1095] Calling MsgHandler: 0xb670e10 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] .
[2025-07-26 18:29:50.155964 +0100][Debug  ][File              ] [0xed55730@root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/fardet-vd/47/ed/prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241024T113206Z_gen_002395_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=a15c7392-5e49-4a36-a5b9-38306c3f9a19] Close returned from dtn04.tier2.hep.manchester.ac.uk:1095 with: [SUCCESS] 
[2025-07-26 18:29:50.156045 +0100][Debug  ][ExDbgMsg          ] [dtn04.tier2.hep.manchester.ac.uk:1095] Destroying MsgHandler: 0xb670e10.
26-Jul-2025 18:29:50 BST  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/fardet-vd/47/ed/prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241024T113206Z_gen_002395_supernova_g4stage1_g4stage2_detsim_reco.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
[ No processed events ]
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3416.03 MB
  Peak resident set size usage (VmHWM): 1937.1 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 26-Jul-2025 18:29:52 BST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- LogicError BEGIN
      Invalid FindOneP
      ---- ProductNotFound BEGIN
        Found zero products matching all selection criteria
          C++ type: art::Assns<simb::MCParticle,simb::MCTruth,void>
          Module label: 'largeant'
          Product instance name: ''
          Process name: (empty)
      ---- ProductNotFound END
      Attempt to use a FindOneP where the underlying art::Assns product was not found.The above exception was thrown while processing module StandardPandora/pandora run: 3808 subRun: 0 event: 23941
    ---- LogicError END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 0 entries while sim::OpDetDivRecs_sipmAr10ppmExt__detsim. has 10 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
[2025-07-26 18:29:52.305822 +0100][Debug  ][JobMgr            ] Stopping the job manager...
[2025-07-26 18:29:52.306252 +0100][Debug  ][JobMgr            ] Job manager stopped
[2025-07-26 18:29:52.306321 +0100][Debug  ][TaskMgr           ] Stopping the task manager...
[2025-07-26 18:29:52.306469 +0100][Debug  ][TaskMgr           ] Task manager stopped
[2025-07-26 18:29:52.306476 +0100][Debug  ][Poller            ] Stopping the poller...
[2025-07-26 18:29:52.306962 +0100][Debug  ][AsyncSock         ] [dtn04.tier2.hep.manchester.ac.uk:1095.0] Closing the socket
[2025-07-26 18:29:52.307068 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.34]:52018><--><[::ffff:195.194.107.91]:1095> Removing socket from the poller
[2025-07-26 18:29:52.307419 +0100][Debug  ][PostMaster        ] [dtn04.tier2.hep.manchester.ac.uk:1095] Destroying stream
[2025-07-26 18:29:52.307543 +0100][Debug  ][AsyncSock         ] [dtn04.tier2.hep.manchester.ac.uk:1095.0] Closing the socket
[2025-07-26 18:29:52.307669 +0100][Debug  ][AsyncSock         ] [meitner.tier2.hep.manchester.ac.uk:1094.0] Closing the socket
[2025-07-26 18:29:52.307679 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.34]:46684><--><[::ffff:195.194.107.39]:1094> Removing socket from the poller
[2025-07-26 18:29:52.307776 +0100][Debug  ][PostMaster        ] [meitner.tier2.hep.manchester.ac.uk:1094] Destroying stream
[2025-07-26 18:29:52.307797 +0100][Debug  ][AsyncSock         ] [meitner.tier2.hep.manchester.ac.uk:1094.0] Closing the socket
Art has completed and will exit with status 1.
justIN time: 2025-08-04 16:20:02 UTC       justIN version: 01.04.00