Jobsub ID 12679.6@dunegpschedd02.fnal.gov

Jobsub ID	12679.6@dunegpschedd02.fnal.gov
Workflow ID	111
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-30 06:41:19
Site	UK_Edinburgh
Entry	DUNE_UK_SGridECDF_ce1
Last heartbeat	2025-07-30 06:46:11
From worker node	Hostname	node2b05.ecdf.ed.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-07-30 06:42:17
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T052148Z_gen_000318_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-07-30 06:46:11
Saved logs	justin-logs:12679.6-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

b25fb0 (message: kXR_read (handle: 0x00000000, offset: 2972983369, size: 137) ) with status: [SUCCESS] .
[2025-07-30 07:45:39.162592 +0100][Debug  ][ExDbgMsg          ] [pubstor2306.fnal.gov:23489] Destroying MsgHandler: 0x28b25fb0.
[2025-07-30 07:45:39.162658 +0100][Debug  ][File              ] [0xe14ad90@root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/92/d6/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T052148Z_gen_000318_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=9b775832-723d-4cb7-ba82-0caedcef6dca] Sending a read command for handle 0x0 to pubstor2306.fnal.gov:23489
[2025-07-30 07:45:39.162683 +0100][Debug  ][ExDbgMsg          ] [pubstor2306.fnal.gov:23489] MsgHandler created: 0x28b25fb0 (message: kXR_read (handle: 0x00000000, offset: 2972983506, size: 151) ).
[2025-07-30 07:45:39.162738 +0100][Debug  ][ExDbgMsg          ] [pubstor2306.fnal.gov:23489] Moving MsgHandler: 0x28b25fb0 (message: kXR_read (handle: 0x00000000, offset: 2972983506, size: 151) ) from out-queu to in-queue.
[2025-07-30 07:45:39.264908 +0100][Debug  ][ExDbgMsg          ] [msg: 0x28b65170] Assigned MsgHandler: 0x28b25fb0.
[2025-07-30 07:45:39.264957 +0100][Debug  ][ExDbgMsg          ] [handler: 0x28b25fb0] Removed MsgHandler: 0x28b25fb0 from the in-queue.
[2025-07-30 07:45:39.264991 +0100][Debug  ][ExDbgMsg          ] [pubstor2306.fnal.gov:23489] Calling MsgHandler: 0x28b25fb0 (message: kXR_read (handle: 0x00000000, offset: 2972983506, size: 151) ) with status: [SUCCESS] .
[2025-07-30 07:45:39.265009 +0100][Debug  ][ExDbgMsg          ] [pubstor2306.fnal.gov:23489] Destroying MsgHandler: 0x28b25fb0.
[2025-07-30 07:45:39.265061 +0100][Debug  ][File              ] [0xe14ad90@root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/92/d6/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T052148Z_gen_000318_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=9b775832-723d-4cb7-ba82-0caedcef6dca] Sending a read command for handle 0x0 to pubstor2306.fnal.gov:23489
[2025-07-30 07:45:39.265084 +0100][Debug  ][ExDbgMsg          ] [pubstor2306.fnal.gov:23489] MsgHandler created: 0x28b25fb0 (message: kXR_read (handle: 0x00000000, offset: 2972983657, size: 113) ).
[2025-07-30 07:45:39.265123 +0100][Debug  ][ExDbgMsg          ] [pubstor2306.fnal.gov:23489] Moving MsgHandler: 0x28b25fb0 (message: kXR_read (handle: 0x00000000, offset: 2972983657, size: 113) ) from out-queu to in-queue.
[2025-07-30 07:45:39.367418 +0100][Debug  ][ExDbgMsg          ] [msg: 0x28b65170] Assigned MsgHandler: 0x28b25fb0.
[2025-07-30 07:45:39.367464 +0100][Debug  ][ExDbgMsg          ] [handler: 0x28b25fb0] Removed MsgHandler: 0x28b25fb0 from the in-queue.
[2025-07-30 07:45:39.367501 +0100][Debug  ][ExDbgMsg          ] [pubstor2306.fnal.gov:23489] Calling MsgHandler: 0x28b25fb0 (message: kXR_read (handle: 0x00000000, offset: 2972983657, size: 113) ) with status: [SUCCESS] .
[2025-07-30 07:45:39.367519 +0100][Debug  ][ExDbgMsg          ] [pubstor2306.fnal.gov:23489] Destroying MsgHandler: 0x28b25fb0.
[2025-07-30 07:45:39.367573 +0100][Debug  ][File              ] [0xe14ad90@root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/92/d6/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T052148Z_gen_000318_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=9b775832-723d-4cb7-ba82-0caedcef6dca] Sending a read command for handle 0x0 to pubstor2306.fnal.gov:23489
[2025-07-30 07:45:39.367596 +0100][Debug  ][ExDbgMsg          ] [pubstor2306.fnal.gov:23489] MsgHandler created: 0x28b25fb0 (message: kXR_read (handle: 0x00000000, offset: 2972983770, size: 127) ).
[2025-07-30 07:45:39.367642 +0100][Debug  ][ExDbgMsg          ] [pubstor2306.fnal.gov:23489] Moving MsgHandler: 0x28b25fb0 (message: kXR_read (handle: 0x00000000, offset: 2972983770, size: 127) ) from out-queu to in-queue.
[2025-07-30 07:45:39.469959 +0100][Debug  ][ExDbgMsg          ] [msg: 0x28b65170] Assigned MsgHandler: 0x28b25fb0.
[2025-07-30 07:45:39.470003 +0100][Debug  ][ExDbgMsg          ] [handler: 0x28b25fb0] Removed MsgHandler: 0x28b25fb0 from the in-queue.
[2025-07-30 07:45:39.470051 +0100][Debug  ][ExDbgMsg          ] [pubstor2306.fnal.gov:23489] Calling MsgHandler: 0x28b25fb0 (message: kXR_read (handle: 0x00000000, offset: 2972983770, size: 127) ) with status: [SUCCESS] .
[2025-07-30 07:45:39.470069 +0100][Debug  ][ExDbgMsg          ] [pubstor2306.fnal.gov:23489] Destroying MsgHandler: 0x28b25fb0.
[2025-07-30 07:45:39.477361 +0100][Debug  ][File              ] [0xe14ad90@root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/92/d6/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T052148Z_gen_000318_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=9b775832-723d-4cb7-ba82-0caedcef6dca] Sending a close command for handle 0x0 to pubstor2306.fnal.gov:23489
[2025-07-30 07:45:39.477421 +0100][Debug  ][ExDbgMsg          ] [pubstor2306.fnal.gov:23489] MsgHandler created: 0xf219260 (message: kXR_close (handle: 0x00000000) ).
[2025-07-30 07:45:39.477486 +0100][Debug  ][ExDbgMsg          ] [pubstor2306.fnal.gov:23489] Moving MsgHandler: 0xf219260 (message: kXR_close (handle: 0x00000000) ) from out-queu to in-queue.
[2025-07-30 07:45:39.580789 +0100][Debug  ][ExDbgMsg          ] [msg: 0x28b27a90] Assigned MsgHandler: 0xf219260.
[2025-07-30 07:45:39.580806 +0100][Debug  ][ExDbgMsg          ] [handler: 0xf219260] Removed MsgHandler: 0xf219260 from the in-queue.
[2025-07-30 07:45:39.580829 +0100][Debug  ][ExDbgMsg          ] [pubstor2306.fnal.gov:23489] Calling MsgHandler: 0xf219260 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] .
[2025-07-30 07:45:39.580918 +0100][Debug  ][File              ] [0xe14ad90@root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/92/d6/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T052148Z_gen_000318_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=9b775832-723d-4cb7-ba82-0caedcef6dca] Close returned from pubstor2306.fnal.gov:23489 with: [SUCCESS] 
[2025-07-30 07:45:39.580974 +0100][Debug  ][ExDbgMsg          ] [pubstor2306.fnal.gov:23489] Destroying MsgHandler: 0xf219260.
30-Jul-2025 07:45:39 BST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/92/d6/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250202T052148Z_gen_000318_supernova_g4stage1_g4stage2_detsim_reco.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
[ No processed events ]
====================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3591.07 MB
  Peak resident set size usage (VmHWM): 2104.82 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 30-Jul-2025 07:45:43 BST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- ProductNotFound BEGIN
      A request to resolve an Ptr to a product containing items of type: simb::MCTruth with ProductID 3658073087
      cannot be satisfied because the product cannot be found.
      The productGetter was not set -- are you trying to dereference a Ptr during mixing?
      The above exception was thrown while processing module StandardPandora/pandora run: 5021 subRun: 0 event: 3171
    ---- ProductNotFound END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 0 entries while sim::OpDetDivRecs_sipmAr10ppmExt__detsim. has 10 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
[2025-07-30 07:45:43.167819 +0100][Debug  ][JobMgr            ] Stopping the job manager...
[2025-07-30 07:45:43.168124 +0100][Debug  ][JobMgr            ] Job manager stopped
[2025-07-30 07:45:43.168180 +0100][Debug  ][TaskMgr           ] Stopping the task manager...
[2025-07-30 07:45:43.168322 +0100][Debug  ][TaskMgr           ] Task manager stopped
[2025-07-30 07:45:43.168330 +0100][Debug  ][Poller            ] Stopping the poller...
[2025-07-30 07:45:43.168759 +0100][Debug  ][AsyncSock         ] [fndca1.fnal.gov:1094.0] Closing the socket
[2025-07-30 07:45:43.168844 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.38]:48276><--><[::ffff:131.225.69.121]:1094> Removing socket from the poller
[2025-07-30 07:45:43.169138 +0100][Debug  ][PostMaster        ] [fndca1.fnal.gov:1094] Destroying stream
[2025-07-30 07:45:43.169233 +0100][Debug  ][AsyncSock         ] [fndca1.fnal.gov:1094.0] Closing the socket
[2025-07-30 07:45:43.169319 +0100][Debug  ][AsyncSock         ] [pubstor2306.fnal.gov:23489.0] Closing the socket
[2025-07-30 07:45:43.169326 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.38]:40806><--><[::ffff:131.225.69.92]:23489> Removing socket from the poller
[2025-07-30 07:45:43.169342 +0100][Debug  ][PostMaster        ] [pubstor2306.fnal.gov:23489] Destroying stream
[2025-07-30 07:45:43.169347 +0100][Debug  ][AsyncSock         ] [pubstor2306.fnal.gov:23489.0] Closing the socket
Art has completed and will exit with status 1.