Jobsub ID 11467.128@dunegpschedd02.fnal.gov

Jobsub ID	11467.128@dunegpschedd02.fnal.gov
Workflow ID	77
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-26 14:39:39
Site	UK_Edinburgh
Entry	DUNE_UK_SGridECDF_ce1_multicore
Last heartbeat	2025-07-26 19:00:44
From worker node	Hostname	node2b07.ecdf.ed.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	jobscript_error
Started	2025-07-26 16:01:58
Input files	fardet-vd:prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250215T044500Z_gen_003476_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	1
	Real time	0m (0s)
	CPU time	0m (0s = 0%)
	Max RSS bytes	0 (0 MiB)
Outputting started
Output files
Finished	2025-07-26 19:00:44
Saved logs	justin-logs:11467.128-dunegpschedd02.fnal.gov.logs.tgz
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

ge2_detsim_reco.root?xrdcl.requuid=d94f9acf-6c7d-49b9-b53f-0ffe286759d4] Sending a read command for handle 0x0 to dtn03.tier2.hep.manchester.ac.uk:1095
[2025-07-26 20:00:17.691151 +0100][Debug  ][ExDbgMsg          ] [dtn03.tier2.hep.manchester.ac.uk:1095] MsgHandler created: 0x45321bb0 (message: kXR_read (handle: 0x00000000, offset: 50635991, size: 7471) ).
[2025-07-26 20:00:17.691189 +0100][Debug  ][ExDbgMsg          ] [dtn03.tier2.hep.manchester.ac.uk:1095] Moving MsgHandler: 0x45321bb0 (message: kXR_read (handle: 0x00000000, offset: 50635991, size: 7471) ) from out-queu to in-queue.
[2025-07-26 20:00:17.697338 +0100][Debug  ][ExDbgMsg          ] [msg: 0x4715b760] Assigned MsgHandler: 0x45321bb0.
[2025-07-26 20:00:17.697346 +0100][Debug  ][ExDbgMsg          ] [handler: 0x45321bb0] Removed MsgHandler: 0x45321bb0 from the in-queue.
[2025-07-26 20:00:17.697437 +0100][Debug  ][ExDbgMsg          ] [dtn03.tier2.hep.manchester.ac.uk:1095] Calling MsgHandler: 0x45321bb0 (message: kXR_read (handle: 0x00000000, offset: 50635991, size: 7471) ) with status: [SUCCESS] .
[2025-07-26 20:00:17.697450 +0100][Debug  ][ExDbgMsg          ] [dtn03.tier2.hep.manchester.ac.uk:1095] Destroying MsgHandler: 0x45321bb0.
[2025-07-26 20:00:17.697809 +0100][Debug  ][File              ] [0xedc9350@root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/fardet-vd/f3/c1/prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250215T044500Z_gen_003476_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=d94f9acf-6c7d-49b9-b53f-0ffe286759d4] Sending a read command for handle 0x0 to dtn03.tier2.hep.manchester.ac.uk:1095
[2025-07-26 20:00:17.697843 +0100][Debug  ][ExDbgMsg          ] [dtn03.tier2.hep.manchester.ac.uk:1095] MsgHandler created: 0x472682a0 (message: kXR_read (handle: 0x00000000, offset: 50643462, size: 166793) ).
[2025-07-26 20:00:17.697887 +0100][Debug  ][ExDbgMsg          ] [dtn03.tier2.hep.manchester.ac.uk:1095] Moving MsgHandler: 0x472682a0 (message: kXR_read (handle: 0x00000000, offset: 50643462, size: 166793) ) from out-queu to in-queue.
[2025-07-26 20:00:17.703992 +0100][Debug  ][ExDbgMsg          ] [msg: 0x47268f20] Assigned MsgHandler: 0x472682a0.
[2025-07-26 20:00:17.704000 +0100][Debug  ][ExDbgMsg          ] [handler: 0x472682a0] Removed MsgHandler: 0x472682a0 from the in-queue.
[2025-07-26 20:00:17.704534 +0100][Debug  ][ExDbgMsg          ] [dtn03.tier2.hep.manchester.ac.uk:1095] Calling MsgHandler: 0x472682a0 (message: kXR_read (handle: 0x00000000, offset: 50643462, size: 166793) ) with status: [SUCCESS] .
[2025-07-26 20:00:17.704547 +0100][Debug  ][ExDbgMsg          ] [dtn03.tier2.hep.manchester.ac.uk:1095] Destroying MsgHandler: 0x472682a0.
[2025-07-26 20:00:17.743610 +0100][Debug  ][File              ] [0xedc9350@root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/fardet-vd/f3/c1/prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250215T044500Z_gen_003476_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=d94f9acf-6c7d-49b9-b53f-0ffe286759d4] Sending a close command for handle 0x0 to dtn03.tier2.hep.manchester.ac.uk:1095
[2025-07-26 20:00:17.743697 +0100][Debug  ][ExDbgMsg          ] [dtn03.tier2.hep.manchester.ac.uk:1095] MsgHandler created: 0xfdeab80 (message: kXR_close (handle: 0x00000000) ).
[2025-07-26 20:00:17.743818 +0100][Debug  ][ExDbgMsg          ] [dtn03.tier2.hep.manchester.ac.uk:1095] Moving MsgHandler: 0xfdeab80 (message: kXR_close (handle: 0x00000000) ) from out-queu to in-queue.
[2025-07-26 20:00:17.749987 +0100][Debug  ][ExDbgMsg          ] [msg: 0x2b000630] Assigned MsgHandler: 0xfdeab80.
[2025-07-26 20:00:17.749999 +0100][Debug  ][ExDbgMsg          ] [handler: 0xfdeab80] Removed MsgHandler: 0xfdeab80 from the in-queue.
[2025-07-26 20:00:17.750032 +0100][Debug  ][ExDbgMsg          ] [dtn03.tier2.hep.manchester.ac.uk:1095] Calling MsgHandler: 0xfdeab80 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] .
[2025-07-26 20:00:17.750149 +0100][Debug  ][File              ] [0xedc9350@root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/fardet-vd/f3/c1/prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250215T044500Z_gen_003476_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=d94f9acf-6c7d-49b9-b53f-0ffe286759d4] Close returned from dtn03.tier2.hep.manchester.ac.uk:1095 with: [SUCCESS] 
[2025-07-26 20:00:17.750207 +0100][Debug  ][ExDbgMsg          ] [dtn03.tier2.hep.manchester.ac.uk:1095] Destroying MsgHandler: 0xfdeab80.
26-Jul-2025 20:00:17 BST  Closed input file "root://meitner.tier2.hep.manchester.ac.uk:1094//cephfs/experiments/dune/RSE/fardet-vd/f3/c1/prodmarley_nue_es_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250215T044500Z_gen_003476_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0863866      1767.71       2167.38       2110.75       791.082         6     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00240713     0.0297848     0.0863866    0.00567052     0.0360187        6     
reco:pandora:StandardPandora                            1907.08       1953.2        1997.09       1947.9        31.1824         5     
reco:pandoraTrack:LArPandoraTrackCreation             0.00276063    0.00969536     0.0366525    0.00301437     0.013479         5     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000838679   0.00203773    0.00637683    0.00097551    0.00217096        5     
reco:pandoracalo:Calorimetry                          0.000273191   0.000645551   0.00193879    0.000314428   0.000648264       5     
reco:pandorapid:Chi2ParticleID                        8.1757e-05    0.000219894   0.000689399   9.6714e-05    0.000235616       5     
reco:linecluster:LineCluster                           0.263033      0.318306      0.388849      0.330613      0.0468628        5     
reco:trajcluster:TrajCluster                           0.258742      0.288669      0.301215      0.297896      0.0157289        5     
reco:pmtracktc:PMAlgTrackMaker                          4.85699       5.44289       5.89575       5.4214       0.377259         5     
reco:emtrkmichelid:EmTrackMichelId                      155.029       161.232        166.9        162.028       4.00424         5     
reco:solarflash:SolarOpFlash                           0.463285      0.482407      0.509236      0.478393      0.0151676        5     
[art]:TriggerResults:TriggerResultInserter            2.0722e-05     2.696e-05    3.9801e-05    2.2878e-05    7.27742e-06       5     
end_path:out1:RootOutput                               2.982e-06    5.7242e-06    1.5072e-05      3.3e-06     4.69103e-06       5     
end_path:out1:RootOutput(write)                         0.21087       0.22829      0.235085      0.233439      0.0090883        5     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4001 MB
  Peak resident set size usage (VmHWM): 2538.75 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 26-Jul-2025 20:00:18 BST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- ProductNotFound BEGIN
      A request to resolve an Ptr to a product containing items of type: simb::MCTruth with ProductID 3501390858
      cannot be satisfied because the product cannot be found.
      The productGetter was not set -- are you trying to dereference a Ptr during mixing?
      The above exception was thrown while processing module StandardPandora/pandora run: 5179 subRun: 0 event: 34756
    ---- ProductNotFound END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 5 entries while sim::OpDetDivRecs_sipmAr10ppmExt__detsim. has 10 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
[2025-07-26 20:00:18.614651 +0100][Debug  ][JobMgr            ] Stopping the job manager...
[2025-07-26 20:00:18.615174 +0100][Debug  ][JobMgr            ] Job manager stopped
[2025-07-26 20:00:18.615239 +0100][Debug  ][TaskMgr           ] Stopping the task manager...
[2025-07-26 20:00:18.615397 +0100][Debug  ][TaskMgr           ] Task manager stopped
[2025-07-26 20:00:18.615403 +0100][Debug  ][Poller            ] Stopping the poller...
[2025-07-26 20:00:18.615666 +0100][Debug  ][AsyncSock         ] [dtn03.tier2.hep.manchester.ac.uk:1095.0] Closing the socket
[2025-07-26 20:00:18.615702 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.40]:52606><--><[::ffff:195.194.107.136]:1095> Removing socket from the poller
[2025-07-26 20:00:18.615878 +0100][Debug  ][PostMaster        ] [dtn03.tier2.hep.manchester.ac.uk:1095] Destroying stream
[2025-07-26 20:00:18.615913 +0100][Debug  ][AsyncSock         ] [dtn03.tier2.hep.manchester.ac.uk:1095.0] Closing the socket
Art has completed and will exit with status 1.