Jobsub ID 13339.90@dunegpschedd02.fnal.gov

Jobsub ID	13339.90@dunegpschedd02.fnal.gov
Workflow ID	163
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-31 15:31:59
Site	UK_Edinburgh
Entry	DUNE_UK_SGridECDF_ce1_multicore
Last heartbeat	2025-07-31 21:23:39
From worker node	Hostname	node2b08.ecdf.ed.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	stalled
Started	2025-07-31 15:33:21
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T162839Z_gen_002511_supernova_g4stage1_g4stage2_detsim_reco.root
Outputting started	2025-07-31 21:22:38
Output files
Finished	2025-07-31 22:25:57
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

turn failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:19:53 BST run: 4984 subRun: 0 event: 25110
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:19:53 BST run: 4984 subRun: 0 event: 25110
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:19:53 BST run: 4984 subRun: 0 event: 25110
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:19:53 BST run: 4984 subRun: 0 event: 25110
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:19:53 BST run: 4984 subRun: 0 event: 25110
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  31-Jul-2025 22:19:53 BST run: 4984 subRun: 0 event: 25110
Comparing two wires in the same plane: return failure
%MSG
[2025-07-31 22:22:32.747574 +0100][Debug  ][File              ] [0xf61f300@root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/34/a5/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T162839Z_gen_002511_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=aa992606-6c0b-4529-892d-eb11ecdfa556] Sending a read command for handle 0x0 to pubstor2305.fnal.gov:21228
[2025-07-31 22:22:32.747739 +0100][Debug  ][ExDbgMsg          ] [pubstor2305.fnal.gov:21228] MsgHandler created: 0x48e3fe90 (message: kXR_read (handle: 0x00000000, offset: 2973748500, size: 1378721) ).
[2025-07-31 22:22:32.747866 +0100][Debug  ][ExDbgMsg          ] [pubstor2305.fnal.gov:21228] Moving MsgHandler: 0x48e3fe90 (message: kXR_read (handle: 0x00000000, offset: 2973748500, size: 1378721) ) from out-queu to in-queue.
[2025-07-31 22:22:32.870370 +0100][Debug  ][ExDbgMsg          ] [msg: 0x1738f840] Assigned MsgHandler: 0x48e3fe90.
[2025-07-31 22:22:32.870631 +0100][Debug  ][ExDbgMsg          ] [handler: 0x48e3fe90] Removed MsgHandler: 0x48e3fe90 from the in-queue.
[2025-07-31 22:22:33.467763 +0100][Debug  ][ExDbgMsg          ] [pubstor2305.fnal.gov:21228] Calling MsgHandler: 0x48e3fe90 (message: kXR_read (handle: 0x00000000, offset: 2973748500, size: 1378721) ) with status: [SUCCESS] .
[2025-07-31 22:22:33.467814 +0100][Debug  ][ExDbgMsg          ] [pubstor2305.fnal.gov:21228] Destroying MsgHandler: 0x48e3fe90.
31-Jul-2025 22:22:34 BST  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T162839Z_gen_002511_supernova_g4stage1_g4stage2_detsim_reco_20250731T153324Z_reco2.root"
[2025-07-31 22:22:34.869013 +0100][Debug  ][File              ] [0xf61f300@root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/34/a5/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T162839Z_gen_002511_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=aa992606-6c0b-4529-892d-eb11ecdfa556] Sending a close command for handle 0x0 to pubstor2305.fnal.gov:21228
[2025-07-31 22:22:34.869081 +0100][Debug  ][ExDbgMsg          ] [pubstor2305.fnal.gov:21228] MsgHandler created: 0x336d03c0 (message: kXR_close (handle: 0x00000000) ).
[2025-07-31 22:22:34.869397 +0100][Debug  ][ExDbgMsg          ] [pubstor2305.fnal.gov:21228] Moving MsgHandler: 0x336d03c0 (message: kXR_close (handle: 0x00000000) ) from out-queu to in-queue.
[2025-07-31 22:22:34.969305 +0100][Debug  ][ExDbgMsg          ] [msg: 0x33c1a420] Assigned MsgHandler: 0x336d03c0.
[2025-07-31 22:22:34.969357 +0100][Debug  ][ExDbgMsg          ] [handler: 0x336d03c0] Removed MsgHandler: 0x336d03c0 from the in-queue.
[2025-07-31 22:22:34.969415 +0100][Debug  ][ExDbgMsg          ] [pubstor2305.fnal.gov:21228] Calling MsgHandler: 0x336d03c0 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] .
[2025-07-31 22:22:34.969573 +0100][Debug  ][File              ] [0xf61f300@root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/34/a5/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T162839Z_gen_002511_supernova_g4stage1_g4stage2_detsim_reco.root?xrdcl.requuid=aa992606-6c0b-4529-892d-eb11ecdfa556] Close returned from pubstor2305.fnal.gov:21228 with: [SUCCESS] 
[2025-07-31 22:22:34.969639 +0100][Debug  ][ExDbgMsg          ] [pubstor2305.fnal.gov:21228] Destroying MsgHandler: 0x336d03c0.
31-Jul-2025 22:22:34 BST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/34/a5/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T162839Z_gen_002511_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1992.41       2073.37       2227.44       2057.99       58.7113        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00280517     0.0294359     0.128414     0.00631052     0.0483063       10     
reco:pandora:StandardPandora                            1824.88        1907         2049.39       1893.36       55.4974        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000199505   0.00569961     0.0393869    0.00276983     0.0112894       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000246611    0.0012671    0.00657386    0.000849771    0.001791        10     
reco:pandoracalo:Calorimetry                          0.000255936   0.00113319     0.0027961    0.000309309   0.00106118       10     
reco:pandorapid:Chi2ParticleID                        5.2266e-05    0.000139612   0.000724091   8.1524e-05    0.000195258      10     
reco:linecluster:LineCluster                           0.964947       1.03158       1.41741      0.982786      0.129589        10     
reco:trajcluster:TrajCluster                           0.269138      0.288162      0.317609      0.288833       0.01383        10     
reco:pmtracktc:PMAlgTrackMaker                          4.88087        5.403        6.20516       5.18717      0.466106        10     
reco:emtrkmichelid:EmTrackMichelId                      151.335       158.223       169.539       158.29        5.04392        10     
reco:solarflash:SolarOpFlash                            1.13124       1.17374       1.39381       1.14941      0.0739957       10     
[art]:TriggerResults:TriggerResultInserter            2.2349e-05    2.74316e-05   4.7191e-05    2.5365e-05    6.7658e-06       10     
end_path:out1:RootOutput                               5.399e-06     7.576e-06    1.7311e-05     5.871e-06    3.59966e-06      10     
end_path:out1:RootOutput(write)                        0.201154      0.218011      0.238614       0.21533      0.0114041       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4006.88 MB
  Peak resident set size usage (VmHWM): 2537.88 MB
====================================================================================================
[2025-07-31 22:22:35.376395 +0100][Debug  ][JobMgr            ] Stopping the job manager...
[2025-07-31 22:22:35.376859 +0100][Debug  ][JobMgr            ] Job manager stopped
[2025-07-31 22:22:35.376938 +0100][Debug  ][TaskMgr           ] Stopping the task manager...
[2025-07-31 22:22:35.377067 +0100][Debug  ][TaskMgr           ] Task manager stopped
[2025-07-31 22:22:35.377074 +0100][Debug  ][Poller            ] Stopping the poller...
[2025-07-31 22:22:35.377274 +0100][Debug  ][AsyncSock         ] [pubstor2305.fnal.gov:21228.0] Closing the socket
[2025-07-31 22:22:35.377294 +0100][Debug  ][Poller            ] <[::ffff:192.41.105.41]:41436><--><[::ffff:131.225.69.91]:21228> Removing socket from the poller
[2025-07-31 22:22:35.377343 +0100][Debug  ][PostMaster        ] [pubstor2305.fnal.gov:21228] Destroying stream
[2025-07-31 22:22:35.377350 +0100][Debug  ][AsyncSock         ] [pubstor2305.fnal.gov:21228.0] Closing the socket
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T162839Z_gen_002511_supernova_g4stage1_g4stage2_detsim_reco_20250731T153324Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T162839Z_gen_002511_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T162839Z_gen_002511_supernova_g4stage1_g4stage2_detsim_reco_20250731T153324Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T162839Z_gen_002511_supernova_g4stage1_g4stage2_detsim_reco_20250731T153324Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250129T162839Z_gen_002511_supernova_g4stage1_g4stage2_detsim_reco_20250731T153324Z_reco2.root.json
reco2_hist.root