Jobsub ID 12132.163@dunegpschedd02.fnal.gov

Jobsub ID	12132.163@dunegpschedd02.fnal.gov
Workflow ID	102
Stage ID	1
User name	higuera@fnal.gov
HTCondor Group	group_dune.prod.mcsim
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-07-28 23:32:39
Site	US_UCSD
Entry	CMSHTPC_T2_US_UCSD_gw7
Last heartbeat	2025-07-29 05:01:40
From worker node	Hostname	sdsc-67.t2.ucsd.edu
	cpuinfo	Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	stalled
Started	2025-07-28 23:51:31
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250128T133836Z_gen_001881_supernova_g4stage1_g4stage2_detsim_reco.root
Outputting started	2025-07-29 04:54:49
Output files
Finished	2025-07-29 06:42:06
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 21:51:06 PDT run: 4966 subRun: 0 event: 18810
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 21:51:06 PDT run: 4966 subRun: 0 event: 18810
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 21:51:06 PDT run: 4966 subRun: 0 event: 18810
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 21:51:06 PDT run: 4966 subRun: 0 event: 18810
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 21:51:06 PDT run: 4966 subRun: 0 event: 18810
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 21:51:06 PDT run: 4966 subRun: 0 event: 18810
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 21:51:06 PDT run: 4966 subRun: 0 event: 18810
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 21:51:06 PDT run: 4966 subRun: 0 event: 18810
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 21:51:06 PDT run: 4966 subRun: 0 event: 18810
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 21:51:06 PDT run: 4966 subRun: 0 event: 18810
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 21:51:06 PDT run: 4966 subRun: 0 event: 18810
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 21:51:06 PDT run: 4966 subRun: 0 event: 18810
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 21:51:06 PDT run: 4966 subRun: 0 event: 18810
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 21:51:06 PDT run: 4966 subRun: 0 event: 18810
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 21:51:06 PDT run: 4966 subRun: 0 event: 18810
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 21:51:06 PDT run: 4966 subRun: 0 event: 18810
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 21:51:06 PDT run: 4966 subRun: 0 event: 18810
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 21:51:06 PDT run: 4966 subRun: 0 event: 18810
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 21:51:06 PDT run: 4966 subRun: 0 event: 18810
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 21:51:06 PDT run: 4966 subRun: 0 event: 18810
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 21:51:06 PDT run: 4966 subRun: 0 event: 18810
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 21:51:06 PDT run: 4966 subRun: 0 event: 18810
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 21:51:06 PDT run: 4966 subRun: 0 event: 18810
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 21:51:06 PDT run: 4966 subRun: 0 event: 18810
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 21:51:06 PDT run: 4966 subRun: 0 event: 18810
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 21:51:06 PDT run: 4966 subRun: 0 event: 18810
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 21:51:06 PDT run: 4966 subRun: 0 event: 18810
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  28-Jul-2025 21:51:06 PDT run: 4966 subRun: 0 event: 18810
Comparing two wires in the same plane: return failure
%MSG
28-Jul-2025 21:54:44 PDT  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250128T133836Z_gen_001881_supernova_g4stage1_g4stage2_detsim_reco_20250728T235134Z_reco2.root"
28-Jul-2025 21:54:44 PDT  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/ec/cc/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250128T133836Z_gen_001881_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1703.94       1798.01       1994.46       1746.26       96.0933        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00304148     0.0177395     0.0586186    0.00900754     0.0205454       10     
reco:pandora:StandardPandora                            1515.96       1598.09       1773.63       1551.55       85.852         10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000253772   0.00756785     0.0502921    0.00382074     0.0144147       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000263428   0.00193373    0.00680581    0.00108441    0.00198105       10     
reco:pandoracalo:Calorimetry                          0.000271361   0.00134973    0.00399431    0.000421951   0.00139719       10     
reco:pandorapid:Chi2ParticleID                        5.7091e-05    0.000195863   0.000729022   8.4097e-05    0.000228472      10     
reco:linecluster:LineCluster                            0.55877      0.663884       0.8944       0.621479      0.107781        10     
reco:trajcluster:TrajCluster                           0.303332      0.405509      0.525708      0.396162       0.07953        10     
reco:pmtracktc:PMAlgTrackMaker                          4.7302        5.66631       8.15226       5.20562       1.10127        10     
reco:emtrkmichelid:EmTrackMichelId                      181.03        191.882       213.446       186.984       11.5387        10     
reco:solarflash:SolarOpFlash                           0.809157      0.983127       1.2528       0.875253      0.185257        10     
[art]:TriggerResults:TriggerResultInserter            2.2075e-05    3.15882e-05   5.3546e-05    2.87315e-05   9.03109e-06      10     
end_path:out1:RootOutput                               5.617e-06    8.6965e-06    2.1721e-05     7.172e-06    4.52602e-06      10     
end_path:out1:RootOutput(write)                        0.216373       0.29095      0.417924      0.232988      0.084013        10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 4045.59 MB
  Peak resident set size usage (VmHWM): 2566.23 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250128T133836Z_gen_001881_supernova_g4stage1_g4stage2_detsim_reco_20250728T235134Z_reco2.root
Extracted metadata
checking file====== fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250128T133836Z_gen_001881_supernova_g4stage1_g4stage2_detsim_reco.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
jobscript.log
justin-processed-pfns.txt
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250128T133836Z_gen_001881_supernova_g4stage1_g4stage2_detsim_reco_20250728T235134Z_reco2.root
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250128T133836Z_gen_001881_supernova_g4stage1_g4stage2_detsim_reco_20250728T235134Z_reco2.root.ext.json
prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250128T133836Z_gen_001881_supernova_g4stage1_g4stage2_detsim_reco_20250728T235134Z_reco2.root.json
reco2_hist.root