Jobsub ID 234970.22@dunegpschedd02.fnal.gov

Jobsub ID	234970.22@dunegpschedd02.fnal.gov
Workflow ID	9454
Stage ID	1
User name	mwrobel@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	9437184000 (9000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-10-29 18:56:46
Site	US_FNAL-FermiGrid
Entry	FNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat	2025-10-29 20:48:34
From worker node	Hostname	dunegli-7568269-0-fnpc23016.fnal.gov
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	10485760000 (10000 MiB)
	Wall seconds limit	172800 (48 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Started	2025-10-29 19:37:58
Input files	usertests:pdhd_prod_beam__231691_98_1_20251008T183359Z_gen_g4_IonScintPDExt.root_227038_125_1_20251010T163818Z_PDInt.root
Jobscript	Exit code	0
	Real time	1h (3833s)
	CPU time	1h (3757s = 98%)
	Max RSS bytes	7099289600 (6770 MiB)
Outputting started	2025-10-29 20:41:51
Output files
Finished	2025-10-29 20:48:34
List job events Cached HTCondor job logs
Jobscript log (last 10,000 characters)

ing with no regularization...
Begin: 5.55359e+07
0 4.77898e+07
1 4.72026e+07
2 4.7161e+07
Now with regularization...
Begin: 4.46222e+07
0 4.46108e+07
Begin processing the 9th record. run: 20250627 subRun: 1 event: 88 at 29-Oct-2025 20:39:01 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 15522!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 101094 traces, 101094 summary
Retagger: tagging trace set: gauss with 80405 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 80405 traces tagged "gauss"
FrameSaver: q=2.35839e+06 n=1247489 tag=gauss
wclsFrameSaver: saving 101094 traces tagged "wiener"
FrameSaver: q=3.04787e+06 n=1197219 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 9555 XUs and 11326 XVs -> 61 XUVs
C:0 T:1 66809 XUs and 67179 XVs -> 2837 XUVs
C:0 T:2 28622 XUs and 16141 XVs -> 418 XUVs
C:0 T:3 16 XUs and 13 XVs -> 0 XUVs
C:0 T:4 47 XUs and 50 XVs -> 3 XUVs
C:0 T:5 79679 XUs and 56837 XVs -> 1611 XUVs
C:0 T:6 15734 XUs and 9118 XVs -> 377 XUVs
C:0 T:7 7 XUs and 4 XVs -> 0 XUVs
5307 XUVs total
2807 collection wire objects
5307 potential space points
Neighbour search...
163535 tests to find 70084 neighbours
Iterating with no regularization...
Begin: 4.02455e+07
0 3.61412e+07
1 3.59333e+07
2 3.59162e+07
Now with regularization...
Begin: 3.45813e+07
0 3.45757e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Begin processing the 10th record. run: 20250627 subRun: 1 event: 89 at 29-Oct-2025 20:40:34 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 10621!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 80996 traces, 80996 summary
Retagger: tagging trace set: gauss with 65782 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 65782 traces tagged "gauss"
FrameSaver: q=1.80211e+06 n=960255 tag=gauss
wclsFrameSaver: saving 80996 traces tagged "wiener"
FrameSaver: q=2.37805e+06 n=945587 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 10722 XUs and 10577 XVs -> 145 XUVs
C:0 T:1 70105 XUs and 71265 XVs -> 2212 XUVs
C:0 T:2 33407 XUs and 34615 XVs -> 618 XUVs
C:0 T:3 8 XUs and 7 XVs -> 2 XUVs
C:0 T:5 6067 XUs and 3769 XVs -> 59 XUVs
C:0 T:6 64057 XUs and 25140 XVs -> 1496 XUVs
C:0 T:7 22 XUs and 8 XVs -> 0 XUVs
4532 XUVs total
1810 collection wire objects
4532 potential space points
Neighbour search...
119406 tests to find 62370 neighbours
Iterating with no regularization...
Begin: 3.11153e+07
0 2.76641e+07
1 2.73023e+07
2 2.72726e+07
3 2.72673e+07
Now with regularization...
Begin: 2.62553e+07
0 2.62527e+07
29-Oct-2025 20:41:51 UTC  Closed output file "pdhd_prod_beam__231691_98_1_20251008T183359Z_gen_g4_IonScintPDExt.root_227038_125_1_20251010T163818Z_PDInt.root_234970_22_1_20251029T193800Z_detsim_reco1.root"
29-Oct-2025 20:41:51 UTC  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            73.7717       92.313        118.548       88.3591       15.8291        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.000530576   0.00404562    0.00749619    0.00430972    0.00221337       10     
reco:opdec:Deconvolution                             0.439998      0.619408      0.740324      0.605369      0.0872743       10     
reco:ophitspe:OpHitFinderDeco                        0.0967165     0.111327      0.152752      0.104764      0.0159746       10     
reco:opflash:OpFlashFinder                           0.0323761     0.075514      0.108128      0.074897      0.0235999       10     
reco:opslicer:OpSlicer                               0.124853       0.1763       0.229845      0.168465      0.0298615       10     
reco:wclsdatahd:WireCellToolkit                       31.1316       31.9232       32.8828       31.9935      0.501065        10     
reco:gaushit:GausHitFinder                            0.38511      0.509167      0.702945      0.492372      0.0867018       10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000179527   0.000242635   0.000366419   0.000227831   5.24089e-05      10     
reco:reco3d:SpacePointSolver                         0.471491      0.646236      0.973783      0.575038      0.181482        10     
reco:hitpdune:DisambigFromSpacePoints                 0.31099      0.480172      0.762544      0.449583      0.150189        10     
reco:pandora:StandardPandora                          11.5738       19.392        46.6635       16.4116       9.98121        10     
reco:pandoraWriter:StandardPandora                    0.11566      0.159069      0.225917      0.156649      0.0292639       10     
reco:pandoraTrack:LArPandoraTrackCreation            0.711143       1.13138       1.89519      0.969218      0.342249        10     
reco:pandoraShower:LArPandoraShowerCreation          0.603735        1.146        2.16757       1.07492      0.458309        10     
reco:pandoracalo:Calorimetry                         0.374513      0.613389       1.08916      0.542623      0.199096        10     
reco:pandoracalonosce:Calorimetry                    0.364547      0.597296       1.0061       0.533794      0.177003        10     
reco:pandoraShowercalo:ShowerCalorimetry             0.585578       1.17679       2.56774       1.12385      0.552743        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.548228       1.11516       2.40062       1.06539      0.514353        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0229267     0.0328893     0.0493658     0.0331223    0.00691522       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0223591     0.0317846     0.0481911     0.0320367    0.00711242       10     
reco:emtrkmichelid:EmTrackMichelId                    17.4495       21.7327       28.532        21.7925       3.31128        10     
reco:crttag:SingleCRTMatchingProducer               0.00265444     0.003743     0.00525618    0.00344659    0.000801172      10     
reco:crtreco:TwoCRTMatchingProducer                  0.0024116    0.00287203    0.00412856    0.00270454    0.00048216       10     
reco:pandora2Track:LArPandoraTrackCreation            1.65113       2.80335       5.20635       2.55919       1.02432        10     
reco:pandora2calo:ShowerCalorimetry                   0.50775       1.03665       2.25721      0.973892      0.485524        10     
reco:pandora2calonosce:ShowerCalorimetry              0.49813       1.02129       2.19853      0.970331      0.472437        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0236674     0.0329574     0.0481959     0.0327018    0.00672755       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.022584      0.0319964     0.0466071     0.0312155      0.00658        10     
reco:pandora2Shower:LArPandoraShowerCreation          0.73542       1.30772       2.32946       1.24246      0.467408        10     
reco:pandora2Showercalo:ShowerCalorimetry            0.818045       1.41814       2.94076       1.2959       0.598495        10     
reco:pandora2Showercalonosce:ShowerCalorimetry       0.799424       1.37377       2.82067       1.26271      0.571335        10     
[art]:TriggerResults:TriggerResultInserter          1.7754e-05    3.16762e-05   0.000125843   2.0715e-05    3.14812e-05      10     
end_path:out1:RootOutput                             3.737e-06    2.77615e-05   0.000226327    6.713e-06    6.62056e-05      10     
end_path:out1:RootOutput(write)                       1.17146       1.54288       1.98818       1.53838      0.217992        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6242.34 MB
  Peak resident set size usage (VmHWM): 4040.77 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12377932
-rw-r--r-- 1 dunegli fnalgrid      39267 Oct 29 20:41 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        201 Oct 29 20:41 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:41 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Oct 29 20:41 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     200704 Oct 29 20:41 mem.db
-rw-r--r-- 1 dunegli fnalgrid      36864 Oct 29 20:41 time.db
-rw-r--r-- 1 dunegli fnalgrid 6487903493 Oct 29 20:41 pdhd_prod_beam__231691_98_1_20251008T183359Z_gen_g4_IonScintPDExt.root_227038_125_1_20251010T163818Z_PDInt.root_234970_22_1_20251029T193800Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   54515690 Oct 29 20:41 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:26 ProtoDUNE.log
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:25 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:25 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 6132275347 Oct 29 20:25 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        122 Oct 29 19:38 all-input-dids.txt