Jobsub ID 242569.7@dunegpschedd01.fnal.gov

Jobsub ID	242569.7@dunegpschedd01.fnal.gov
Workflow ID	9566
Stage ID	1
User name	mwrobel@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	9437184000 (9000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-11-03 19:35:12
Site	US_UChicago
Entry	Engage_US_MWT2_iut2_condce_mcore
Last heartbeat	2025-11-03 22:52:14
From worker node	Hostname	uct2-c506.mwt2.org
	cpuinfo	Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	10485760000 (10000 MiB)
	Wall seconds limit	86400 (24 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Started	2025-11-03 19:47:52
Input files	usertests:pdhd_prod_beam__231896_169_1_20251008T212430Z_gen_g4_IonScintPDExt.root_227040_5_1_20251010T165959Z_PDInt.root
Jobscript	Exit code	0
	Real time	3h (10936s)
	CPU time	2h (10696s = 97%)
	Max RSS bytes	7453274112 (7107 MiB)
Outputting started	2025-11-03 22:50:08
Output files	usertests:pdhd_prod_beam__231896_169_1_20251008T212430Z_gen_g4_IonScintPDExt.root_227040_5_1_20251010T165959Z_PDInt.root_242569_7_1_20251103T194756Z_detsim_reco1.root
Finished	2025-11-03 22:52:14
Saved logs	justin-logs:242569.7-dunegpschedd01.fnal.gov.logs.tgz
List job events Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ed to calculate correlation coefficient p-value for these numbers
----view 0: 0.00362563 0.0334303 0.0406923 0.0284319 0.012742 0.0109339 0.00470901 0.00458598 0.00447631 0.00434875 0.00416374 
----view 1: 0.000235379 0.000235379 0.000235379 0.000235379 0.000235319 0.000235379 0.000235379 0.000235379 0.000235379 0.000235379 0.000235319 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.0334303 0.0406923 0.0284319 0.012742 0.0109339 0.00470901 0.00458598 0.00447631 0.00434875 0.00416374 0.00491309 
----view 1: 0.000235379 0.000235379 0.000235379 0.000235319 0.000235379 0.000235379 0.000235379 0.000235379 0.000235379 0.000235319 0.000235438 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.0406923 0.0284319 0.012742 0.0109339 0.00470901 0.00458598 0.00447631 0.00434875 0.00416374 0.00491309 0.00472379 
----view 1: 0.000235379 0.000235379 0.000235319 0.000235379 0.000235379 0.000235379 0.000235379 0.000235379 0.000235319 0.000235438 0.000235319 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.0284319 0.012742 0.0109339 0.00470901 0.00458598 0.00447631 0.00434875 0.00416374 0.00491309 0.00472379 0.00428391 
----view 1: 0.000235379 0.000235319 0.000235379 0.000235379 0.000235379 0.000235379 0.000235379 0.000235319 0.000235438 0.000235319 0.000235379 
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Begin processing the 10th record. run: 20250627 subRun: 1 event: 99 at 03-Nov-2025 16:42:29 CST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 13504!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 100504 traces, 100504 summary
Retagger: tagging trace set: gauss with 81602 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 81602 traces tagged "gauss"
FrameSaver: q=2.67395e+06 n=1377071 tag=gauss
wclsFrameSaver: saving 100504 traces tagged "wiener"
FrameSaver: q=3.37701e+06 n=1354160 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 12949 XUs and 8174 XVs -> 119 XUVs
C:0 T:1 106677 XUs and 77183 XVs -> 4773 XUVs
C:0 T:2 71703 XUs and 37264 XVs -> 715 XUVs
C:0 T:3 111 XUs and 32 XVs -> 1 XUVs
C:0 T:4 9 XUs and 41 XVs -> 1 XUVs
C:0 T:5 70959 XUs and 31038 XVs -> 1058 XUVs
C:0 T:6 78088 XUs and 55824 XVs -> 2371 XUVs
C:0 T:7 27 XUs and 27 XVs -> 0 XUVs
9038 XUVs total
3587 collection wire objects
9038 potential space points
Neighbour search...
538972 tests to find 185404 neighbours
Iterating with no regularization...
Begin: 6.0231e+07
0 5.40536e+07
1 5.35961e+07
2 5.35541e+07
Now with regularization...
Begin: 5.19984e+07
0 5.19855e+07
03-Nov-2025 16:50:06 CST  Closed output file "pdhd_prod_beam__231896_169_1_20251008T212430Z_gen_g4_IonScintPDExt.root_227040_5_1_20251010T165959Z_PDInt.root_242569_7_1_20251103T194756Z_detsim_reco1.root"
03-Nov-2025 16:50:06 CST  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            225.199       383.034       512.766       417.278       97.1437        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00179632    0.00949966     0.0171823    0.00872676    0.00535082       10     
reco:opdec:Deconvolution                              1.3337        1.67969       2.37232       1.59713      0.282622        10     
reco:ophitspe:OpHitFinderDeco                        0.342138      0.406557      0.494986      0.384657      0.0487904       10     
reco:opflash:OpFlashFinder                            0.11318       0.15038      0.235702      0.129846      0.0365416       10     
reco:opslicer:OpSlicer                               0.276486      0.390186       0.60769      0.385762      0.086153        10     
reco:wclsdatahd:WireCellToolkit                       82.7355       87.4125       97.5785       85.8125       4.62321        10     
reco:gaushit:GausHitFinder                            1.33122       1.72352       2.08666       1.72906      0.270181        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000656625   0.000855778    0.0010199    0.000881101   0.000124146      10     
reco:reco3d:SpacePointSolver                         0.846872       2.66932       4.39478       2.78492       1.05019        10     
reco:hitpdune:DisambigFromSpacePoints                0.815537       1.97898       3.06973       1.93651       0.64744        10     
reco:pandora:StandardPandora                          28.7654       74.6656       159.359       60.134        39.7233        10     
reco:pandoraWriter:StandardPandora                   0.472136      0.639002      0.754182      0.664171      0.104072        10     
reco:pandoraTrack:LArPandoraTrackCreation             5.53651       11.1195       17.6969       12.9551       4.02495        10     
reco:pandoraShower:LArPandoraShowerCreation           5.29079       11.6951       18.591        12.8278       4.18496        10     
reco:pandoracalo:Calorimetry                          2.21523       5.35598       8.40919       6.03155       2.12337        10     
reco:pandoracalonosce:Calorimetry                     2.38723       5.31111       8.3633        5.90718       2.03965        10     
reco:pandoraShowercalo:ShowerCalorimetry              4.25365       11.6136       19.3628       13.3129       4.98915        10     
reco:pandoraShowercalonosce:ShowerCalorimetry         4.22751       10.9636       17.4959       12.7832       4.45148        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0833927      0.12711      0.214217      0.114264      0.0359638       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0766662      0.12218      0.169369      0.122855      0.0257241       10     
reco:emtrkmichelid:EmTrackMichelId                    37.8181       61.6302       80.5665       65.7322       15.0337        10     
reco:crttag:SingleCRTMatchingProducer                0.0124242     0.0215609     0.0306942     0.0207838    0.00502395       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00980971     0.0209197     0.0294177     0.020953     0.00523648       10     
reco:pandora2Track:LArPandoraTrackCreation            11.7426       28.5191       40.9184       30.8235       10.0218        10     
reco:pandora2calo:ShowerCalorimetry                   3.73521       10.2711       16.0815       11.6214       4.24846        10     
reco:pandora2calonosce:ShowerCalorimetry              3.64499       10.1122       15.6898       11.4344       4.17751        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0794123     0.123659      0.167092       0.13065      0.024421        10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.078822      0.114099      0.162019      0.117429      0.0235788       10     
reco:pandora2Shower:LArPandoraShowerCreation          5.49514       12.7777       17.8921       14.192        4.31867        10     
reco:pandora2Showercalo:ShowerCalorimetry             5.11694       13.7762       19.8561       15.9565       5.25373        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        5.16663       13.3365       19.4388       15.3584       5.02419        10     
[art]:TriggerResults:TriggerResultInserter          7.4589e-05    0.000104494   0.000194902   9.53975e-05   3.16986e-05      10     
end_path:out1:RootOutput                            1.5218e-05    2.28172e-05   5.1794e-05    2.01465e-05   9.86803e-06      10     
end_path:out1:RootOutput(write)                       3.17512       4.02269       5.20123       3.93008      0.663579        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6270.32 MB
  Peak resident set size usage (VmHWM): 4037.61 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12539144
-rw-r--r--. 1 dune osgvo      47292 Nov  3 16:50 jobscript.log
-rw-r--r--. 1 dune osgvo        200 Nov  3 16:50 justin-processed-pfns.txt
-rw-r--r--. 1 dune osgvo        519 Nov  3 16:50 reco_protodunehd_hist.root
-rw-r--r--. 1 dune osgvo       3574 Nov  3 16:50 Pandora_Geometry.xml
-rw-r--r--. 1 dune osgvo     200704 Nov  3 16:50 mem.db
-rw-r--r--. 1 dune osgvo      36864 Nov  3 16:50 time.db
-rw-r--r--. 1 dune osgvo 6579343564 Nov  3 16:50 pdhd_prod_beam__231896_169_1_20251008T212430Z_gen_g4_IonScintPDExt.root_227040_5_1_20251010T165959Z_PDInt.root_242569_7_1_20251103T194756Z_detsim_reco1.root
-rw-r--r--. 1 dune osgvo   59323350 Nov  3 16:45 Pandora_Events.pndr
-rw-r--r--. 1 dune osgvo          0 Nov  3 15:47 ProtoDUNE.log
-rw-r--r--. 1 dune osgvo          0 Nov  3 15:43 debugprod.log
-rw-r--r--. 1 dune osgvo        519 Nov  3 15:42 detsim_single_protodunehd_hist.root
-rw-r--r--. 1 dune osgvo 6201081691 Nov  3 15:42 temp_detsim.root
-rw-r--r--. 1 dune osgvo        121 Nov  3 13:47 all-input-dids.txt