justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 241480.2@dunegpschedd01.fnal.gov

Jobsub ID241480.2@dunegpschedd01.fnal.gov
Workflow ID9454
Stage ID1
User namemwrobel@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-31 19:51:22
SiteUS_UChicago
EntryEngage_US_MWT2_uct2_gk02_condce_mcore
Last heartbeat2025-10-31 22:00:40
From worker nodeHostnamemwt2-c194.campuscluster.illinois.edu
cpuinfoAMD EPYC 7402 24-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit86400 (24 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-10-31 20:01:17
Input filesusertests:pdhd_prod_beam__231691_25_1_20251008T183116Z_gen_g4_IonScintPDExt.root_227038_32_1_20251010T161152Z_PDInt.root
JobscriptExit code0
Real time1h (7037s)
CPU time1h (6801s = 96%)
Max RSS bytes7108665344 (6779 MiB)
Outputting started2025-10-31 21:58:34
Output filesusertests:pdhd_prod_beam__231691_25_1_20251008T183116Z_gen_g4_IonScintPDExt.root_227038_32_1_20251010T161152Z_PDInt.root_241480_2_1_20251031T200125Z_detsim_reco1.root
Finished2025-10-31 22:00:40
Saved logsjustin-logs:241480.2-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

4 XUs and 8049 XVs -> 339 XUVs
C:0 T:7 2 XUs and 5 XVs -> 0 XUVs
6885 XUVs total
3197 collection wire objects
6885 potential space points
Neighbour search...
309155 tests to find 95100 neighbours
Iterating with no regularization...
Begin: 1.18286e+08
0 1.08082e+08
1 1.0758e+08
2 1.07541e+08
Now with regularization...
Begin: 1.05085e+08
0 1.05077e+08
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Begin processing the 9th record. run: 20250627 subRun: 1 event: 88 at 31-Oct-2025 16:52:49 CDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 17933!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 94192 traces, 94192 summary
Retagger: tagging trace set: gauss with 77360 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 77360 traces tagged "gauss"
FrameSaver: q=2.33847e+06 n=1225980 tag=gauss
wclsFrameSaver: saving 94192 traces tagged "wiener"
FrameSaver: q=2.97367e+06 n=1188998 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 14249 XUs and 12679 XVs -> 170 XUVs
C:0 T:1 107116 XUs and 90858 XVs -> 3501 XUVs
C:0 T:2 39228 XUs and 42123 XVs -> 2363 XUVs
C:0 T:3 14 XUs and 27 XVs -> 4 XUVs
C:0 T:5 11765 XUs and 10713 XVs -> 376 XUVs
C:0 T:6 8933 XUs and 11317 XVs -> 459 XUVs
6873 XUVs total
3284 collection wire objects
6873 potential space points
Neighbour search...
292859 tests to find 106682 neighbours
Iterating with no regularization...
Begin: 4.00242e+07
0 3.56181e+07
1 3.52643e+07
2 3.52322e+07
Now with regularization...
Begin: 3.32168e+07
0 3.32069e+07
Begin processing the 10th record. run: 20250627 subRun: 1 event: 89 at 31-Oct-2025 16:55:45 CDT

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 13822!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 95150 traces, 95150 summary
Retagger: tagging trace set: gauss with 77358 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 77358 traces tagged "gauss"
FrameSaver: q=2.50237e+06 n=1230460 tag=gauss
wclsFrameSaver: saving 95150 traces tagged "wiener"
FrameSaver: q=3.16254e+06 n=1188028 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 9011 XUs and 9553 XVs -> 80 XUVs
C:0 T:1 70454 XUs and 65031 XVs -> 2180 XUVs
C:0 T:2 26094 XUs and 9043 XVs -> 429 XUVs
C:0 T:4 15 XUs and 22 XVs -> 0 XUVs
C:0 T:5 48668 XUs and 55324 XVs -> 3211 XUVs
C:0 T:6 8599 XUs and 9702 XVs -> 650 XUVs
6550 XUVs total
3536 collection wire objects
6550 potential space points
Neighbour search...
219868 tests to find 84944 neighbours
Iterating with no regularization...
Begin: 5.8299e+07
0 5.30893e+07
1 5.26437e+07
2 5.2615e+07
Now with regularization...
Begin: 4.86315e+07
0 4.8621e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
31-Oct-2025 16:58:32 CDT  Closed output file "pdhd_prod_beam__231691_25_1_20251008T183116Z_gen_g4_IonScintPDExt.root_227038_32_1_20251010T161152Z_PDInt.root_241480_2_1_20251031T200125Z_detsim_reco1.root"
31-Oct-2025 16:58:32 CDT  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            143.923       173.222       241.953       167.66        25.7583        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                               0.0012703    0.00713991     0.0125939    0.00699873    0.00361936       10     
reco:opdec:Deconvolution                             0.837699      0.948111       1.15336      0.915635      0.0969701       10     
reco:ophitspe:OpHitFinderDeco                        0.169174      0.219804      0.234904      0.222423      0.0177222       10     
reco:opflash:OpFlashFinder                           0.0669521     0.0883553     0.118935      0.0809408     0.0188913       10     
reco:opslicer:OpSlicer                               0.230265      0.287504      0.387362      0.278463      0.0462356       10     
reco:wclsdatahd:WireCellToolkit                       48.7072       51.1178       57.497        50.1721       2.45851        10     
reco:gaushit:GausHitFinder                           0.713918      0.919886       1.22171      0.912339      0.136749        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000446922   0.000688582    0.001235     0.000634043   0.000211291      10     
reco:reco3d:SpacePointSolver                          0.77963       1.45217       2.82698       1.36023      0.571629        10     
reco:hitpdune:DisambigFromSpacePoints                0.607406      0.908773       1.5719       0.856102      0.267456        10     
reco:pandora:StandardPandora                          19.291        36.1489       66.9117       32.6586       12.5969        10     
reco:pandoraWriter:StandardPandora                    0.34247      0.432915      0.533255      0.429677      0.0497763       10     
reco:pandoraTrack:LArPandoraTrackCreation             2.82824       3.70992       6.91067       3.04172       1.18241        10     
reco:pandoraShower:LArPandoraShowerCreation           2.32492       3.75126       5.3708        3.44655      0.934234        10     
reco:pandoracalo:Calorimetry                          1.23568       1.90949       4.13231       1.66729      0.785324        10     
reco:pandoracalonosce:Calorimetry                     1.18597       1.81755       4.02459       1.57756       0.77233        10     
reco:pandoraShowercalo:ShowerCalorimetry              1.96397       3.18089       5.43856       2.86773      0.979068        10     
reco:pandoraShowercalonosce:ShowerCalorimetry         1.95521       2.97705       4.51413       2.82463      0.724574        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0529797     0.0782904     0.0924749     0.0807226     0.0110104       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0511659     0.0755259     0.0890582     0.0778326     0.0107123       10     
reco:emtrkmichelid:EmTrackMichelId                    33.2659       37.8764       47.3403       36.4918        3.991         10     
reco:crttag:SingleCRTMatchingProducer               0.00719434    0.00968694     0.013534      0.0093243    0.00168146       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00661022    0.00836267     0.0116617    0.00786841    0.00157441       10     
reco:pandora2Track:LArPandoraTrackCreation            5.24245       8.03092       12.696        7.43243       1.99904        10     
reco:pandora2calo:ShowerCalorimetry                   1.37212       2.43094       3.99797       2.28191      0.739197        10     
reco:pandora2calonosce:ShowerCalorimetry              1.25528       2.26177       3.14115       2.24819      0.540168        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0564371     0.0730289     0.0947374     0.0720468     0.0111963       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0608388     0.0728792     0.0927673     0.0702905     0.0102348       10     
reco:pandora2Shower:LArPandoraShowerCreation          2.43884       3.26409       4.75481       2.97528      0.790275        10     
reco:pandora2Showercalo:ShowerCalorimetry             2.18918       3.08436       5.21474       2.91398      0.814933        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        2.03305       3.13857       5.91592       2.90774       1.0291         10     
[art]:TriggerResults:TriggerResultInserter          2.9766e-05    5.03848e-05   0.000120106   3.9374e-05    2.73634e-05      10     
end_path:out1:RootOutput                             7.665e-06    1.14977e-05   3.2942e-05    9.3475e-06    7.21368e-06      10     
end_path:out1:RootOutput(write)                       2.32794       2.73015       3.53996       2.75674      0.335177        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6453.56 MB
  Peak resident set size usage (VmHWM): 4204.35 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11863848
-rw-r--r-- 1 dune osgvo      39009 Oct 31 16:58 jobscript.log
-rw-r--r-- 1 dune osgvo        200 Oct 31 16:58 justin-processed-pfns.txt
-rw-r--r-- 1 dune osgvo        519 Oct 31 16:58 reco_protodunehd_hist.root
-rw-r--r-- 1 dune osgvo       3574 Oct 31 16:58 Pandora_Geometry.xml
-rw-r--r-- 1 dune osgvo     200704 Oct 31 16:58 mem.db
-rw-r--r-- 1 dune osgvo      36864 Oct 31 16:58 time.db
-rw-r--r-- 1 dune osgvo 6225294313 Oct 31 16:58 pdhd_prod_beam__231691_25_1_20251008T183116Z_gen_g4_IonScintPDExt.root_227038_32_1_20251010T161152Z_PDInt.root_241480_2_1_20251031T200125Z_detsim_reco1.root
-rw-r--r-- 1 dune osgvo   55422530 Oct 31 16:57 Pandora_Events.pndr
-rw-r--r-- 1 dune osgvo          0 Oct 31 16:31 ProtoDUNE.log
-rw-r--r-- 1 dune osgvo          0 Oct 31 16:27 debugprod.log
-rw-r--r-- 1 dune osgvo        519 Oct 31 16:27 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dune osgvo 5867528903 Oct 31 16:27 temp_detsim.root
-rw-r--r-- 1 dune osgvo        121 Oct 31 15:01 all-input-dids.txt
justIN time: 2025-11-04 01:29:08 UTC       justIN version: 01.05.01