justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 242475.5@dunegpschedd01.fnal.gov

Jobsub ID242475.5@dunegpschedd01.fnal.gov
Workflow ID9566
Stage ID1
User namemwrobel@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-03 18:03:07
SiteUS_Wisconsin
EntryHCCHTPC_US_Wisconsin_osg01_rhel7
Last heartbeat2025-11-03 20:14:23
From worker nodeHostnamee4002
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit82800 (23 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-03 18:05:11
Input filesusertests:pdhd_prod_beam__231896_186_1_20251008T212406Z_gen_g4_IonScintPDExt.root_227040_83_1_20251010T170410Z_PDInt.root
JobscriptExit code0
Real time2h (7602s)
CPU time1h (7118s = 93%)
Max RSS bytes6804492288 (6489 MiB)
Outputting started2025-11-03 20:11:54
Output filesusertests:pdhd_prod_beam__231896_186_1_20251008T212406Z_gen_g4_IonScintPDExt.root_227040_83_1_20251010T170410Z_PDInt.root_242475_5_1_20251103T180519Z_detsim_reco1.root
Finished2025-11-03 20:14:23
Saved logsjustin-logs:242475.5-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

Begin: 2.05232e+07
0 1.87225e+07
1 1.85899e+07
2 1.85789e+07
Now with regularization...
Begin: 1.65194e+07
0 1.65167e+07
Begin processing the 9th record. run: 20250627 subRun: 1 event: 98 at 03-Nov-2025 14:05:55 CST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 14527!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 80989 traces, 80989 summary
Retagger: tagging trace set: gauss with 66611 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 66611 traces tagged "gauss"
FrameSaver: q=1.8787e+06 n=993809 tag=gauss
wclsFrameSaver: saving 80989 traces tagged "wiener"
FrameSaver: q=2.49635e+06 n=985744 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 11307 XUs and 8976 XVs -> 139 XUVs
C:0 T:1 99226 XUs and 76872 XVs -> 5413 XUVs
C:0 T:2 15949 XUs and 9077 XVs -> 240 XUVs
C:0 T:3 1 XUs and 8 XVs -> 0 XUVs
C:0 T:5 16159 XUs and 19159 XVs -> 281 XUVs
C:0 T:6 21719 XUs and 27201 XVs -> 531 XUVs
C:0 T:7 5 XUs and 2 XVs -> 0 XUVs
6604 XUVs total
2254 collection wire objects
6604 potential space points
Neighbour search...
700430 tests to find 193972 neighbours
Iterating with no regularization...
Begin: 1.64789e+08
0 1.53195e+08
1 1.52669e+08
2 1.52629e+08
Now with regularization...
Begin: 1.50518e+08
0 1.50507e+08
Begin processing the 10th record. run: 20250627 subRun: 1 event: 99 at 03-Nov-2025 14:08:33 CST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 18632!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 103566 traces, 103566 summary
Retagger: tagging trace set: gauss with 84941 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 84941 traces tagged "gauss"
FrameSaver: q=2.79583e+06 n=1459075 tag=gauss
wclsFrameSaver: saving 103566 traces tagged "wiener"
FrameSaver: q=3.7128e+06 n=1415804 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 16573 XUs and 16974 XVs -> 483 XUVs
C:0 T:1 169608 XUs and 183799 XVs -> 20766 XUVs
C:0 T:2 11002 XUs and 12115 XVs -> 570 XUVs
C:0 T:4 9 XUs and 14 XVs -> 7 XUVs
C:0 T:5 28198 XUs and 35382 XVs -> 1234 XUVs
C:0 T:6 38400 XUs and 30436 XVs -> 1370 XUVs
24430 XUVs total
4049 collection wire objects
24430 potential space points
Neighbour search...
6135088 tests to find 2447440 neighbours
Iterating with no regularization...
Begin: 4.89048e+07
0 4.30823e+07
1 4.26473e+07
2 4.25974e+07
3 4.25801e+07
Now with regularization...
Begin: 3.99558e+07
0 3.99336e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
03-Nov-2025 14:11:52 CST  Closed output file "pdhd_prod_beam__231896_186_1_20251008T212406Z_gen_g4_IonScintPDExt.root_227040_83_1_20251010T170410Z_PDInt.root_242475_5_1_20251103T180519Z_detsim_reco1.root"
03-Nov-2025 14:11:52 CST  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            99.3937       279.155       647.019       171.652       190.844        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00215873     0.0181063     0.0686006     0.0112953     0.0201756       10     
reco:opdec:Deconvolution                             0.606742       1.0208        1.87244      0.794032       0.43009        10     
reco:ophitspe:OpHitFinderDeco                        0.127603      0.221994      0.417021      0.183034      0.101983        10     
reco:opflash:OpFlashFinder                           0.0561859     0.100887      0.155377      0.0966406     0.0319299       10     
reco:opslicer:OpSlicer                               0.155992      0.262779      0.419303       0.21632      0.100556        10     
reco:wclsdatahd:WireCellToolkit                       47.018        98.1172       216.044       65.5471       54.8693        10     
reco:gaushit:GausHitFinder                           0.518877      0.872613       1.52909      0.747912      0.337419        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000239467   0.00216116    0.00812743    0.00113803    0.00237696       10     
reco:reco3d:SpacePointSolver                         0.351861       1.96475       3.85894       1.66355       1.18959        10     
reco:hitpdune:DisambigFromSpacePoints                0.300457      0.834323       1.95787      0.646324      0.532239        10     
reco:pandora:StandardPandora                          10.3134       29.924        73.5323       22.7088       19.6966        10     
reco:pandoraWriter:StandardPandora                   0.189482       1.07208       3.0033       0.414239       1.03269        10     
reco:pandoraTrack:LArPandoraTrackCreation            0.907991       3.79501       15.6386       1.99639       4.23275        10     
reco:pandoraShower:LArPandoraShowerCreation           1.02128       3.44958       13.0603       1.75492       3.59229        10     
reco:pandoracalo:Calorimetry                           0.542        1.51984       5.95765       1.03036       1.5303         10     
reco:pandoracalonosce:Calorimetry                    0.536619       1.8227        8.82271        1.019        2.37403        10     
reco:pandoraShowercalo:ShowerCalorimetry             0.918258       2.41988       7.37203       1.68833       1.85438        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.921528       1.98047       3.30963       1.6121        0.91312        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0389543     0.0854532     0.178774      0.0799376     0.0407695       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0377019     0.077693      0.114619      0.0818505     0.0270521       10     
reco:emtrkmichelid:EmTrackMichelId                    23.7386       96.6432       303.991       44.4082       85.9298        10     
reco:crttag:SingleCRTMatchingProducer               0.00463133     0.0201011     0.0992763    0.00745179     0.0278805       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00360411     0.0150607     0.0801287    0.00628736     0.0221801       10     
reco:pandora2Track:LArPandoraTrackCreation            1.94855       13.4663       51.645        3.5594        17.0766        10     
reco:pandora2calo:ShowerCalorimetry                  0.627176       2.18117       8.94003       1.11889       2.36609        10     
reco:pandora2calonosce:ShowerCalorimetry             0.582417       2.20737       6.08217       1.10319       1.8356         10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0373393     0.0890952     0.296772      0.0515856     0.0773758       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0373825     0.0866973     0.267016      0.0528508     0.069333        10     
reco:pandora2Shower:LArPandoraShowerCreation          0.79964       4.22377       20.2487       1.75799       5.60097        10     
reco:pandora2Showercalo:ShowerCalorimetry            0.833598       3.52416       9.8774        1.74298       3.06566        10     
reco:pandora2Showercalonosce:ShowerCalorimetry       0.777933       2.49935       6.66312       1.68234       1.72951        10     
[art]:TriggerResults:TriggerResultInserter          2.3524e-05    0.000182742   0.00119566    3.77005e-05   0.000344665      10     
end_path:out1:RootOutput                             6.162e-06    4.0642e-05    0.000271597   1.0034e-05    7.78662e-05      10     
end_path:out1:RootOutput(write)                       1.90887       4.22741       12.6739       2.69518       3.52047        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6149.82 MB
  Peak resident set size usage (VmHWM): 3823.61 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11472384
-rw-r--r-- 1 slot1_23 slot1_23      39019 Nov  3 14:11 jobscript.log
-rw-r--r-- 1 slot1_23 slot1_23        201 Nov  3 14:11 justin-processed-pfns.txt
-rw-r--r-- 1 slot1_23 slot1_23        519 Nov  3 14:11 reco_protodunehd_hist.root
-rw-r--r-- 1 slot1_23 slot1_23       3574 Nov  3 14:11 Pandora_Geometry.xml
-rw-r--r-- 1 slot1_23 slot1_23     200704 Nov  3 14:11 mem.db
-rw-r--r-- 1 slot1_23 slot1_23      36864 Nov  3 14:11 time.db
-rw-r--r-- 1 slot1_23 slot1_23 6015502710 Nov  3 14:11 pdhd_prod_beam__231896_186_1_20251008T212406Z_gen_g4_IonScintPDExt.root_227040_83_1_20251010T170410Z_PDInt.root_242475_5_1_20251103T180519Z_detsim_reco1.root
-rw-r--r-- 1 slot1_23 slot1_23   49784150 Nov  3 14:10 Pandora_Events.pndr
-rw-r--r-- 1 slot1_23 slot1_23          0 Nov  3 13:26 ProtoDUNE.log
-rw-r--r-- 1 slot1_23 slot1_23          0 Nov  3 13:17 debugprod.log
-rw-r--r-- 1 slot1_23 slot1_23        519 Nov  3 13:16 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 slot1_23 slot1_23 5682106832 Nov  3 13:16 temp_detsim.root
-rw-r--r-- 1 slot1_23 slot1_23        122 Nov  3 12:05 all-input-dids.txt
justIN time: 2025-11-04 01:54:57 UTC       justIN version: 01.05.01