justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 237979.0@dunegpschedd02.fnal.gov

Jobsub ID237979.0@dunegpschedd02.fnal.gov
Workflow ID9566
Stage ID1
User namemwrobel@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-05 19:29:44
SiteUS_Wisconsin
EntryHCCHTPC_US_Wisconsin_osg01_rhel7
Last heartbeat2025-11-05 20:51:00
From worker nodeHostnamee4008
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit82800 (23 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-05 19:36:14
Input filesusertests:pdhd_prod_beam__226470_88_1_20251008T212522Z_gen_g4_IonScintPDExt.root_227040_141_1_20251010T171719Z_PDInt.root
JobscriptExit code0
Real time1h (4203s)
CPU time1h (3957s = 94%)
Max RSS bytes6994112512 (6670 MiB)
Outputting started2025-11-05 20:46:17
Output filesusertests:pdhd_prod_beam__226470_88_1_20251008T212522Z_gen_g4_IonScintPDExt.root_227040_141_1_20251010T171719Z_PDInt.root_237979_0_1_20251105T193618Z_detsim_reco1.root
Finished2025-11-05 20:51:00
Saved logsjustin-logs:237979.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

s -> 0 XUVs
10143 XUVs total
2423 collection wire objects
10143 potential space points
Neighbour search...
1393709 tests to find 684860 neighbours
Iterating with no regularization...
Begin: 2.93841e+07
0 2.69529e+07
1 2.68013e+07
2 2.67739e+07
3 2.67618e+07
Now with regularization...
Begin: 2.47574e+07
0 2.47417e+07
Begin processing the 9th record. run: 20250627 subRun: 1 event: 98 at 05-Nov-2025 14:42:24 CST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 16010!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 80324 traces, 80324 summary
Retagger: tagging trace set: gauss with 65588 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 65588 traces tagged "gauss"
FrameSaver: q=1.88098e+06 n=1009669 tag=gauss
wclsFrameSaver: saving 80324 traces tagged "wiener"
FrameSaver: q=2.42129e+06 n=987758 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 7457 XUs and 8607 XVs -> 64 XUVs
C:0 T:1 44866 XUs and 46488 XVs -> 1553 XUVs
C:0 T:2 13602 XUs and 8606 XVs -> 310 XUVs
C:0 T:3 2 XUs and 3 XVs -> 0 XUVs
C:0 T:5 8197 XUs and 10374 XVs -> 410 XUVs
C:0 T:6 11153 XUs and 11719 XVs -> 643 XUVs
C:0 T:7 15 XUs and 7 XVs -> 0 XUVs
2980 XUVs total
1771 collection wire objects
2980 potential space points
Neighbour search...
36560 tests to find 21156 neighbours
Iterating with no regularization...
Begin: 1.57961e+07
0 1.51255e+07
1 1.50925e+07
2 1.509e+07
Now with regularization...
Begin: 1.40167e+07
0 1.4016e+07
Begin processing the 10th record. run: 20250627 subRun: 1 event: 99 at 05-Nov-2025 14:44:09 CST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 13052!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 95098 traces, 95098 summary
Retagger: tagging trace set: gauss with 74926 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 74926 traces tagged "gauss"
FrameSaver: q=2.33356e+06 n=1181436 tag=gauss
wclsFrameSaver: saving 95098 traces tagged "wiener"
FrameSaver: q=2.96069e+06 n=1149620 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 12944 XUs and 10082 XVs -> 71 XUVs
C:0 T:1 61561 XUs and 42402 XVs -> 2305 XUVs
C:0 T:2 17603 XUs and 17194 XVs -> 1296 XUVs
C:0 T:3 4 XUs and 7 XVs -> 0 XUVs
C:0 T:4 6 XUs and 12 XVs -> 5 XUVs
C:0 T:5 21521 XUs and 27464 XVs -> 1527 XUVs
C:0 T:6 11948 XUs and 14559 XVs -> 654 XUVs
C:0 T:7 16 XUs and 12 XVs -> 0 XUVs
5858 XUVs total
2260 collection wire objects
5858 potential space points
Neighbour search...
234540 tests to find 140502 neighbours
Iterating with no regularization...
Begin: 3.38545e+07
0 3.1882e+07
1 3.16804e+07
2 3.16598e+07
Now with regularization...
Begin: 2.95722e+07
0 2.95639e+07
05-Nov-2025 14:46:16 CST  Closed output file "pdhd_prod_beam__226470_88_1_20251008T212522Z_gen_g4_IonScintPDExt.root_227040_141_1_20251010T171719Z_PDInt.root_237979_0_1_20251105T193618Z_detsim_reco1.root"
05-Nov-2025 14:46:16 CST  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            99.6024       123.77        158.453       128.493       19.5232        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.000932802   0.00618388     0.0138658    0.00600632    0.00380789       10     
reco:opdec:Deconvolution                              0.6892       0.797876      0.888824      0.771091      0.0699387       10     
reco:ophitspe:OpHitFinderDeco                        0.134465      0.158204      0.220817      0.155136      0.0220179       10     
reco:opflash:OpFlashFinder                           0.0728797     0.0915845     0.131603       0.08563      0.0182697       10     
reco:opslicer:OpSlicer                               0.190998      0.228118      0.275289      0.226757      0.0271225       10     
reco:wclsdatahd:WireCellToolkit                       43.8992       45.4807       46.9428       45.4685      0.861937        10     
reco:gaushit:GausHitFinder                           0.502899      0.699712      0.863584      0.726742      0.116828        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000225281   0.000319959   0.000475779   0.000324581   7.40183e-05      10     
reco:reco3d:SpacePointSolver                         0.283713      0.712426       1.3418       0.710577      0.291697        10     
reco:hitpdune:DisambigFromSpacePoints                0.292505      0.648174       1.17624      0.658469      0.269635        10     
reco:pandora:StandardPandora                          8.1273        19.6652       29.5175       18.3474       6.97007        10     
reco:pandoraWriter:StandardPandora                   0.158903      0.244833       0.38392      0.228852      0.0620781       10     
reco:pandoraTrack:LArPandoraTrackCreation            0.963322       1.46702       2.35448       1.39942      0.397099        10     
reco:pandoraShower:LArPandoraShowerCreation          0.622932       1.48537       3.00106       1.44845      0.663697        10     
reco:pandoracalo:Calorimetry                         0.519567      0.838779       1.41454      0.784683      0.262356        10     
reco:pandoracalonosce:Calorimetry                    0.504654      0.797855       1.29477      0.781394      0.224072        10     
reco:pandoraShowercalo:ShowerCalorimetry             0.604433       1.52511       3.28577       1.39931      0.731864        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.583471       1.45573       3.07367       1.39486      0.676321        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0292693     0.0479403     0.0759392     0.0466042     0.0120093       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0318528     0.0469414     0.076042      0.0436004     0.0127867       10     
reco:emtrkmichelid:EmTrackMichelId                    22.4637       33.1568       39.7612       35.4651       5.74856        10     
reco:crttag:SingleCRTMatchingProducer               0.00377459    0.00585294    0.00817595    0.00570101     0.0015604       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00319085    0.00465387    0.00612924    0.00469113    0.00099993       10     
reco:pandora2Track:LArPandoraTrackCreation            1.83033       3.8354        7.80486       3.75886       1.63591        10     
reco:pandora2calo:ShowerCalorimetry                  0.517396       1.31716       2.93072       1.27177      0.664104        10     
reco:pandora2calonosce:ShowerCalorimetry             0.498617       1.28194       2.91609       1.18846      0.665554        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0309566     0.0425562     0.0564007     0.0428987    0.00805299       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0284097     0.0453758     0.0696395     0.0448139     0.0104452       10     
reco:pandora2Shower:LArPandoraShowerCreation         0.857395       1.63982       3.07104       1.60365      0.631211        10     
reco:pandora2Showercalo:ShowerCalorimetry            0.864004       1.81963       3.63192       1.7259       0.734653        10     
reco:pandora2Showercalonosce:ShowerCalorimetry       0.867095       1.77471       3.54722       1.8196       0.766914        10     
[art]:TriggerResults:TriggerResultInserter          1.8555e-05    3.23524e-05   9.9215e-05    2.4851e-05    2.25859e-05      10     
end_path:out1:RootOutput                             6.633e-06    1.25625e-05   3.5296e-05     7.529e-06    1.03071e-05      10     
end_path:out1:RootOutput(write)                       1.72563       2.36799       3.14758       2.33861      0.472675        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6126.98 MB
  Peak resident set size usage (VmHWM): 3868.07 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11523856
-rw-r--r-- 1 slot1_11 slot1_11      38979 Nov  5 14:46 jobscript.log
-rw-r--r-- 1 slot1_11 slot1_11        201 Nov  5 14:46 justin-processed-pfns.txt
-rw-r--r-- 1 slot1_11 slot1_11        519 Nov  5 14:46 reco_protodunehd_hist.root
-rw-r--r-- 1 slot1_11 slot1_11       3574 Nov  5 14:46 Pandora_Geometry.xml
-rw-r--r-- 1 slot1_11 slot1_11     200704 Nov  5 14:46 mem.db
-rw-r--r-- 1 slot1_11 slot1_11      36864 Nov  5 14:46 time.db
-rw-r--r-- 1 slot1_11 slot1_11 6044404639 Nov  5 14:46 pdhd_prod_beam__226470_88_1_20251008T212522Z_gen_g4_IonScintPDExt.root_227040_141_1_20251010T171719Z_PDInt.root_237979_0_1_20251105T193618Z_detsim_reco1.root
-rw-r--r-- 1 slot1_11 slot1_11   50805940 Nov  5 14:45 Pandora_Events.pndr
-rw-r--r-- 1 slot1_11 slot1_11          0 Nov  5 14:26 ProtoDUNE.log
-rw-r--r-- 1 slot1_11 slot1_11          0 Nov  5 14:24 debugprod.log
-rw-r--r-- 1 slot1_11 slot1_11        519 Nov  5 14:23 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 slot1_11 slot1_11 5704894302 Nov  5 14:23 temp_detsim.root
-rw-r--r-- 1 slot1_11 slot1_11        122 Nov  5 13:36 all-input-dids.txt
justIN time: 2026-02-11 12:16:27 UTC       justIN version: 01.06.00