justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 237924.0@dunegpschedd02.fnal.gov

Jobsub ID237924.0@dunegpschedd02.fnal.gov
Workflow ID9566
Stage ID1
User namemwrobel@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-05 17:53:39
SiteUS_Wisconsin
EntryHCCHTPC_US_Wisconsin_osg01_rhel7
Last heartbeat2025-11-05 19:17:53
From worker nodeHostnamee4008
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit82800 (23 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-05 18:00:34
Input filesusertests:pdhd_prod_beam__226699_12_1_20251009T172328Z_gen_g4_IonScintPDExt.root_227040_95_1_20251010T170701Z_PDInt.root
JobscriptExit code0
Real time1h (4480s)
CPU time1h (4259s = 95%)
Max RSS bytes6412087296 (6115 MiB)
Outputting started2025-11-05 19:15:14
Output filesusertests:pdhd_prod_beam__226699_12_1_20251009T172328Z_gen_g4_IonScintPDExt.root_227040_95_1_20251010T170701Z_PDInt.root_237924_0_1_20251105T180037Z_detsim_reco1.root
Finished2025-11-05 19:17:53
Saved logsjustin-logs:237924.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

g with no regularization...
Begin: 2.70644e+07
0 2.54315e+07
1 2.53562e+07
2 2.53532e+07
Now with regularization...
Begin: 2.44698e+07
0 2.44683e+07
Begin processing the 9th record. run: 20250627 subRun: 1 event: 98 at 05-Nov-2025 13:10:55 CST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 18367!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 110861 traces, 110861 summary
Retagger: tagging trace set: gauss with 84093 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 84093 traces tagged "gauss"
FrameSaver: q=2.87063e+06 n=1386672 tag=gauss
wclsFrameSaver: saving 110861 traces tagged "wiener"
FrameSaver: q=3.45133e+06 n=1304373 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 10896 XUs and 5265 XVs -> 31 XUVs
C:0 T:1 60168 XUs and 26789 XVs -> 895 XUVs
C:0 T:2 15561 XUs and 21021 XVs -> 657 XUVs
C:0 T:3 5 XUs and 6 XVs -> 3 XUVs
C:0 T:4 1 XUs and 5 XVs -> 0 XUVs
C:0 T:5 48097 XUs and 24438 XVs -> 656 XUVs
C:0 T:6 48823 XUs and 62546 XVs -> 5680 XUVs
C:0 T:7 26 XUs and 28 XVs -> 3 XUVs
7925 XUVs total
2877 collection wire objects
7925 potential space points
Neighbour search...
350791 tests to find 207192 neighbours
Iterating with no regularization...
Begin: 4.27655e+07
0 3.9238e+07
1 3.89293e+07
2 3.88923e+07
Now with regularization...
Begin: 3.59767e+07
0 3.59631e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Begin processing the 10th record. run: 20250627 subRun: 1 event: 99 at 05-Nov-2025 13:13:14 CST

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 16307!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 84970 traces, 84970 summary
Retagger: tagging trace set: gauss with 72458 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 72458 traces tagged "gauss"
FrameSaver: q=2.17252e+06 n=1106464 tag=gauss
wclsFrameSaver: saving 84970 traces tagged "wiener"
FrameSaver: q=3.10082e+06 n=1107182 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 14096 XUs and 9702 XVs -> 118 XUVs
C:0 T:1 135063 XUs and 113774 XVs -> 7494 XUVs
C:0 T:2 8333 XUs and 17305 XVs -> 405 XUVs
C:0 T:4 71 XUs and 65 XVs -> 13 XUVs
C:0 T:5 49764 XUs and 18265 XVs -> 556 XUVs
C:0 T:6 10021 XUs and 31760 XVs -> 292 XUVs
C:0 T:7 15 XUs and 21 XVs -> 0 XUVs
8878 XUVs total
2666 collection wire objects
8878 potential space points
Neighbour search...
877270 tests to find 333512 neighbours
Iterating with no regularization...
Begin: 5.50465e+07
0 4.86826e+07
1 4.81196e+07
2 4.80754e+07
Now with regularization...
Begin: 4.70651e+07
0 4.7053e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
05-Nov-2025 13:15:13 CST  Closed output file "pdhd_prod_beam__226699_12_1_20251009T172328Z_gen_g4_IonScintPDExt.root_227040_95_1_20251010T170701Z_PDInt.root_237924_0_1_20251105T180037Z_detsim_reco1.root"
05-Nov-2025 13:15:13 CST  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            85.1198       139.944       219.107       120.103       42.7804        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.00124414    0.00660081     0.0119258    0.00680344    0.00344107       10     
reco:opdec:Deconvolution                             0.655209      0.845717       1.08876      0.830532      0.119496        10     
reco:ophitspe:OpHitFinderDeco                        0.137195      0.166766      0.207186      0.160161      0.0249151       10     
reco:opflash:OpFlashFinder                           0.0465249     0.0956468     0.148879      0.0962164     0.0275303       10     
reco:opslicer:OpSlicer                               0.189473       0.23913      0.317358      0.250694      0.0383493       10     
reco:wclsdatahd:WireCellToolkit                       41.3685       48.3509        60.97        45.3006       6.50091        10     
reco:gaushit:GausHitFinder                           0.577819      0.750333      0.893019      0.755904      0.110491        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000216244   0.000544714   0.00139424    0.000368023   0.00035265       10     
reco:reco3d:SpacePointSolver                         0.325636      0.823172       1.64531      0.721901      0.451267        10     
reco:hitpdune:DisambigFromSpacePoints                 0.29502      0.700025       1.32185      0.659797      0.316459        10     
reco:pandora:StandardPandora                          7.69237       26.7599       77.5089       16.7567       20.7807        10     
reco:pandoraWriter:StandardPandora                   0.182336      0.306026      0.435269      0.279469      0.0881147       10     
reco:pandoraTrack:LArPandoraTrackCreation            0.913602       1.83421       3.11346       1.67751      0.774222        10     
reco:pandoraShower:LArPandoraShowerCreation          0.884255       1.73559       3.01723       1.57456       0.78059        10     
reco:pandoracalo:Calorimetry                          0.50107      0.938548       1.45015       0.8563       0.347779        10     
reco:pandoracalonosce:Calorimetry                    0.532409      0.935176       1.45399      0.887945       0.33971        10     
reco:pandoraShowercalo:ShowerCalorimetry             0.891238       1.71373       3.06562       1.58126      0.713806        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.846685       1.64478       2.97992       1.61013      0.668916        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0369942     0.0558415     0.0783901     0.0545398     0.0150365       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0348133     0.0503137     0.0694288     0.0494827     0.0111302       10     
reco:emtrkmichelid:EmTrackMichelId                    20.3466       36.1527       51.3927       33.6246       10.4156        10     
reco:crttag:SingleCRTMatchingProducer               0.00390056    0.00630183     0.0144473    0.00503785     0.0030315       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00328246    0.00490686    0.00675937    0.00446085     0.0011849       10     
reco:pandora2Track:LArPandoraTrackCreation            2.19171       4.43418       7.57952       3.97607       1.78499        10     
reco:pandora2calo:ShowerCalorimetry                  0.764875       1.5175        2.88318       1.41184      0.629604        10     
reco:pandora2calonosce:ShowerCalorimetry             0.741209       1.46582       2.75283       1.35667      0.587799        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.035036      0.0511738     0.0665098     0.0513394     0.0109084       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0344101     0.0541838     0.0690342     0.0566386     0.0116547       10     
reco:pandora2Shower:LArPandoraShowerCreation         0.994769        1.864         3.037        1.75199      0.724744        10     
reco:pandora2Showercalo:ShowerCalorimetry             1.05172       2.02886       3.73559       1.8574       0.851873        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        1.02249       1.93872       3.63172       1.83111      0.802605        10     
[art]:TriggerResults:TriggerResultInserter          2.3865e-05    3.85771e-05   0.000106508   2.9505e-05    2.3754e-05       10     
end_path:out1:RootOutput                             9.548e-06    2.29897e-05   0.000110776   1.0595e-05    3.02192e-05      10     
end_path:out1:RootOutput(write)                       1.59049       2.35383       3.24049       2.16722      0.492769        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6166.29 MB
  Peak resident set size usage (VmHWM): 3861.54 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11297428
-rw-r--r-- 1 slot1_11 slot1_11      38944 Nov  5 13:15 jobscript.log
-rw-r--r-- 1 slot1_11 slot1_11        200 Nov  5 13:15 justin-processed-pfns.txt
-rw-r--r-- 1 slot1_11 slot1_11        519 Nov  5 13:15 reco_protodunehd_hist.root
-rw-r--r-- 1 slot1_11 slot1_11       3574 Nov  5 13:15 Pandora_Geometry.xml
-rw-r--r-- 1 slot1_11 slot1_11     200704 Nov  5 13:15 mem.db
-rw-r--r-- 1 slot1_11 slot1_11      36864 Nov  5 13:15 time.db
-rw-r--r-- 1 slot1_11 slot1_11 5934210133 Nov  5 13:15 pdhd_prod_beam__226699_12_1_20251009T172328Z_gen_g4_IonScintPDExt.root_227040_95_1_20251010T170701Z_PDInt.root_237924_0_1_20251105T180037Z_detsim_reco1.root
-rw-r--r-- 1 slot1_11 slot1_11   53053460 Nov  5 13:14 Pandora_Events.pndr
-rw-r--r-- 1 slot1_11 slot1_11          0 Nov  5 12:54 ProtoDUNE.log
-rw-r--r-- 1 slot1_11 slot1_11          0 Nov  5 12:50 debugprod.log
-rw-r--r-- 1 slot1_11 slot1_11        519 Nov  5 12:50 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 slot1_11 slot1_11 5580979714 Nov  5 12:50 temp_detsim.root
-rw-r--r-- 1 slot1_11 slot1_11        121 Nov  5 12:00 all-input-dids.txt
justIN time: 2026-02-11 12:17:43 UTC       justIN version: 01.06.00