justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 234970.30@dunegpschedd02.fnal.gov

Jobsub ID234970.30@dunegpschedd02.fnal.gov
Workflow ID9454
Stage ID1
User namemwrobel@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-29 18:56:46
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-10-29 20:45:17
From worker nodeHostnamedunegli-7568338-0-fnpc23050.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-10-29 19:38:01
Input filesusertests:pdhd_prod_beam__226242_25_1_20251008T182736Z_gen_g4_IonScintPDExt.root_227038_137_1_20251010T164623Z_PDInt.root
JobscriptExit code0
Real time1h (3630s)
CPU time59m (3550s = 97%)
Max RSS bytes6458994688 (6159 MiB)
Outputting started2025-10-29 20:38:31
Output files
Finished2025-10-29 20:45:17
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

collection wire objects
6580 potential space points
Neighbour search...
375264 tests to find 150124 neighbours
Iterating with no regularization...
Begin: 6.15294e+07
0 5.46291e+07
1 5.40591e+07
2 5.402e+07
Now with regularization...
Begin: 5.20861e+07
0 5.20719e+07
Begin processing the 9th record. run: 20250627 subRun: 1 event: 88 at 29-Oct-2025 20:35:43 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 12737!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 87871 traces, 87871 summary
Retagger: tagging trace set: gauss with 71204 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 71204 traces tagged "gauss"
FrameSaver: q=2.06648e+06 n=1092347 tag=gauss
wclsFrameSaver: saving 87871 traces tagged "wiener"
FrameSaver: q=2.68281e+06 n=1062329 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 11970 XUs and 10871 XVs -> 113 XUVs
C:0 T:1 67471 XUs and 63040 XVs -> 2984 XUVs
C:0 T:2 23681 XUs and 9290 XVs -> 322 XUVs
C:0 T:3 14 XUs and 6 XVs -> 0 XUVs
C:0 T:4 5 XUs and 40 XVs -> 0 XUVs
C:0 T:5 10053 XUs and 26537 XVs -> 355 XUVs
C:0 T:6 31555 XUs and 31142 XVs -> 1114 XUVs
C:0 T:7 7 XUs and 28 XVs -> 3 XUVs
4891 XUVs total
2373 collection wire objects
4891 potential space points
Neighbour search...
111405 tests to find 61888 neighbours
Iterating with no regularization...
Begin: 2.93339e+07
0 2.74822e+07
1 2.73093e+07
2 2.72953e+07
Now with regularization...
Begin: 2.59472e+07
0 2.59422e+07
Begin processing the 10th record. run: 20250627 subRun: 1 event: 89 at 29-Oct-2025 20:37:03 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 17409!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 91440 traces, 91440 summary
Retagger: tagging trace set: gauss with 72330 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 72330 traces tagged "gauss"
FrameSaver: q=2.30973e+06 n=1135117 tag=gauss
wclsFrameSaver: saving 91440 traces tagged "wiener"
FrameSaver: q=3.00572e+06 n=1118750 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 7535 XUs and 6410 XVs -> 117 XUVs
C:0 T:1 91260 XUs and 77854 XVs -> 4777 XUVs
C:0 T:2 29904 XUs and 27830 XVs -> 771 XUVs
C:0 T:3 42 XUs and 43 XVs -> 0 XUVs
C:0 T:4 9 XUs and 16 XVs -> 0 XUVs
C:0 T:5 24320 XUs and 20778 XVs -> 475 XUVs
C:0 T:6 37812 XUs and 95862 XVs -> 4497 XUVs
C:0 T:7 52 XUs and 90 XVs -> 26 XUVs
10663 XUVs total
2962 collection wire objects
10663 potential space points
Neighbour search...
1065473 tests to find 456680 neighbours
Iterating with no regularization...
Begin: 5.05642e+07
0 4.44347e+07
1 4.3857e+07
2 4.37886e+07
3 4.37725e+07
Now with regularization...
Begin: 4.21565e+07
0 4.21445e+07
29-Oct-2025 20:38:31 UTC  Closed output file "pdhd_prod_beam__226242_25_1_20251008T182736Z_gen_g4_IonScintPDExt.root_227038_137_1_20251010T164623Z_PDInt.root_234970_30_1_20251029T193803Z_detsim_reco1.root"
29-Oct-2025 20:38:31 UTC  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            63.939        82.2801       108.571       80.0671       11.9384        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.000390037   0.00366798    0.00633894    0.00440885    0.00200532       10     
reco:opdec:Deconvolution                              0.40222      0.570028      0.665844      0.571807      0.0778479       10     
reco:ophitspe:OpHitFinderDeco                        0.0963482     0.103596       0.10872      0.103711     0.00359812       10     
reco:opflash:OpFlashFinder                           0.0319366     0.0663615     0.0868015     0.0624299     0.0162804       10     
reco:opslicer:OpSlicer                               0.109782      0.159762      0.204417      0.155995      0.0314836       10     
reco:wclsdatahd:WireCellToolkit                       31.078        31.6425       33.5543       31.4258      0.678186        10     
reco:gaushit:GausHitFinder                           0.325251      0.475464      0.607181      0.478003      0.0670152       10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000164004   0.000231236   0.00043858    0.000204035   7.27211e-05      10     
reco:reco3d:SpacePointSolver                         0.252219      0.560457       1.22192      0.447413      0.272273        10     
reco:hitpdune:DisambigFromSpacePoints                0.172669      0.403574       0.81917       0.35716       0.18312        10     
reco:pandora:StandardPandora                          7.3696        13.2721       19.6245       13.2511       3.56497        10     
reco:pandoraWriter:StandardPandora                    0.10408      0.147898      0.214462       0.14265      0.0276941       10     
reco:pandoraTrack:LArPandoraTrackCreation            0.430251       1.01103       1.71933      0.892293      0.355964        10     
reco:pandoraShower:LArPandoraShowerCreation          0.431674       1.00625       1.84255      0.910259      0.375984        10     
reco:pandoracalo:Calorimetry                         0.240671       0.55147      0.885915       0.51769      0.181491        10     
reco:pandoracalonosce:Calorimetry                    0.237314      0.544507      0.860706      0.516398      0.176506        10     
reco:pandoraShowercalo:ShowerCalorimetry             0.454402       1.03803       2.01014      0.948866      0.393835        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.440219      0.988782       1.95665      0.903406      0.384602        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0204476     0.0315253     0.0501541     0.0308486    0.00721192       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0201698     0.0304706     0.043946      0.0299701    0.00595878       10     
reco:emtrkmichelid:EmTrackMichelId                    13.7957       20.2348       27.737        19.6642       3.31048        10     
reco:crttag:SingleCRTMatchingProducer               0.00253372    0.00355189    0.00626745    0.00328306    0.000994635      10     
reco:crtreco:TwoCRTMatchingProducer                 0.00189908    0.00282746    0.00391385    0.00279677    0.000480135      10     
reco:pandora2Track:LArPandoraTrackCreation            1.06695       2.50954       4.5553        2.1959       0.928655        10     
reco:pandora2calo:ShowerCalorimetry                  0.403891       0.91429       1.84243      0.827105      0.364414        10     
reco:pandora2calonosce:ShowerCalorimetry             0.393388      0.895808       1.82898      0.796033      0.364296        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0207784     0.0310839     0.0450853     0.0300851    0.00580828       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0198044     0.0299641     0.0427057     0.0293082     0.0055162       10     
reco:pandora2Shower:LArPandoraShowerCreation         0.492781       1.12666       1.99838      0.988458      0.410245        10     
reco:pandora2Showercalo:ShowerCalorimetry            0.543349       1.23892       2.42961       1.06393      0.481355        10     
reco:pandora2Showercalonosce:ShowerCalorimetry       0.532251       1.21508       2.39174       1.0378       0.472842        10     
[art]:TriggerResults:TriggerResultInserter          1.8556e-05    2.96969e-05   8.5644e-05    2.3961e-05    1.87841e-05      10     
end_path:out1:RootOutput                             6.763e-06    9.9221e-06    2.7533e-05    7.9405e-06    5.91163e-06      10     
end_path:out1:RootOutput(write)                       1.00755       1.41529       1.89622       1.39856      0.210062        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6199.17 MB
  Peak resident set size usage (VmHWM): 3998.7 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11347856
-rw-r--r-- 1 dunegli fnalgrid      38548 Oct 29 20:38 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        201 Oct 29 20:38 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:38 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Oct 29 20:38 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     200704 Oct 29 20:38 mem.db
-rw-r--r-- 1 dunegli fnalgrid      36864 Oct 29 20:38 time.db
-rw-r--r-- 1 dunegli fnalgrid 5951814991 Oct 29 20:38 pdhd_prod_beam__226242_25_1_20251008T182736Z_gen_g4_IonScintPDExt.root_227038_137_1_20251010T164623Z_PDInt.root_234970_30_1_20251029T193803Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   50364950 Oct 29 20:37 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:25 ProtoDUNE.log
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:23 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:23 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 5617709385 Oct 29 20:23 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        122 Oct 29 19:38 all-input-dids.txt
justIN time: 2025-11-03 18:32:35 UTC       justIN version: 01.05.01