justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 234970.33@dunegpschedd02.fnal.gov

Jobsub ID234970.33@dunegpschedd02.fnal.gov
Workflow ID9454
Stage ID1
User namemwrobel@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-29 18:56:46
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-10-29 20:44:21
From worker nodeHostnamedunegli-7568266-0-fnpc22003.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-10-29 19:37:54
Input filesusertests:pdhd_prod_beam__226242_24_1_20251008T182741Z_gen_g4_IonScintPDExt.root_227038_136_1_20251010T164558Z_PDInt.root
JobscriptExit code0
Real time59m (3584s)
CPU time58m (3519s = 98%)
Max RSS bytes7344517120 (7004 MiB)
Outputting started2025-10-29 20:37:39
Output files
Finished2025-10-29 20:44:21
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

..
334252 tests to find 178008 neighbours
Iterating with no regularization...
Begin: 2.92517e+07
0 2.65253e+07
1 2.62776e+07
2 2.62446e+07
3 2.62341e+07
Now with regularization...
Begin: 2.46377e+07
0 2.46304e+07
Begin processing the 9th record. run: 20250627 subRun: 1 event: 88 at 29-Oct-2025 20:33:41 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 12228!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 88066 traces, 88066 summary
Retagger: tagging trace set: gauss with 70431 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 70431 traces tagged "gauss"
FrameSaver: q=2.0026e+06 n=1059368 tag=gauss
wclsFrameSaver: saving 88066 traces tagged "wiener"
FrameSaver: q=2.62421e+06 n=1016686 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 11378 XUs and 8353 XVs -> 119 XUVs
C:0 T:1 64991 XUs and 74835 XVs -> 4774 XUVs
C:0 T:2 21217 XUs and 64671 XVs -> 4963 XUVs
C:0 T:3 37 XUs and 170 XVs -> 4 XUVs
C:0 T:5 34680 XUs and 24906 XVs -> 607 XUVs
C:0 T:6 5830 XUs and 42298 XVs -> 764 XUVs
C:0 T:7 5 XUs and 59 XVs -> 0 XUVs
11231 XUVs total
1899 collection wire objects
11231 potential space points
Neighbour search...
1231089 tests to find 672288 neighbours
Iterating with no regularization...
Begin: 4.51886e+07
0 4.0836e+07
1 4.05951e+07
2 4.05499e+07
3 4.0533e+07
Now with regularization...
Begin: 3.94591e+07
0 3.94461e+07
Begin processing the 10th record. run: 20250627 subRun: 1 event: 89 at 29-Oct-2025 20:35:03 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 22415!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 122795 traces, 122795 summary
Retagger: tagging trace set: gauss with 98051 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 98051 traces tagged "gauss"
FrameSaver: q=3.55175e+06 n=1671610 tag=gauss
wclsFrameSaver: saving 122795 traces tagged "wiener"
FrameSaver: q=4.38595e+06 n=1607968 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 19989 XUs and 17353 XVs -> 307 XUVs
C:0 T:1 188322 XUs and 180149 XVs -> 7650 XUVs
C:0 T:2 41788 XUs and 89820 XVs -> 2232 XUVs
C:0 T:3 5 XUs and 12 XVs -> 0 XUVs
C:0 T:4 13 XUs and 9 XVs -> 0 XUVs
C:0 T:5 48620 XUs and 34526 XVs -> 1259 XUVs
C:0 T:6 29128 XUs and 44390 XVs -> 776 XUVs
C:0 T:7 10 XUs and 12 XVs -> 0 XUVs
12224 XUVs total
5606 collection wire objects
12224 potential space points
Neighbour search...
493940 tests to find 184984 neighbours
Iterating with no regularization...
Begin: 1.06393e+08
0 9.33128e+07
1 9.24411e+07
2 9.23696e+07
Now with regularization...
Begin: 8.88012e+07
0 8.87779e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
29-Oct-2025 20:37:38 UTC  Closed output file "pdhd_prod_beam__226242_24_1_20251008T182741Z_gen_g4_IonScintPDExt.root_227038_136_1_20251010T164558Z_PDInt.root_234970_33_1_20251029T193756Z_detsim_reco1.root"
29-Oct-2025 20:37:38 UTC  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            62.1454       84.274        150.467       80.2357       23.011         10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.000386462   0.00332302    0.00549104    0.00338561    0.00172705       10     
reco:opdec:Deconvolution                             0.432343      0.574229       0.86755      0.544124      0.109546        10     
reco:ophitspe:OpHitFinderDeco                        0.0973945     0.103546      0.116789      0.102592     0.00491564       10     
reco:opflash:OpFlashFinder                           0.0395118     0.065152      0.136037      0.061796      0.0261999       10     
reco:opslicer:OpSlicer                               0.124444      0.159891      0.281033      0.143613      0.0450432       10     
reco:wclsdatahd:WireCellToolkit                       31.9138       32.6415       33.6536       32.7532      0.529496        10     
reco:gaushit:GausHitFinder                           0.359422       0.47278      0.757024       0.45276      0.102088        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000174892   0.00023716    0.000332099   0.000217798   4.90243e-05      10     
reco:reco3d:SpacePointSolver                         0.159668      0.567228       1.21522      0.509267      0.287829        10     
reco:hitpdune:DisambigFromSpacePoints                0.145475       0.43796       1.12263       0.36774      0.262878        10     
reco:pandora:StandardPandora                          5.75541       13.7588       44.8219       10.581        10.7876        10     
reco:pandoraWriter:StandardPandora                   0.0950866     0.151408      0.280007      0.140157      0.0459029       10     
reco:pandoraTrack:LArPandoraTrackCreation            0.546563       1.04153       2.3617       0.944135      0.498682        10     
reco:pandoraShower:LArPandoraShowerCreation          0.522792      0.986496       2.68658      0.787614      0.600151        10     
reco:pandoracalo:Calorimetry                         0.314126      0.577757       1.25107      0.537334      0.253016        10     
reco:pandoracalonosce:Calorimetry                    0.310296      0.566548       1.20315      0.530626      0.241175        10     
reco:pandoraShowercalo:ShowerCalorimetry             0.501944      0.979172       2.69859      0.798383      0.600095        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.473404      0.932881       2.54369      0.763492      0.563128        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0229395     0.0333666     0.0460732     0.0324415    0.00630007       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0223428     0.0321487     0.0467011     0.0314763      0.00672        10     
reco:emtrkmichelid:EmTrackMichelId                    15.0904       20.8126       31.6764       19.9274       4.06997        10     
reco:crttag:SingleCRTMatchingProducer                0.0023868    0.00355477    0.00643915    0.00324392    0.00106608       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00217994    0.00297717    0.00535077    0.00273339    0.000848021      10     
reco:pandora2Track:LArPandoraTrackCreation            1.28313       2.4904        6.21453       2.1267        1.33929        10     
reco:pandora2calo:ShowerCalorimetry                  0.430455      0.872861       2.40407      0.727535      0.536863        10     
reco:pandora2calonosce:ShowerCalorimetry             0.421451      0.862449       2.39229      0.717066      0.535504        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0226344     0.0327999     0.0444531     0.0320432    0.00628965       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0220141     0.0313685     0.0441939     0.0308176    0.00634625       10     
reco:pandora2Shower:LArPandoraShowerCreation         0.576227       1.13113       2.69452      0.973447      0.574063        10     
reco:pandora2Showercalo:ShowerCalorimetry            0.601566       1.23237       3.25501       1.05923      0.713836        10     
reco:pandora2Showercalonosce:ShowerCalorimetry       0.585648       1.20198       3.17259       1.02512      0.696117        10     
[art]:TriggerResults:TriggerResultInserter          1.4908e-05    2.25277e-05   5.7649e-05    1.85655e-05   1.19634e-05      10     
end_path:out1:RootOutput                             2.875e-06    9.7455e-06    5.8551e-05    4.0525e-06    1.63208e-05      10     
end_path:out1:RootOutput(write)                       1.11909       1.45428       2.33705       1.34222      0.322877        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6410.91 MB
  Peak resident set size usage (VmHWM): 4216.91 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11267928
-rw-r--r-- 1 dunegli fnalgrid      38978 Oct 29 20:37 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        201 Oct 29 20:37 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:37 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Oct 29 20:37 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     200704 Oct 29 20:37 mem.db
-rw-r--r-- 1 dunegli fnalgrid      36864 Oct 29 20:37 time.db
-rw-r--r-- 1 dunegli fnalgrid 5910508911 Oct 29 20:37 pdhd_prod_beam__226242_24_1_20251008T182741Z_gen_g4_IonScintPDExt.root_227038_136_1_20251010T164558Z_PDInt.root_234970_33_1_20251029T193756Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   51043650 Oct 29 20:36 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:23 ProtoDUNE.log
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:22 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:22 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 5576501351 Oct 29 20:22 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        122 Oct 29 19:37 all-input-dids.txt
justIN time: 2025-11-05 04:48:29 UTC       justIN version: 01.05.01