justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 234970.9@dunegpschedd02.fnal.gov

Jobsub ID234970.9@dunegpschedd02.fnal.gov
Workflow ID9454
Stage ID1
User namemwrobel@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-29 18:56:46
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-10-29 20:46:30
From worker nodeHostnamedunegli-7568338-0-fnpc23050.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-10-29 19:37:28
Input filesusertests:pdhd_prod_beam__231691_82_1_20251008T183311Z_gen_g4_IonScintPDExt.root_227038_112_1_20251010T163714Z_PDInt.root
JobscriptExit code0
Real time1h (3738s)
CPU time1h (3654s = 97%)
Max RSS bytes6832955392 (6516 MiB)
Outputting started2025-10-29 20:39:47
Output files
Finished2025-10-29 20:46:30
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

'recob'
Found hits: 17541!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 108001 traces, 108001 summary
Retagger: tagging trace set: gauss with 86023 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 86023 traces tagged "gauss"
FrameSaver: q=2.40731e+06 n=1382384 tag=gauss
wclsFrameSaver: saving 108001 traces tagged "wiener"
FrameSaver: q=3.01775e+06 n=1300169 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 16284 XUs and 15972 XVs -> 166 XUVs
C:0 T:1 88850 XUs and 80041 XVs -> 2553 XUVs
C:0 T:2 51383 XUs and 98740 XVs -> 2298 XUVs
C:0 T:4 5 XUs and 3 XVs -> 0 XUVs
C:0 T:5 26298 XUs and 31549 XVs -> 626 XUVs
C:0 T:6 46608 XUs and 41746 XVs -> 972 XUVs
6615 XUVs total
2768 collection wire objects
6615 potential space points
Neighbour search...
320131 tests to find 150830 neighbours
Iterating with no regularization...
Begin: 3.04648e+07
0 2.79422e+07
1 2.7728e+07
2 2.77012e+07
Now with regularization...
Begin: 2.66126e+07
0 2.66026e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Begin processing the 10th record. run: 20250627 subRun: 1 event: 89 at 29-Oct-2025 20:37:48 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 18244!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 112121 traces, 112121 summary
Retagger: tagging trace set: gauss with 87529 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 87529 traces tagged "gauss"
FrameSaver: q=2.70729e+06 n=1358443 tag=gauss
wclsFrameSaver: saving 112121 traces tagged "wiener"
FrameSaver: q=3.48113e+06 n=1297184 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 18920 XUs and 14288 XVs -> 184 XUVs
C:0 T:1 131387 XUs and 96464 XVs -> 4685 XUVs
C:0 T:2 54788 XUs and 78103 XVs -> 1046 XUVs
C:0 T:4 8 XUs and 3 XVs -> 0 XUVs
C:0 T:5 24933 XUs and 18843 XVs -> 983 XUVs
C:0 T:6 81756 XUs and 83771 XVs -> 1422 XUVs
C:0 T:7 11 XUs and 9 XVs -> 0 XUVs
8320 XUVs total
3507 collection wire objects
8320 potential space points
Neighbour search...
423574 tests to find 128156 neighbours
Iterating with no regularization...
Begin: 5.14574e+07
0 4.49841e+07
1 4.45602e+07
2 4.45349e+07
Now with regularization...
Begin: 4.25899e+07
0 4.25834e+07
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.00115013 0.00115204 0.00115013 0.00115204 0.00115013 0.00115204 0.00115108 0.00115108 0.00115108 0.00115013 0.00115204 
----view 1: 0.0181122 0.0424805 0.0425873 0.0251617 0.024292 0.0350952 0.0319214 0.0360718 0.0176849 0.0205231 0.0101624 
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
29-Oct-2025 20:39:46 UTC  Closed output file "pdhd_prod_beam__231691_82_1_20251008T183311Z_gen_g4_IonScintPDExt.root_227038_112_1_20251010T163714Z_PDInt.root_234970_9_1_20251029T193730Z_detsim_reco1.root"
29-Oct-2025 20:39:46 UTC  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            71.6404       88.1838       117.263       82.0927       15.4702        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.000401309   0.00389569    0.00693527    0.00425156    0.00211413       10     
reco:opdec:Deconvolution                             0.498376      0.588252       0.72932      0.568209      0.0796306       10     
reco:ophitspe:OpHitFinderDeco                        0.0997903     0.107592      0.140857      0.104178      0.0115234       10     
reco:opflash:OpFlashFinder                           0.0530071     0.0718994     0.100427      0.0668891     0.0168111       10     
reco:opslicer:OpSlicer                               0.133577       0.1717       0.215767      0.168548      0.0309341       10     
reco:wclsdatahd:WireCellToolkit                       31.4899       32.2945       33.0202       32.2739      0.430587        10     
reco:gaushit:GausHitFinder                           0.372791      0.500385      0.613424      0.488612      0.0736801       10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000169955   0.000234709   0.00040215    0.000216425   6.67623e-05      10     
reco:reco3d:SpacePointSolver                          0.28391      0.631544       1.47836      0.514766      0.341039        10     
reco:hitpdune:DisambigFromSpacePoints                 0.20281      0.424282      0.744107      0.372107      0.174832        10     
reco:pandora:StandardPandora                          9.98575       16.7722       32.9684       12.9165       7.80023        10     
reco:pandoraWriter:StandardPandora                   0.111554      0.151463       0.2004       0.141676      0.0280826       10     
reco:pandoraTrack:LArPandoraTrackCreation             0.58865       1.09261       1.76867      0.970786      0.426918        10     
reco:pandoraShower:LArPandoraShowerCreation          0.630944       1.02116       1.46698      0.908352       0.26062        10     
reco:pandoracalo:Calorimetry                         0.328883      0.610323       1.00962      0.557942      0.234744        10     
reco:pandoracalonosce:Calorimetry                     0.3232       0.600962       0.99229      0.548381      0.231962        10     
reco:pandoraShowercalo:ShowerCalorimetry             0.627637       1.06422       1.66568      0.911375      0.326175        10     
reco:pandoraShowercalonosce:ShowerCalorimetry         0.60556       1.01677       1.60296      0.873999      0.313227        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0230719     0.0317156     0.042037      0.0310575    0.00561037       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0220113     0.030186      0.0366503     0.0302668    0.00475947       10     
reco:emtrkmichelid:EmTrackMichelId                    16.2129       21.0832       26.3217       20.5152       3.23644        10     
reco:crttag:SingleCRTMatchingProducer               0.00259102    0.00362709    0.00640538    0.00341819    0.00104093       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00199754     0.0027892    0.00330851    0.00275569    0.00041491       10     
reco:pandora2Track:LArPandoraTrackCreation            1.53413       2.59737       3.99276       2.41455      0.755518        10     
reco:pandora2calo:ShowerCalorimetry                  0.567357      0.933607       1.51892       0.82546      0.290187        10     
reco:pandora2calonosce:ShowerCalorimetry              0.56011      0.919032       1.49233      0.812968      0.285595        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0230829     0.0314878     0.038355      0.0316624    0.00496153       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0222368     0.0302321     0.036596      0.0302644    0.00467431       10     
reco:pandora2Shower:LArPandoraShowerCreation         0.730858       1.18628       1.73375       1.09505      0.338386        10     
reco:pandora2Showercalo:ShowerCalorimetry             0.77117       1.30817       2.15495       1.14793      0.434869        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        0.75978       1.27679       2.09256       1.11187      0.422407        10     
[art]:TriggerResults:TriggerResultInserter          1.6873e-05    3.23089e-05   0.000117585   2.2778e-05    2.85875e-05      10     
end_path:out1:RootOutput                             5.811e-06    9.9711e-06    3.5839e-05     7.234e-06    8.64144e-06      10     
end_path:out1:RootOutput(write)                       1.19629       1.56791       2.39984       1.52227      0.339061        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6167.92 MB
  Peak resident set size usage (VmHWM): 3962.26 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11623384
-rw-r--r-- 1 dunegli fnalgrid      47553 Oct 29 20:39 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        201 Oct 29 20:39 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:39 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Oct 29 20:39 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     200704 Oct 29 20:39 mem.db
-rw-r--r-- 1 dunegli fnalgrid      36864 Oct 29 20:39 time.db
-rw-r--r-- 1 dunegli fnalgrid 6097592624 Oct 29 20:39 pdhd_prod_beam__231691_82_1_20251008T183311Z_gen_g4_IonScintPDExt.root_227038_112_1_20251010T163714Z_PDInt.root_234970_9_1_20251029T193730Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   52889120 Oct 29 20:38 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:25 ProtoDUNE.log
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:23 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:23 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 5751547703 Oct 29 20:23 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        122 Oct 29 19:37 all-input-dids.txt
justIN time: 2025-11-05 04:48:22 UTC       justIN version: 01.05.01