justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 234970.26@dunegpschedd02.fnal.gov

Jobsub ID234970.26@dunegpschedd02.fnal.gov
Workflow ID9454
Stage ID1
User namemwrobel@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-29 18:56:46
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-10-29 20:52:20
From worker nodeHostnamedunegli-7568340-0-fnpc23018.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-10-29 19:38:03
Input filesusertests:pdhd_prod_beam__231691_43_1_20251008T183105Z_gen_g4_IonScintPDExt.root_227038_130_1_20251010T164127Z_PDInt.root
JobscriptExit code0
Real time1h (4054s)
CPU time1h (3878s = 95%)
Max RSS bytes7831646208 (7468 MiB)
Outputting started2025-10-29 20:45:38
Output files
Finished2025-10-29 20:52:20
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

0 0 0 0 0 0 0 0 0 0 0 
----view 1: 0.00223255 0.00838542 0.0173388 0.0494995 0.0441589 0.0390549 0.0439377 0.0283051 0.0325165 0.0214233 0.019371 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0 0 0 0 0 0 0 0 0 0 0 
----view 1: 0.00838542 0.0173388 0.0494995 0.0441589 0.0390549 0.0439377 0.0283051 0.0325165 0.0214233 0.019371 0.0426254 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0 0 0 0 0 0 0 0 0 0 0 0 0 
----view 1: 0.00600886 0.000376463 0.000376701 0.000376463 0.00223255 0.00838542 0.0173388 0.0494995 0.0441589 0.0390549 0.0439377 0.0283051 0.0325165 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0 0 0 0 0 0 0 0 0 0 0 0 0 
----view 1: 0.000376463 0.000376701 0.000376463 0.00223255 0.00838542 0.0173388 0.0494995 0.0441589 0.0390549 0.0439377 0.0283051 0.0325165 0.0214233 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0 0 0 0 0 0 0 0 0 0 0 0 0 
----view 1: 0.000376701 0.000376463 0.00223255 0.00838542 0.0173388 0.0494995 0.0441589 0.0390549 0.0439377 0.0283051 0.0325165 0.0214233 0.019371 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0 0 0 0 0 0 0 0 0 0 0 0 0 
----view 1: 0.000376463 0.00223255 0.00838542 0.0173388 0.0494995 0.0441589 0.0390549 0.0439377 0.0283051 0.0325165 0.0214233 0.019371 0.0426254 
Begin processing the 10th record. run: 20250627 subRun: 1 event: 89 at 29-Oct-2025 20:44:37 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 10704!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 63929 traces, 63929 summary
Retagger: tagging trace set: gauss with 54784 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 54784 traces tagged "gauss"
FrameSaver: q=1.29797e+06 n=793352 tag=gauss
wclsFrameSaver: saving 63929 traces tagged "wiener"
FrameSaver: q=1.64623e+06 n=784230 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 7623 XUs and 6360 XVs -> 61 XUVs
C:0 T:1 37043 XUs and 32714 XVs -> 744 XUVs
C:0 T:2 3725 XUs and 3674 XVs -> 276 XUVs
C:0 T:5 7490 XUs and 7741 XVs -> 419 XUVs
C:0 T:6 8540 XUs and 5053 XVs -> 92 XUVs
C:0 T:7 8 XUs and 12 XVs -> 0 XUVs
1592 XUVs total
1121 collection wire objects
1592 potential space points
Neighbour search...
17382 tests to find 8152 neighbours
Iterating with no regularization...
Begin: 1.4467e+07
0 1.37586e+07
1 1.37422e+07
2 1.37413e+07
Now with regularization...
Begin: 1.3096e+07
0 1.30957e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
29-Oct-2025 20:45:37 UTC  Closed output file "pdhd_prod_beam__231691_43_1_20251008T183105Z_gen_g4_IonScintPDExt.root_227038_130_1_20251010T164127Z_PDInt.root_234970_26_1_20251029T193805Z_detsim_reco1.root"
29-Oct-2025 20:45:37 UTC  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            57.7127       90.2518       126.691       93.6056       17.1496        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.000630255   0.00472812    0.00718494    0.00556307    0.00234285       10     
reco:opdec:Deconvolution                             0.443777      0.626651      0.769573      0.649239      0.100932        10     
reco:ophitspe:OpHitFinderDeco                        0.0972385     0.106476      0.113878      0.107362     0.00501685       10     
reco:opflash:OpFlashFinder                           0.038615      0.0820675     0.134032      0.0838218     0.0265098       10     
reco:opslicer:OpSlicer                               0.117623       0.18513      0.240617       0.19507      0.0380485       10     
reco:wclsdatahd:WireCellToolkit                       31.1163       32.3261        33.56        32.3556      0.753247        10     
reco:gaushit:GausHitFinder                           0.296389       0.51991      0.669884      0.554847      0.106879        10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000155757   0.000303507   0.000776033   0.000235681   0.000176453      10     
reco:reco3d:SpacePointSolver                         0.153377      0.648994       1.29834       0.66823      0.312832        10     
reco:hitpdune:DisambigFromSpacePoints                0.107334      0.497501       1.1269       0.479588      0.256168        10     
reco:pandora:StandardPandora                          5.60923       16.3836       36.2508       14.4789       7.85241        10     
reco:pandoraWriter:StandardPandora                   0.0859651     0.166823      0.216359      0.174103      0.0383892       10     
reco:pandoraTrack:LArPandoraTrackCreation            0.499309       1.13772       1.88472       1.11188      0.363207        10     
reco:pandoraShower:LArPandoraShowerCreation          0.463748       1.16204       1.82946       1.14795      0.391951        10     
reco:pandoracalo:Calorimetry                         0.266322      0.633773       1.06856      0.612568        0.207         10     
reco:pandoracalonosce:Calorimetry                    0.255498      0.632282       1.03828      0.635548      0.204112        10     
reco:pandoraShowercalo:ShowerCalorimetry             0.422027       1.27165       2.11914       1.19164      0.483754        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.397232       1.20228       1.98363       1.12292      0.462497        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0176146     0.0346444     0.0496191     0.034709     0.00841022       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0170471     0.0329362     0.0421909     0.0345818    0.00708407       10     
reco:emtrkmichelid:EmTrackMichelId                    13.0965       21.6483       27.0116       22.5768       3.91083        10     
reco:crttag:SingleCRTMatchingProducer               0.00214698    0.00387212    0.00635567    0.00397181    0.00105928       10     
reco:crtreco:TwoCRTMatchingProducer                 0.00167476    0.00290361    0.00378104    0.00311856    0.000573132      10     
reco:pandora2Track:LArPandoraTrackCreation            1.06378       2.85625       4.3287        2.80246      0.920732        10     
reco:pandora2calo:ShowerCalorimetry                  0.340416       1.09424       1.7911        1.00428      0.431182        10     
reco:pandora2calonosce:ShowerCalorimetry             0.332623       1.08341       1.76194       0.98461      0.432555        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0179584     0.0341565     0.0436072     0.0346645    0.00735199       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.017007      0.0331613     0.0438043     0.0338355    0.00755247       10     
reco:pandora2Shower:LArPandoraShowerCreation          0.49575       1.29765       1.97728       1.27293       0.4383         10     
reco:pandora2Showercalo:ShowerCalorimetry            0.472289       1.46649       2.45483        1.35        0.564789        10     
reco:pandora2Showercalonosce:ShowerCalorimetry        0.45636       1.43622       2.39628       1.30818      0.559349        10     
[art]:TriggerResults:TriggerResultInserter          1.8406e-05    2.96919e-05   6.5656e-05    2.6626e-05    1.28811e-05      10     
end_path:out1:RootOutput                             4.208e-06    9.2549e-06    2.2353e-05    8.0055e-06    4.55813e-06      10     
end_path:out1:RootOutput(write)                      0.965758       1.56251       2.18757       1.56345      0.311724        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6528.5 MB
  Peak resident set size usage (VmHWM): 4323.38 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12688696
-rw-r--r-- 1 dunegli fnalgrid      45298 Oct 29 20:45 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        201 Oct 29 20:45 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:45 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Oct 29 20:45 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     200704 Oct 29 20:45 mem.db
-rw-r--r-- 1 dunegli fnalgrid      36864 Oct 29 20:45 time.db
-rw-r--r-- 1 dunegli fnalgrid 6648820981 Oct 29 20:45 pdhd_prod_beam__231691_43_1_20251008T183105Z_gen_g4_IonScintPDExt.root_227038_130_1_20251010T164127Z_PDInt.root_234970_26_1_20251029T193805Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   53335170 Oct 29 20:45 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:31 ProtoDUNE.log
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:29 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:29 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 6290750946 Oct 29 20:29 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        122 Oct 29 19:38 all-input-dids.txt
justIN time: 2025-11-04 01:29:51 UTC       justIN version: 01.05.01