justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 234970.21@dunegpschedd02.fnal.gov

Jobsub ID234970.21@dunegpschedd02.fnal.gov
Workflow ID9454
Stage ID1
User namemwrobel@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-29 18:56:46
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-10-29 20:43:11
From worker nodeHostnamedunegli-7568311-0-fnpc23022.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-10-29 19:37:19
Input filesusertests:pdhd_prod_beam__231691_5_1_20251008T183040Z_gen_g4_IonScintPDExt.root_227038_87_1_20251010T163642Z_PDInt.root
JobscriptExit code0
Real time59m (3545s)
CPU time57m (3468s = 97%)
Max RSS bytes6533345280 (6230 MiB)
Outputting started2025-10-29 20:36:25
Output files
Finished2025-10-29 20:43:11
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

aver: q=2.52111e+06 n=1240024 tag=gauss
wclsFrameSaver: saving 104274 traces tagged "wiener"
FrameSaver: q=3.08454e+06 n=1177260 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 8208 XUs and 9630 XVs -> 68 XUVs
C:0 T:1 41644 XUs and 53554 XVs -> 847 XUVs
C:0 T:2 47013 XUs and 66833 XVs -> 5906 XUVs
C:0 T:3 10 XUs and 48 XVs -> 0 XUVs
C:0 T:5 22751 XUs and 42500 XVs -> 1544 XUVs
C:0 T:6 45428 XUs and 45214 XVs -> 1970 XUVs
C:0 T:7 42 XUs and 46 XVs -> 7 XUVs
10342 XUVs total
2504 collection wire objects
10342 potential space points
Neighbour search...
773228 tests to find 469270 neighbours
Iterating with no regularization...
Begin: 2.95656e+07
0 2.69385e+07
1 2.66524e+07
2 2.65886e+07
3 2.65661e+07
Now with regularization...
Begin: 2.4274e+07
0 2.42585e+07
this->SetupTestBeamPfo(pNuPfo, pTestBeamPfo, testBeamStartVertex) return STATUS_CODE_NOT_FOUND
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_16_01-buildFW/src/larpandoracontent/LArThreeDReco/LArEventBuilding/TestBeamParticleCreationAlgorithm.cc line#: 43
iter->second->Run() throw STATUS_CODE_NOT_FOUND
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0087, LArTestBeamParticleCreation, STATUS_CODE_NOT_FOUND
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
Begin processing the 10th record. run: 20250627 subRun: 1 event: 89 at 29-Oct-2025 20:34:47 UTC

Running Ophitfinder with InputDigiType = 'recob'
Found hits: 14374!
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=6000 keeping as is
Retagger: tagging trace set: wiener with 98784 traces, 98784 summary
Retagger: tagging trace set: gauss with 78550 traces, 0 summary
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 78550 traces tagged "gauss"
FrameSaver: q=2.45898e+06 n=1223892 tag=gauss
wclsFrameSaver: saving 98784 traces tagged "wiener"
FrameSaver: q=3.29396e+06 n=1194621 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:0 12990 XUs and 10972 XVs -> 194 XUVs
C:0 T:1 126539 XUs and 106013 XVs -> 4372 XUVs
C:0 T:2 19848 XUs and 22782 XVs -> 928 XUVs
C:0 T:3 2 XUs and 5 XVs -> 0 XUVs
C:0 T:5 8774 XUs and 12238 XVs -> 300 XUVs
C:0 T:6 30936 XUs and 33992 XVs -> 1118 XUVs
C:0 T:7 26 XUs and 40 XVs -> 2 XUVs
6914 XUVs total
3051 collection wire objects
6914 potential space points
Neighbour search...
228970 tests to find 96550 neighbours
Iterating with no regularization...
Begin: 5.1118e+07
0 4.64309e+07
1 4.60958e+07
2 4.60767e+07
Now with regularization...
Begin: 4.41324e+07
0 4.41277e+07
29-Oct-2025 20:36:24 UTC  Closed output file "pdhd_prod_beam__231691_5_1_20251008T183040Z_gen_g4_IonScintPDExt.root_227038_87_1_20251010T163642Z_PDInt.root_234970_21_1_20251029T193721Z_detsim_reco1.root"
29-Oct-2025 20:36:24 UTC  Closed input file "temp_detsim.root"

======================================================================================================================================
TimeTracker printout (sec)                              Min           Avg           Max         Median          RMS         nEvts   
======================================================================================================================================
Full event                                            68.5637       84.3406       114.427       83.6355       13.2924        10     
--------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.000373422   0.00389651     0.0072967    0.00362531    0.00212105       10     
reco:opdec:Deconvolution                             0.499023      0.581235      0.727255      0.570696      0.0724609       10     
reco:ophitspe:OpHitFinderDeco                        0.100998      0.105913      0.110581      0.105235     0.00342583       10     
reco:opflash:OpFlashFinder                           0.0477685     0.0653518     0.0993628     0.0587792     0.0155325       10     
reco:opslicer:OpSlicer                               0.119917      0.159552      0.213113       0.15645      0.0297896       10     
reco:wclsdatahd:WireCellToolkit                       32.4692       33.0254       33.6509       32.8918      0.461283        10     
reco:gaushit:GausHitFinder                           0.365441      0.475312      0.593213      0.490952      0.0787553       10     
reco:nhitsfilter:NumberOfHitsFilter                 0.000183961   0.000262606   0.000426563   0.000266077   6.31623e-05      10     
reco:reco3d:SpacePointSolver                         0.240204      0.492607      0.659768      0.526879      0.138283        10     
reco:hitpdune:DisambigFromSpacePoints                0.227307      0.409653      0.554758      0.425241      0.128885        10     
reco:pandora:StandardPandora                          6.71054       12.8178       27.1587       11.2934       5.74097        10     
reco:pandoraWriter:StandardPandora                   0.116002      0.151509      0.189272      0.156147      0.0250039       10     
reco:pandoraTrack:LArPandoraTrackCreation            0.618272      0.939976       1.37928      0.928992       0.2445         10     
reco:pandoraShower:LArPandoraShowerCreation          0.629833       1.04364       1.74307      0.959661      0.340017        10     
reco:pandoracalo:Calorimetry                         0.360049      0.537151      0.870098      0.514714      0.151024        10     
reco:pandoracalonosce:Calorimetry                    0.351808      0.528526       0.85263      0.508007      0.147574        10     
reco:pandoraShowercalo:ShowerCalorimetry             0.658453       1.09105       2.05666       1.00483      0.399239        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.620382       1.04088       1.95031      0.947975      0.377718        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry           0.0259861     0.0337354     0.0420462     0.0331139    0.00532667       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry      0.0246379     0.0320234     0.0383187     0.0305946     0.0047399       10     
reco:emtrkmichelid:EmTrackMichelId                    16.7962       20.9765       25.9519       20.3904       3.16174        10     
reco:crttag:SingleCRTMatchingProducer               0.00273838    0.00374146    0.00541011    0.00342146    0.000760337      10     
reco:crtreco:TwoCRTMatchingProducer                 0.00241514    0.00295819    0.00358871    0.00309057    0.000416919      10     
reco:pandora2Track:LArPandoraTrackCreation            1.64805       2.54536       4.18047       2.47296       0.76899        10     
reco:pandora2calo:ShowerCalorimetry                   0.57073      0.987176       1.89498      0.895554      0.373083        10     
reco:pandora2calonosce:ShowerCalorimetry             0.560411      0.967052       1.85959      0.882713      0.367066        10     
reco:pandora2GnocchiCalo:GnocchiCalorimetry          0.0252518     0.0331541     0.0408127     0.0325167    0.00503364       10     
reco:pandora2GnocchiCalonosce:GnocchiCalorimetry     0.0241066     0.0315828     0.038291      0.030662     0.00463656       10     
reco:pandora2Shower:LArPandoraShowerCreation         0.744276       1.16236       1.8864        1.13182       0.35687        10     
reco:pandora2Showercalo:ShowerCalorimetry            0.830378       1.29642       2.40098       1.2753        0.45274        10     
reco:pandora2Showercalonosce:ShowerCalorimetry       0.808667       1.26517       2.33978       1.24922      0.441861        10     
[art]:TriggerResults:TriggerResultInserter          1.6942e-05    2.59093e-05   8.7667e-05    1.9457e-05    2.06438e-05      10     
end_path:out1:RootOutput                             6.332e-06    9.0392e-06    2.6861e-05     7.114e-06     5.963e-06       10     
end_path:out1:RootOutput(write)                       1.21281       1.48325       1.79585       1.45914      0.186562        10     
======================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6156.89 MB
  Peak resident set size usage (VmHWM): 3947.84 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11227676
-rw-r--r-- 1 dunegli fnalgrid      39535 Oct 29 20:36 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        199 Oct 29 20:36 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:36 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Oct 29 20:36 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     200704 Oct 29 20:36 mem.db
-rw-r--r-- 1 dunegli fnalgrid      36864 Oct 29 20:36 time.db
-rw-r--r-- 1 dunegli fnalgrid 5891158292 Oct 29 20:36 pdhd_prod_beam__231691_5_1_20251008T183040Z_gen_g4_IonScintPDExt.root_227038_87_1_20251010T163642Z_PDInt.root_234970_21_1_20251029T193721Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   51165090 Oct 29 20:35 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:23 ProtoDUNE.log
-rw-r--r-- 1 dunegli fnalgrid          0 Oct 29 20:21 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Oct 29 20:21 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 5554504677 Oct 29 20:21 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        120 Oct 29 19:37 all-input-dids.txt
justIN time: 2025-11-03 18:32:48 UTC       justIN version: 01.05.01