justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 236805.80@dunegpschedd02.fnal.gov

Jobsub ID236805.80@dunegpschedd02.fnal.gov
Workflow ID9567
Stage ID1
User namemwrobel@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-03 18:19:10
SiteUS_Wisconsin
EntryHCCHTPC_US_Wisconsin_osg01_rhel7
Last heartbeat2025-11-03 20:53:38
From worker nodeHostnamee4038
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit82800 (23 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-03 19:16:16
Input filesusertests:pdhd_prod_beam__226470_35_1_20251008T212509Z_gen_g4_IonScintPDExt.root_227040_183_1_20251010T172347Z_PDInt.root
JobscriptExit code0
Real time1h (5661s)
CPU time1h (5317s = 93%)
Max RSS bytes9120116736 (8697 MiB)
Outputting started2025-11-03 20:50:38
Output filesusertests:pdhd_prod_beam__226470_35_1_20251008T212509Z_gen_g4_IonScintPDExt.root_227040_183_1_20251010T172347Z_PDInt.root_236805_80_1_20251103T191625Z_detsim_reco1.root
Finished2025-11-03 20:53:38
Saved logsjustin-logs:236805.80-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":23707 [0] "gauss1":18395 [0] "gauss2":17520 [0] "gauss3":17456 [0] "wiener0":26523 [26523] "wiener1":26847 [26847] "wiener2":23207 [23207] "wiener3":23765 [23765] ] cmm:[ bad:0 lf_noisy:49 ] 
Retagger: tagging trace set: wiener with 100342 traces, 100342 summary
Retagger: tagging trace set: gauss with 77078 traces, 0 summary
[14:49:46.213] D [  glue  ] sink frame: #99 @0 with 177420 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[14:49:46.213] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[14:49:46.213] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[14:49:46.213] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[14:49:46.213] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[14:49:46.213] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[14:49:46.213] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[14:49:46.213] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[14:49:46.213] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[14:49:46.213] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[14:49:46.213] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[14:49:46.213] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[14:49:46.213] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[14:49:46.213] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[14:49:46.213] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[14:49:46.213] D [  glue  ] frame sink sees EOS
[14:49:46.213] D [ pgraph ] <Pgrapher:> graph execution complete 
[14:49:46.213] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 14.58 sec
[14:49:46.213] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.44 sec
[14:49:46.213] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.39 sec
[14:49:46.213] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.15 sec
[14:49:46.213] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.37 sec
[14:49:46.213] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.33 sec
[14:49:46.213] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.28 sec
[14:49:46.213] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.2 sec
[14:49:46.213] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[14:49:46.213] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[14:49:46.213] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:49:46.213] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:49:46.213] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:49:46.213] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[14:49:46.213] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[14:49:46.213] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[14:49:46.213] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[14:49:46.213] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[14:49:46.213] I [ timer  ] Timer: Total node execution : 58.75999929383397 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 77078 traces tagged "gauss"
FrameSaver: q=2.39132e+06 n=1168996 tag=gauss
wclsFrameSaver: saving 100342 traces tagged "wiener"
FrameSaver: q=3.00415e+06 n=1124314 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 46246 XUs and 49748 XVs -> 615 XUVs
C:0 T:5 28930 XUs and 17421 XVs -> 642 XUVs
C:0 T:6 27544 XUs and 63442 XVs -> 946 XUVs
C:0 T:7 12 XUs and 26 XVs -> 2 XUVs
2205 XUVs total
1205 collection wire objects
2205 potential space points
Neighbour search...
37265 tests to find 22916 neighbours
Iterating with no regularization...
Begin: 1.70987e+07
0 1.60801e+07
1 1.60105e+07
2 1.60025e+07
Now with regularization...
Begin: 1.51288e+07
0 1.51264e+07
this->SetupTestBeamPfo(pNuPfo, pTestBeamPfo, testBeamStartVertex) return STATUS_CODE_NOT_FOUND
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_16_01-buildFW/src/larpandoracontent/LArThreeDReco/LArEventBuilding/TestBeamParticleCreationAlgorithm.cc line#: 43
iter->second->Run() throw STATUS_CODE_NOT_FOUND
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0087, LArTestBeamParticleCreation, STATUS_CODE_NOT_FOUND
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
03-Nov-2025 14:50:35 CST  Closed output file "pdhd_prod_beam__226470_35_1_20251008T212509Z_gen_g4_IonScintPDExt.root_227040_183_1_20251010T172347Z_PDInt.root_236805_80_1_20251103T191625Z_detsim_reco1.root"
03-Nov-2025 14:50:35 CST  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           92.6043       113.66        157.128       109.886       18.338         10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.00102631    0.00608153     0.0139207    0.00619134    0.00334971       10     
reco:wclsdatahd:WireCellToolkit                      53.335        58.8001       68.1748       57.6533       4.35343        10     
reco:wirefilter:WireFilter                          0.0140349     0.0232013     0.0294863     0.0226507    0.00470844       10     
reco:gaushit:GausHitFinder                          0.528862      0.809114       1.23844      0.794339      0.203152        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000234448   0.000474728   0.000807179   0.000458723   0.000194349      10     
reco:reco3d:SpacePointSolver                        0.349515      0.874966       1.58639      0.771777      0.373863        10     
reco:hitpdune:DisambigFromSpacePoints                0.32473      0.613031       1.46452      0.544653      0.329026        10     
reco:pandora:StandardPandora                         6.32977       12.1903       29.1884       10.3427       6.56055        10     
reco:pandoraWriter:StandardPandora                  0.215411      0.293727      0.503046      0.257573      0.0885881       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.715434       1.19041       2.59203      0.885876      0.588292        10     
reco:pandoraShower:LArPandoraShowerCreation         0.609785       1.2642        2.90249       1.05322      0.684932        10     
reco:pandoracalo:Calorimetry                        0.491167      0.741274       1.49534      0.672304      0.293198        10     
reco:pandoracalonosce:Calorimetry                   0.463146      0.715148       1.49217      0.617892      0.303103        10     
reco:pandoraShowercalo:ShowerCalorimetry             0.63376       1.27214       2.77166       1.08367      0.640878        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.619602       1.24782       2.6842        1.07247      0.619863        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0465079     0.0707362     0.0987969     0.068782      0.017024        10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0372406     0.0684271     0.104474      0.0653235     0.0193948       10     
reco:emtrkmichelid:EmTrackMichelId                    23.81        31.6671       42.0532       30.6173       5.28679        10     
[art]:TriggerResults:TriggerResultInserter         2.5938e-05    5.48104e-05   0.000121056   4.8761e-05    2.60296e-05      10     
end_path:out1:RootOutput                            9.678e-06    1.39339e-05   3.7681e-05    1.04145e-05   8.09916e-06      10     
end_path:out1:RootOutput(write)                      1.30068       1.74068       2.4732        1.66741       0.33817        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6153.83 MB
  Peak resident set size usage (VmHWM): 3953.5 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 14399776
-rw-r--r-- 1 slot1_26 slot1_26     232047 Nov  3 14:50 jobscript.log
-rw-r--r-- 1 slot1_26 slot1_26        201 Nov  3 14:50 justin-processed-pfns.txt
-rw-r--r-- 1 slot1_26 slot1_26        519 Nov  3 14:50 reco_protodunehd_hist.root
-rw-r--r-- 1 slot1_26 slot1_26       3574 Nov  3 14:50 Pandora_Geometry.xml
-rw-r--r-- 1 slot1_26 slot1_26     139264 Nov  3 14:50 mem.db
-rw-r--r-- 1 slot1_26 slot1_26      28672 Nov  3 14:50 time.db
-rw-r--r-- 1 slot1_26 slot1_26 7459733889 Nov  3 14:50 pdhd_prod_beam__226470_35_1_20251008T212509Z_gen_g4_IonScintPDExt.root_227040_183_1_20251010T172347Z_PDInt.root_236805_80_1_20251103T191625Z_detsim_reco1.root
-rw-r--r-- 1 slot1_26 slot1_26   40703430 Nov  3 14:49 Pandora_Events.pndr
-rw-r--r-- 1 slot1_26 slot1_26          0 Nov  3 14:28 debugprod.log
-rw-r--r-- 1 slot1_26 slot1_26        519 Nov  3 14:28 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 slot1_26 slot1_26 7244482510 Nov  3 14:28 temp_detsim.root
-rw-r--r-- 1 slot1_26 slot1_26        122 Nov  3 13:16 all-input-dids.txt
justIN time: 2025-11-05 07:23:23 UTC       justIN version: 01.05.01