justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 243139.1@dunegpschedd01.fnal.gov

Jobsub ID243139.1@dunegpschedd01.fnal.gov
Workflow ID9567
Stage ID1
User namemwrobel@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-04 19:36:28
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-11-04 20:32:43
From worker nodeHostnamedunegli-7630197-0-fnpc22033.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-04 19:37:17
Input filesusertests:pdhd_prod_beam__226470_47_1_20251008T212526Z_gen_g4_IonScintPDExt.root_227040_75_1_20251010T170233Z_PDInt.root
JobscriptExit code0
Real time54m (3277s)
CPU time50m (3025s = 92%)
Max RSS bytes6482284544 (6181 MiB)
Outputting started2025-11-04 20:31:55
Output filesusertests:pdhd_prod_beam__226470_47_1_20251008T212526Z_gen_g4_IonScintPDExt.root_227040_75_1_20251010T170233Z_PDInt.root_243139_1_1_20251104T193721Z_detsim_reco1.root
Finished2025-11-04 20:32:43
Saved logsjustin-logs:243139.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

> call=18 input 0: frame: ident=99 time=0 tick=512 with 40372 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":19231 [0] "wiener0":21141 [21141] ] cmm:[ bad:0 lf_noisy:127 ] input 1: frame: ident=99 time=0 tick=512 with 37975 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":16971 [0] "wiener1":21004 [21004] ] cmm:[ bad:0 ] input 2: frame: ident=99 time=0 tick=512 with 56661 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":23755 [0] "wiener2":32906 [32906] ] cmm:[ bad:0 ] input 3: frame: ident=99 time=0 tick=512 with 33459 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":14877 [0] "wiener3":18582 [18582] ] cmm:[ bad:0 ] output: frame: ident=99 time=0 tick=512 with 168467 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":19231 [0] "gauss1":16971 [0] "gauss2":23755 [0] "gauss3":14877 [0] "wiener0":21141 [21141] "wiener1":21004 [21004] "wiener2":32906 [32906] "wiener3":18582 [18582] ] cmm:[ bad:0 lf_noisy:127 ] 
Retagger: tagging trace set: wiener with 93633 traces, 93633 summary
Retagger: tagging trace set: gauss with 74834 traces, 0 summary
[20:31:27.761] D [  glue  ] sink frame: #99 @0 with 168467 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[20:31:27.761] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[20:31:27.761] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[20:31:27.761] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[20:31:27.761] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[20:31:27.761] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[20:31:27.761] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[20:31:27.761] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[20:31:27.761] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[20:31:27.761] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[20:31:27.761] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[20:31:27.761] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[20:31:27.761] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[20:31:27.761] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[20:31:27.761] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[20:31:27.761] D [  glue  ] frame sink sees EOS
[20:31:27.761] D [ pgraph ] <Pgrapher:> graph execution complete 
[20:31:27.761] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.73 sec
[20:31:27.761] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.46 sec
[20:31:27.761] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.3 sec
[20:31:27.761] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.16 sec
[20:31:27.761] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.91 sec
[20:31:27.761] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.89 sec
[20:31:27.761] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.89 sec
[20:31:27.761] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.89 sec
[20:31:27.761] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[20:31:27.761] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:31:27.761] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:31:27.761] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:31:27.761] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:31:27.761] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[20:31:27.761] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[20:31:27.761] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[20:31:27.761] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[20:31:27.761] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[20:31:27.761] I [ timer  ] Timer: Total node execution : 31.24000001884997 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 74834 traces tagged "gauss"
FrameSaver: q=2.18748e+06 n=1195592 tag=gauss
wclsFrameSaver: saving 93633 traces tagged "wiener"
FrameSaver: q=2.6679e+06 n=1147101 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 13883 XUs and 17254 XVs -> 479 XUVs
C:0 T:3 15 XUs and 7 XVs -> 0 XUVs
C:0 T:4 21 XUs and 40 XVs -> 13 XUVs
C:0 T:5 61705 XUs and 65462 XVs -> 3540 XUVs
C:0 T:6 8976 XUs and 12884 XVs -> 454 XUVs
C:0 T:7 4 XUs and 6 XVs -> 0 XUVs
4486 XUVs total
1953 collection wire objects
4486 potential space points
Neighbour search...
145822 tests to find 82418 neighbours
Iterating with no regularization...
Begin: 1.61924e+07
0 1.45595e+07
1 1.43772e+07
2 1.4351e+07
3 1.43426e+07
Now with regularization...
Begin: 1.26026e+07
0 1.25961e+07
04-Nov-2025 20:31:54 UTC  Closed output file "pdhd_prod_beam__226470_47_1_20251008T212526Z_gen_g4_IonScintPDExt.root_227040_75_1_20251010T170233Z_PDInt.root_243139_1_1_20251104T193721Z_detsim_reco1.root"
04-Nov-2025 20:31:54 UTC  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           44.4266       50.8111       58.0351       50.2234       4.13253        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.000733162   0.00212076    0.00329509     0.0022347    0.000844625      10     
reco:wclsdatahd:WireCellToolkit                      31.4968       31.9628       32.4893       31.9701      0.290179        10     
reco:wirefilter:WireFilter                         0.00654457    0.00885929     0.011441     0.00904591     0.0012824       10     
reco:gaushit:GausHitFinder                          0.241779      0.306888      0.411049      0.296591      0.0481147       10     
reco:nhitsfilter:NumberOfHitsFilter                7.1936e-05    0.000124454   0.000299355   0.000104978   6.17762e-05      10     
reco:reco3d:SpacePointSolver                        0.0691282     0.259282      0.835145      0.201586      0.209051        10     
reco:hitpdune:DisambigFromSpacePoints               0.076538      0.170626      0.301058      0.154064      0.0630741       10     
reco:pandora:StandardPandora                         2.04193       3.3992        6.0318        3.04828       1.28966        10     
reco:pandoraWriter:StandardPandora                  0.0602104     0.0813415     0.123096      0.0771182     0.0177704       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.206557       0.37654      0.620134       0.39356       0.11471        10     
reco:pandoraShower:LArPandoraShowerCreation         0.188176      0.364078      0.610233      0.361884      0.118656        10     
reco:pandoracalo:Calorimetry                        0.126925      0.235172      0.371139      0.245269      0.0684013       10     
reco:pandoracalonosce:Calorimetry                   0.121516      0.226524      0.355608      0.241328      0.0655725       10     
reco:pandoraShowercalo:ShowerCalorimetry            0.179713      0.388421      0.665294      0.392122       0.13636        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.177847      0.383218      0.652544      0.381213      0.134641        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0163968     0.0225357     0.0271662     0.0234092    0.00356027       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0162945      0.02208      0.0264939     0.0231433    0.00330968       10     
reco:emtrkmichelid:EmTrackMichelId                   8.85495       11.9601       14.8949       11.6865       1.88554        10     
[art]:TriggerResults:TriggerResultInserter         1.2223e-05    2.09005e-05    5.279e-05    1.77785e-05   1.09318e-05      10     
end_path:out1:RootOutput                            5.23e-06     1.13293e-05   2.2102e-05     7.639e-06    6.35942e-06      10     
end_path:out1:RootOutput(write)                     0.496774       0.62347       0.74583      0.633217      0.0718512       10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5757.69 MB
  Peak resident set size usage (VmHWM): 3596.02 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 10642636
-rw-r--r-- 1 dunegli fnalgrid     230682 Nov  4 20:31 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        194 Nov  4 20:31 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Nov  4 20:31 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Nov  4 20:31 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     139264 Nov  4 20:31 mem.db
-rw-r--r-- 1 dunegli fnalgrid      28672 Nov  4 20:31 time.db
-rw-r--r-- 1 dunegli fnalgrid 5518087687 Nov  4 20:31 pdhd_prod_beam__226470_47_1_20251008T212526Z_gen_g4_IonScintPDExt.root_227040_75_1_20251010T170233Z_PDInt.root_243139_1_1_20251104T193721Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   30316460 Nov  4 20:31 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Nov  4 20:22 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Nov  4 20:22 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 5349127047 Nov  4 20:22 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        121 Nov  4 19:37 all-input-dids.txt
justIN time: 2025-11-05 12:14:15 UTC       justIN version: 01.05.01