justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 269118.0@dunegpschedd01.fnal.gov

Jobsub ID269118.0@dunegpschedd01.fnal.gov
Workflow ID11239
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 19:07:05
SiteUS_FNAL-T1
EntryCMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat2025-12-18 21:16:35
From worker nodeHostnamedunegli-49199-0-cmswn2413.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 19:07:22
Input filesusertests:pdhd_prod_beam__267379_161_1_20251212T162445Z_gen_g4_IonScintPDExt.root_262726_84_1_20251215T212639Z_PDInt.root
JobscriptExit code0
Real time2h (7678s)
CPU time1h (6346s = 82%)
Max RSS bytes7550648320 (7200 MiB)
Outputting started2025-12-18 21:15:21
Output filesusertests:pdhd_prod_beam__267379_161_1_20251212T162445Z_gen_g4_IonScintPDExt.root_262726_84_1_20251215T212639Z_PDInt.root_269118_0_1_20251218T190728Z_detsim_reco1.root
Finished2025-12-18 21:16:35
Saved logsjustin-logs:269118.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

anin:sn_mag_nf> call=18 input 0: frame: ident=189 time=0 tick=512 with 34442 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":16315 [0] "wiener0":18127 [18127] ] cmm:[ bad:0 lf_noisy:232 ] input 1: frame: ident=189 time=0 tick=512 with 51614 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":20708 [0] "wiener1":30906 [30906] ] cmm:[ bad:0 ] input 2: frame: ident=189 time=0 tick=512 with 46816 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":20105 [0] "wiener2":26711 [26711] ] cmm:[ bad:0 ] input 3: frame: ident=189 time=0 tick=512 with 43762 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":18061 [0] "wiener3":25701 [25701] ] cmm:[ bad:0 ] output: frame: ident=189 time=0 tick=512 with 176634 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":16315 [0] "gauss1":20708 [0] "gauss2":20105 [0] "gauss3":18061 [0] "wiener0":18127 [18127] "wiener1":30906 [30906] "wiener2":26711 [26711] "wiener3":25701 [25701] ] cmm:[ bad:0 lf_noisy:232 ] 
Retagger: tagging trace set: wiener with 101445 traces, 101445 summary
Retagger: tagging trace set: gauss with 75189 traces, 0 summary
[21:14:26.011] D [  glue  ] sink frame: #189 @0 with 176634 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[21:14:26.012] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[21:14:26.012] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[21:14:26.012] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[21:14:26.012] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[21:14:26.012] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[21:14:26.012] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[21:14:26.012] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[21:14:26.012] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[21:14:26.012] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[21:14:26.012] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[21:14:26.012] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[21:14:26.012] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[21:14:26.012] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[21:14:26.012] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[21:14:26.012] D [  glue  ] frame sink sees EOS
[21:14:26.012] D [ pgraph ] <Pgrapher:> graph execution complete 
[21:14:26.012] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.65 sec
[21:14:26.012] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.07 sec
[21:14:26.012] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.93 sec
[21:14:26.012] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.74 sec
[21:14:26.012] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.86 sec
[21:14:26.012] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.79 sec
[21:14:26.012] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.75 sec
[21:14:26.012] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.73 sec
[21:14:26.012] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[21:14:26.012] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[21:14:26.012] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:14:26.012] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[21:14:26.012] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[21:14:26.012] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:14:26.012] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:14:26.012] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:14:26.012] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[21:14:26.012] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[21:14:26.012] I [ timer  ] Timer: Total node execution : 41.53999926522374 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 75189 traces tagged "gauss"
FrameSaver: q=2.36142e+06 n=1162191 tag=gauss
wclsFrameSaver: saving 101445 traces tagged "wiener"
FrameSaver: q=2.90411e+06 n=1101261 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 68687 XUs and 63410 XVs -> 3230 XUVs
C:0 T:3 14 XUs and 31 XVs -> 2 XUVs
C:0 T:5 29436 XUs and 41364 XVs -> 794 XUVs
C:0 T:6 53322 XUs and 54424 XVs -> 2593 XUVs
C:0 T:7 21 XUs and 102 XVs -> 8 XUVs
6627 XUVs total
2222 collection wire objects
6627 potential space points
Neighbour search...
256967 tests to find 155636 neighbours
Iterating with no regularization...
Begin: 2.6494e+07
0 2.24115e+07
1 2.21045e+07
2 2.20539e+07
3 2.20382e+07
Now with regularization...
Begin: 2.02701e+07
0 2.02623e+07
18-Dec-2025 21:15:19 UTC  Closed output file "pdhd_prod_beam__267379_161_1_20251212T162445Z_gen_g4_IonScintPDExt.root_262726_84_1_20251215T212639Z_PDInt.root_269118_0_1_20251218T190728Z_detsim_reco1.root"
18-Dec-2025 21:15:19 UTC  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           61.978        92.2687       105.928       96.974        12.3357        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.00101314     0.0027995    0.00395809    0.00317696    0.00103769       10     
reco:wclsdatahd:WireCellToolkit                      42.4153       44.7505       49.152        44.4141        1.862         10     
reco:wirefilter:WireFilter                          0.0105203     0.0180291     0.0245144     0.0173844    0.00375106       10     
reco:gaushit:GausHitFinder                          0.263607       0.60967      0.733053      0.636927      0.138694        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000116011   0.00030357    0.000541353   0.000302366   0.000105651      10     
reco:reco3d:SpacePointSolver                        0.187247      0.470986       0.88368      0.381568      0.224975        10     
reco:hitpdune:DisambigFromSpacePoints               0.0753393     0.383389       0.51754      0.409646      0.125971        10     
reco:pandora:StandardPandora                         2.3683        7.88775       12.4999       7.71954       2.9287         10     
reco:pandoraWriter:StandardPandora                  0.073588      0.179689      0.231138      0.185721      0.0437931       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.223189      0.890424       1.41675      0.969796      0.326645        10     
reco:pandoraShower:LArPandoraShowerCreation         0.108668       0.93878       1.4999       0.960629      0.392615        10     
reco:pandoracalo:Calorimetry                        0.131804      0.458151      0.662027      0.485274      0.152192        10     
reco:pandoracalonosce:Calorimetry                   0.127418        0.446       0.636163      0.475529      0.147386        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.0929636     0.866234       1.40312      0.953583       0.34705        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.090071      0.872254       1.49307      0.938271      0.357974        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0187159     0.0490878     0.0647656     0.0525075     0.0131272       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0187129     0.0475773     0.0631962     0.0528586     0.0122022       10     
reco:emtrkmichelid:EmTrackMichelId                   15.1022       32.149        39.3259       35.4123       7.12268        10     
[art]:TriggerResults:TriggerResultInserter          1.68e-05     2.9513e-05     8.883e-05    2.4205e-05    2.00636e-05      10     
end_path:out1:RootOutput                             4.6e-06      7.005e-06     2.483e-05     4.975e-06    5.94916e-06      10     
end_path:out1:RootOutput(write)                     0.643691       1.20494       1.5749        1.25853      0.244119        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5909.46 MB
  Peak resident set size usage (VmHWM): 3696.62 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12282368
-rw-r--r-- 1 dunegli fnalgrid     231361 Dec 18 21:15 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        195 Dec 18 21:15 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 18 21:15 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Dec 18 21:15 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     139264 Dec 18 21:15 mem.db
-rw-r--r-- 1 dunegli fnalgrid      28672 Dec 18 21:15 time.db
-rw-r--r-- 1 dunegli fnalgrid 6369732617 Dec 18 21:15 pdhd_prod_beam__267379_161_1_20251212T162445Z_gen_g4_IonScintPDExt.root_262726_84_1_20251215T212639Z_PDInt.root_269118_0_1_20251218T190728Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   37876870 Dec 18 21:14 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Dec 18 20:58 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 18 20:57 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 6169084191 Dec 18 20:57 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        122 Dec 18 19:07 all-input-dids.txt
justIN time: 2025-12-19 13:10:06 UTC       justIN version: 01.05.03