justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 263305.35@dunegpschedd02.fnal.gov

Jobsub ID263305.35@dunegpschedd02.fnal.gov
Workflow ID11264
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 19:31:07
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-18 21:51:01
From worker nodeHostnamedunegli-7951944-0-fnpc23110.fnal.gov
cpuinfoAMD EPYC 9354 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 21:01:47
Input filesusertests:pdhd_prod_beam__261612_47_1_20251212T155732Z_gen_g4_IonScintPDExt.root_262727_28_1_20251215T214525Z_PDInt.root
JobscriptExit code0
Real time48m (2914s)
CPU time47m (2839s = 97%)
Max RSS bytes6240206848 (5951 MiB)
Outputting started2025-12-18 21:50:21
Output filesusertests:pdhd_prod_beam__261612_47_1_20251212T155732Z_gen_g4_IonScintPDExt.root_262727_28_1_20251215T214525Z_PDInt.root_263305_35_1_20251218T210149Z_detsim_reco1.root
Finished2025-12-18 21:51:01
Saved logsjustin-logs:263305.35-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

[21:50:09.846] D [  glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=179 time=0 tick=512 with 52761 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":24938 [0] "wiener0":27823 [27823] ] cmm:[ bad:0 lf_noisy:168 ] input 1: frame: ident=179 time=0 tick=512 with 26424 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":12068 [0] "wiener1":14356 [14356] ] cmm:[ bad:0 ] input 2: frame: ident=179 time=0 tick=512 with 38644 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":16259 [0] "wiener2":22385 [22385] ] cmm:[ bad:0 ] input 3: frame: ident=179 time=0 tick=512 with 33945 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":15602 [0] "wiener3":18343 [18343] ] cmm:[ bad:0 ] output: frame: ident=179 time=0 tick=512 with 151774 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":24938 [0] "gauss1":12068 [0] "gauss2":16259 [0] "gauss3":15602 [0] "wiener0":27823 [27823] "wiener1":14356 [14356] "wiener2":22385 [22385] "wiener3":18343 [18343] ] cmm:[ bad:0 lf_noisy:168 ] 
Retagger: tagging trace set: wiener with 82907 traces, 82907 summary
Retagger: tagging trace set: gauss with 68867 traces, 0 summary
[21:50:09.850] D [  glue  ] sink frame: #179 @0 with 151774 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[21:50:09.850] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[21:50:09.850] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[21:50:09.850] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[21:50:09.850] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[21:50:09.850] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[21:50:09.850] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[21:50:09.850] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[21:50:09.850] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[21:50:09.850] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[21:50:09.850] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[21:50:09.850] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[21:50:09.850] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[21:50:09.850] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[21:50:09.850] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[21:50:09.851] D [  glue  ] frame sink sees EOS
[21:50:09.851] D [ pgraph ] <Pgrapher:> graph execution complete 
[21:50:09.851] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.65 sec
[21:50:09.851] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 4.47 sec
[21:50:09.851] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 4.37 sec
[21:50:09.851] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 4.31 sec
[21:50:09.851] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.8 sec
[21:50:09.851] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.79 sec
[21:50:09.851] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.78 sec
[21:50:09.851] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.78 sec
[21:50:09.851] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[21:50:09.851] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[21:50:09.851] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:50:09.851] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:50:09.851] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:50:09.851] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:50:09.851] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[21:50:09.851] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0 sec
[21:50:09.851] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[21:50:09.851] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[21:50:09.851] I [ timer  ] Timer: Total node execution : 26.95999957062304 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 68867 traces tagged "gauss"
FrameSaver: q=1.96677e+06 n=1045645 tag=gauss
wclsFrameSaver: saving 82907 traces tagged "wiener"
FrameSaver: q=2.65077e+06 n=1041862 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 11718 XUs and 5793 XVs -> 166 XUVs
C:0 T:4 36 XUs and 63 XVs -> 4 XUVs
C:0 T:5 20913 XUs and 17220 XVs -> 546 XUVs
C:0 T:6 13407 XUs and 11725 XVs -> 269 XUVs
C:0 T:7 4 XUs and 6 XVs -> 0 XUVs
985 XUVs total
689 collection wire objects
985 potential space points
Neighbour search...
12271 tests to find 7648 neighbours
Iterating with no regularization...
Begin: 1.0035e+07
0 9.61262e+06
1 9.58697e+06
2 9.58545e+06
Now with regularization...
Begin: 9.1344e+06
0 9.13409e+06
18-Dec-2025 21:50:20 UTC  Closed output file "pdhd_prod_beam__261612_47_1_20251212T155732Z_gen_g4_IonScintPDExt.root_262727_28_1_20251215T214525Z_PDInt.root_263305_35_1_20251218T210149Z_detsim_reco1.root"
18-Dec-2025 21:50:20 UTC  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           38.1249       41.9554       48.4387       40.6175       3.25984        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.00043929     0.003048      0.0053282    0.00306714    0.00162235       10     
reco:wclsdatahd:WireCellToolkit                      27.4051       27.8608       28.7968       27.8478      0.367427        10     
reco:wirefilter:WireFilter                         0.00661133    0.00817082    0.00973336    0.00804578    0.00108005       10     
reco:gaushit:GausHitFinder                          0.208184       0.25222      0.329737      0.247511      0.0376509       10     
reco:nhitsfilter:NumberOfHitsFilter                7.3241e-05    9.6497e-05    0.000210952    8.524e-05    3.8772e-05       10     
reco:reco3d:SpacePointSolver                        0.0857903     0.220085      0.416319      0.161421      0.124914        10     
reco:hitpdune:DisambigFromSpacePoints               0.0814257     0.135385       0.24425      0.116017      0.048817        10     
reco:pandora:StandardPandora                         2.16633       3.50176       6.1014        3.10514       1.35106        10     
reco:pandoraWriter:StandardPandora                  0.0584151     0.0812009     0.128241      0.0722287     0.0245507       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.197776       0.32071      0.458758      0.306432      0.0763914       10     
reco:pandoraShower:LArPandoraShowerCreation         0.140099      0.301369      0.541857      0.277488      0.121148        10     
reco:pandoracalo:Calorimetry                        0.134503      0.198686      0.282293      0.196269      0.0469404       10     
reco:pandoracalonosce:Calorimetry                   0.129976      0.192777      0.274374      0.190442      0.0462547       10     
reco:pandoraShowercalo:ShowerCalorimetry            0.151555      0.304043      0.538344       0.2968       0.127361        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.148752      0.299442      0.529788      0.292904      0.125932        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0157235     0.0195644     0.0244575     0.0196895    0.00315137       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0150846     0.0185331     0.0237503     0.0177098    0.00315575       10     
reco:emtrkmichelid:EmTrackMichelId                   6.33635       7.64321       9.91839       7.06087       1.18096        10     
[art]:TriggerResults:TriggerResultInserter         1.2168e-05    1.99604e-05   6.5801e-05    1.5148e-05    1.53553e-05      10     
end_path:out1:RootOutput                            3.045e-06    6.3737e-06    2.7411e-05    3.5505e-06    7.10674e-06      10     
end_path:out1:RootOutput(write)                     0.503101      0.575696      0.681892       0.55613      0.063163        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6177.7 MB
  Peak resident set size usage (VmHWM): 3803.03 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 10447088
-rw-r--r-- 1 dunegli fnalgrid     231050 Dec 18 21:50 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        200 Dec 18 21:50 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 18 21:50 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Dec 18 21:50 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     139264 Dec 18 21:50 mem.db
-rw-r--r-- 1 dunegli fnalgrid      28672 Dec 18 21:50 time.db
-rw-r--r-- 1 dunegli fnalgrid 5417909771 Dec 18 21:50 pdhd_prod_beam__261612_47_1_20251212T155732Z_gen_g4_IonScintPDExt.root_262727_28_1_20251215T214525Z_PDInt.root_263305_35_1_20251218T210149Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   30858760 Dec 18 21:50 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Dec 18 21:42 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 18 21:42 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 5248527551 Dec 18 21:42 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        121 Dec 18 21:01 all-input-dids.txt
justIN time: 2025-12-18 23:05:59 UTC       justIN version: 01.05.03