justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 263305.19@dunegpschedd02.fnal.gov

Jobsub ID263305.19@dunegpschedd02.fnal.gov
Workflow ID11264
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 19:31:07
SiteCA_SFU
EntryDUNE_CA_SFU_lcg-ce3
Last heartbeat2025-12-18 21:15:05
From worker nodeHostnamefc30557.int.fir.alliancecan.ca
cpuinfoAMD EPYC 9655 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit84598 (23 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 20:42:15
Input filesusertests:pdhd_prod_beam__261612_58_1_20251212T155846Z_gen_g4_IonScintPDExt.root_262727_94_1_20251215T215106Z_PDInt.root
JobscriptExit code0
Real time31m (1915s)
CPU time28m (1721s = 89%)
Max RSS bytes6566559744 (6262 MiB)
Outputting started2025-12-18 21:14:11
Output filesusertests:pdhd_prod_beam__261612_58_1_20251212T155846Z_gen_g4_IonScintPDExt.root_262727_94_1_20251215T215106Z_PDInt.root_263305_19_1_20251218T204217Z_detsim_reco1.root
Finished2025-12-18 21:15:05
Saved logsjustin-logs:263305.19-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

[13:13:56.893] D [  glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=179 time=0 tick=512 with 76356 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":35531 [0] "wiener0":40825 [40825] ] cmm:[ bad:0 lf_noisy:72 ] input 1: frame: ident=179 time=0 tick=512 with 28955 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":12601 [0] "wiener1":16354 [16354] ] cmm:[ bad:0 ] input 2: frame: ident=179 time=0 tick=512 with 40139 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":17665 [0] "wiener2":22474 [22474] ] cmm:[ bad:0 ] input 3: frame: ident=179 time=0 tick=512 with 35705 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":15425 [0] "wiener3":20280 [20280] ] cmm:[ bad:0 ] output: frame: ident=179 time=0 tick=512 with 181155 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":35531 [0] "gauss1":12601 [0] "gauss2":17665 [0] "gauss3":15425 [0] "wiener0":40825 [40825] "wiener1":16354 [16354] "wiener2":22474 [22474] "wiener3":20280 [20280] ] cmm:[ bad:0 lf_noisy:72 ] 
Retagger: tagging trace set: wiener with 99933 traces, 99933 summary
Retagger: tagging trace set: gauss with 81222 traces, 0 summary
[13:13:56.903] D [  glue  ] sink frame: #179 @0 with 181155 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[13:13:56.903] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[13:13:56.903] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[13:13:56.903] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[13:13:56.903] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[13:13:56.903] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[13:13:56.903] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[13:13:56.903] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[13:13:56.903] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[13:13:56.903] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[13:13:56.903] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[13:13:56.903] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[13:13:56.903] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[13:13:56.903] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[13:13:56.903] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[13:13:56.903] D [  glue  ] frame sink sees EOS
[13:13:56.903] D [ pgraph ] <Pgrapher:> graph execution complete 
[13:13:56.903] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.19 sec
[13:13:56.903] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.19 sec
[13:13:56.903] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 4.82 sec
[13:13:56.903] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 4.6 sec
[13:13:56.903] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.73 sec
[13:13:56.904] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.64 sec
[13:13:56.904] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.63 sec
[13:13:56.904] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.54 sec
[13:13:56.904] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[13:13:56.904] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[13:13:56.904] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[13:13:56.904] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[13:13:56.904] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:13:56.904] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:13:56.904] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[13:13:56.904] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[13:13:56.904] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[13:13:56.904] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[13:13:56.904] I [ timer  ] Timer: Total node execution : 28.38000015169382 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 81222 traces tagged "gauss"
FrameSaver: q=2.56671e+06 n=1274431 tag=gauss
wclsFrameSaver: saving 99933 traces tagged "wiener"
FrameSaver: q=3.33287e+06 n=1237725 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 9597 XUs and 6925 XVs -> 416 XUVs
C:0 T:3 5 XUs and 3 XVs -> 0 XUVs
C:0 T:4 10 XUs and 9 XVs -> 2 XUVs
C:0 T:5 19192 XUs and 17133 XVs -> 558 XUVs
C:0 T:6 12872 XUs and 9480 XVs -> 1370 XUVs
C:0 T:7 0 XUs and 4 XVs -> 0 XUVs
2346 XUVs total
755 collection wire objects
2346 potential space points
Neighbour search...
86486 tests to find 53972 neighbours
Iterating with no regularization...
Begin: 1.17986e+07
0 1.10903e+07
1 1.1026e+07
2 1.10143e+07
3 1.10105e+07
Now with regularization...
Begin: 1.00591e+07
0 1.00565e+07
18-Dec-2025 13:14:10 PST  Closed output file "pdhd_prod_beam__261612_58_1_20251212T155846Z_gen_g4_IonScintPDExt.root_262727_94_1_20251215T215106Z_PDInt.root_263305_19_1_20251218T204217Z_detsim_reco1.root"
18-Dec-2025 13:14:10 PST  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           27.6685       36.2336       48.2597       35.0755       6.91864        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.000248187   0.00456267    0.00910891    0.00361156    0.00306197       10     
reco:wclsdatahd:WireCellToolkit                      21.6778       25.743        29.8731       25.6435       3.7174         10     
reco:wirefilter:WireFilter                         0.00636913     0.013821      0.0339802     0.0104514     0.0082562       10     
reco:gaushit:GausHitFinder                          0.143706      0.196413      0.297425      0.180867      0.047574        10     
reco:nhitsfilter:NumberOfHitsFilter                5.6159e-05    0.000281258   0.000787151   0.000159633   0.000237564      10     
reco:reco3d:SpacePointSolver                        0.0813084     0.138568      0.344214      0.105971      0.0806583       10     
reco:hitpdune:DisambigFromSpacePoints               0.0533102     0.102974       0.21477      0.0827957     0.0507435       10     
reco:pandora:StandardPandora                         1.39983       3.03131       5.53467       2.71848       1.18467        10     
reco:pandoraWriter:StandardPandora                  0.0430003     0.0997226     0.200453      0.0952537     0.044903        10     
reco:pandoraTrack:LArPandoraTrackCreation           0.158638      0.319924      0.847834      0.246231      0.198779        10     
reco:pandoraShower:LArPandoraShowerCreation          0.11329      0.287944      0.818575      0.209075      0.205952        10     
reco:pandoracalo:Calorimetry                        0.0828974     0.158195      0.385114      0.124742      0.0881007       10     
reco:pandoracalonosce:Calorimetry                   0.0805091     0.150795      0.349229      0.116713      0.0817655       10     
reco:pandoraShowercalo:ShowerCalorimetry            0.105916       0.25993      0.679211      0.183467      0.176027        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.100756      0.254497      0.665293      0.180468      0.171942        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0116036     0.0177723     0.0339423     0.0150368    0.00695373       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0113624     0.0178452     0.0333362     0.0146192    0.00703226       10     
reco:emtrkmichelid:EmTrackMichelId                   2.65773       4.88109       8.41764       4.23812       1.95862        10     
[art]:TriggerResults:TriggerResultInserter         1.2649e-05    2.35729e-05    4.536e-05    2.1665e-05    9.63537e-06      10     
end_path:out1:RootOutput                            3.18e-06     8.4389e-06     1.457e-05     7.96e-06     3.1205e-06       10     
end_path:out1:RootOutput(write)                     0.362609      0.507558      0.794854      0.483934      0.121506        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6174.91 MB
  Peak resident set size usage (VmHWM): 3876.53 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 10912688
-rw-r--r-- 1 nilic nilic     230953 Dec 18 13:14 jobscript.log
-rw-r--r-- 1 nilic nilic        200 Dec 18 13:14 justin-processed-pfns.txt
-rw-r--r-- 1 nilic nilic        519 Dec 18 13:14 reco_protodunehd_hist.root
-rw-r--r-- 1 nilic nilic       3574 Dec 18 13:14 Pandora_Geometry.xml
-rw-r--r-- 1 nilic nilic     139264 Dec 18 13:14 mem.db
-rw-r--r-- 1 nilic nilic      28672 Dec 18 13:14 time.db
-rw-r--r-- 1 nilic nilic 5656050827 Dec 18 13:14 pdhd_prod_beam__261612_58_1_20251212T155846Z_gen_g4_IonScintPDExt.root_262727_94_1_20251215T215106Z_PDInt.root_263305_19_1_20251218T204217Z_detsim_reco1.root
-rw-r--r-- 1 nilic nilic   28673390 Dec 18 13:14 Pandora_Events.pndr
-rw-r--r-- 1 nilic nilic          0 Dec 18 13:07 debugprod.log
-rw-r--r-- 1 nilic nilic        519 Dec 18 13:07 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 nilic nilic 5489410736 Dec 18 13:07 temp_detsim.root
-rw-r--r-- 1 nilic nilic        121 Dec 18 12:42 all-input-dids.txt
justIN time: 2025-12-18 23:06:52 UTC       justIN version: 01.05.03