justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 263519.0@dunegpschedd02.fnal.gov

Jobsub ID263519.0@dunegpschedd02.fnal.gov
Workflow ID11264
Stage ID1
User nameavizcaya@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-20 02:55:38
SiteCA_SFU
EntryDUNE_CA_SFU_lcg-ce3
Last heartbeat2025-12-20 03:36:00
From worker nodeHostnamefc30569.int.fir.alliancecan.ca
cpuinfoAMD EPYC 9655 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit84598 (23 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-20 02:56:45
Input filesusertests:pdhd_prod_beam__261612_49_1_20251212T155723Z_gen_g4_IonScintPDExt.root_262727_58_1_20251215T214613Z_PDInt.root
JobscriptExit code0
Real time38m (2310s)
CPU time32m (1949s = 84%)
Max RSS bytes6401253376 (6104 MiB)
Outputting started2025-12-20 03:35:16
Output filesusertests:pdhd_prod_beam__261612_49_1_20251212T155723Z_gen_g4_IonScintPDExt.root_262727_58_1_20251215T214613Z_PDInt.root_263519_0_1_20251220T025718Z_detsim_reco1.root
Finished2025-12-20 03:36:00
Saved logsjustin-logs:263519.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

"sigproc" ] 2 tagged trace sets:[ "gauss0":22110 [0] "wiener0":25518 [25518] ] cmm:[ bad:0 lf_noisy:133 ] 
[19:35:05.191] D [  glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=179 time=0 tick=512 with 47628 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":22110 [0] "wiener0":25518 [25518] ] cmm:[ bad:0 lf_noisy:133 ] input 1: frame: ident=179 time=0 tick=512 with 35106 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":15700 [0] "wiener1":19406 [19406] ] cmm:[ bad:0 ] input 2: frame: ident=179 time=0 tick=512 with 47148 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":19922 [0] "wiener2":27226 [27226] ] cmm:[ bad:0 ] input 3: frame: ident=179 time=0 tick=512 with 37091 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":16218 [0] "wiener3":20873 [20873] ] cmm:[ bad:0 lf_noisy:1 ] output: frame: ident=179 time=0 tick=512 with 166973 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":22110 [0] "gauss1":15700 [0] "gauss2":19922 [0] "gauss3":16218 [0] "wiener0":25518 [25518] "wiener1":19406 [19406] "wiener2":27226 [27226] "wiener3":20873 [20873] ] cmm:[ bad:0 lf_noisy:134 ] 
Retagger: tagging trace set: wiener with 93023 traces, 93023 summary
Retagger: tagging trace set: gauss with 73950 traces, 0 summary
[19:35:05.196] D [  glue  ] sink frame: #179 @0 with 166973 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[19:35:05.196] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[19:35:05.196] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[19:35:05.196] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[19:35:05.196] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[19:35:05.196] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[19:35:05.196] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[19:35:05.196] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[19:35:05.196] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[19:35:05.196] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[19:35:05.196] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[19:35:05.196] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[19:35:05.196] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[19:35:05.196] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[19:35:05.196] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[19:35:05.196] D [  glue  ] frame sink sees EOS
[19:35:05.196] D [ pgraph ] <Pgrapher:> graph execution complete 
[19:35:05.196] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.44 sec
[19:35:05.196] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 3.72 sec
[19:35:05.196] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 3.67 sec
[19:35:05.196] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 3.63 sec
[19:35:05.196] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.57 sec
[19:35:05.196] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.57 sec
[19:35:05.196] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.56 sec
[19:35:05.196] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.55 sec
[19:35:05.196] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[19:35:05.196] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[19:35:05.196] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[19:35:05.196] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[19:35:05.196] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[19:35:05.196] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[19:35:05.196] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[19:35:05.196] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[19:35:05.196] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[19:35:05.196] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[19:35:05.196] I [ timer  ] Timer: Total node execution : 22.740000275895 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 73950 traces tagged "gauss"
FrameSaver: q=2.20049e+06 n=1172631 tag=gauss
wclsFrameSaver: saving 93023 traces tagged "wiener"
FrameSaver: q=2.65093e+06 n=1106329 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 15161 XUs and 16392 XVs -> 630 XUVs
C:0 T:5 37176 XUs and 35685 XVs -> 1193 XUVs
C:0 T:6 10060 XUs and 23798 XVs -> 566 XUVs
2389 XUVs total
1587 collection wire objects
2389 potential space points
Neighbour search...
36957 tests to find 22492 neighbours
Iterating with no regularization...
Begin: 1.85904e+07
0 1.76837e+07
1 1.76575e+07
2 1.76529e+07
Now with regularization...
Begin: 1.65003e+07
0 1.64986e+07
19-Dec-2025 19:35:15 PST  Closed output file "pdhd_prod_beam__261612_49_1_20251212T155723Z_gen_g4_IonScintPDExt.root_262727_58_1_20251215T214613Z_PDInt.root_263519_0_1_20251220T025718Z_detsim_reco1.root"
19-Dec-2025 19:35:15 PST  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           28.9834       35.718        39.9169       35.9409       3.49567        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.000475468   0.00331352      0.0057      0.00380209    0.00181538       10     
reco:wclsdatahd:WireCellToolkit                      24.0149       24.5594       25.4356       24.4541      0.425195        10     
reco:wirefilter:WireFilter                         0.00675442    0.00941388     0.015133     0.00900943    0.00217719       10     
reco:gaushit:GausHitFinder                          0.133225      0.249263      0.332366      0.246278      0.0560867       10     
reco:nhitsfilter:NumberOfHitsFilter                 9.436e-05    0.000146421   0.000291279   0.000133484   5.38054e-05      10     
reco:reco3d:SpacePointSolver                        0.0711664      0.2078       0.440756      0.205037      0.105602        10     
reco:hitpdune:DisambigFromSpacePoints               0.0487509     0.149207      0.216587      0.153697      0.0523293       10     
reco:pandora:StandardPandora                         1.02926       3.30164       5.53361       3.09434       1.32384        10     
reco:pandoraWriter:StandardPandora                  0.047091      0.0853772     0.108793      0.0917975     0.0179899       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.193959      0.388654      0.594001      0.415797      0.117956        10     
reco:pandoraShower:LArPandoraShowerCreation         0.0881558      0.37559      0.589402      0.385108      0.148787        10     
reco:pandoracalo:Calorimetry                        0.0858035     0.201121      0.278492      0.222723      0.0533249       10     
reco:pandoracalonosce:Calorimetry                   0.0992166      0.20287      0.273613      0.221657      0.0509607       10     
reco:pandoraShowercalo:ShowerCalorimetry            0.0972812     0.365704      0.577515       0.38043      0.143214        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.096849      0.369224      0.590084      0.373939      0.143745        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0137734     0.0209604     0.031468      0.0211655    0.00514648       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0112525     0.0193379     0.0265951     0.0200942    0.00433793       10     
reco:emtrkmichelid:EmTrackMichelId                   2.49233       4.60702       6.33527       4.75014       1.0781         10     
[art]:TriggerResults:TriggerResultInserter          1.527e-05    2.1474e-05     6.383e-05    1.6075e-05    1.42089e-05      10     
end_path:out1:RootOutput                            4.04e-06      7.796e-06     2.772e-05     6.34e-06     6.7134e-06       10     
end_path:out1:RootOutput(write)                     0.373161      0.580593      0.702485      0.600933      0.0958044       10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6168.44 MB
  Peak resident set size usage (VmHWM): 3797.72 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 10908300
-rw-r--r-- 1 nilic nilic     231084 Dec 19 19:35 jobscript.log
-rw-r--r-- 1 nilic nilic        200 Dec 19 19:35 justin-processed-pfns.txt
-rw-r--r-- 1 nilic nilic        519 Dec 19 19:35 reco_protodunehd_hist.root
-rw-r--r-- 1 nilic nilic       3574 Dec 19 19:35 Pandora_Geometry.xml
-rw-r--r-- 1 nilic nilic     139264 Dec 19 19:35 mem.db
-rw-r--r-- 1 nilic nilic      28672 Dec 19 19:35 time.db
-rw-r--r-- 1 nilic nilic 5660445249 Dec 19 19:35 pdhd_prod_beam__261612_49_1_20251212T155723Z_gen_g4_IonScintPDExt.root_262727_58_1_20251215T214613Z_PDInt.root_263519_0_1_20251220T025718Z_detsim_reco1.root
-rw-r--r-- 1 nilic nilic   33804560 Dec 19 19:35 Pandora_Events.pndr
-rw-r--r-- 1 nilic nilic          0 Dec 19 19:28 debugprod.log
-rw-r--r-- 1 nilic nilic        519 Dec 19 19:27 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 nilic nilic 5475391651 Dec 19 19:27 temp_detsim.root
-rw-r--r-- 1 nilic nilic        121 Dec 19 18:57 all-input-dids.txt
justIN time: 2026-02-04 08:00:22 UTC       justIN version: 01.06.00