justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 262940.105@dunegpschedd02.fnal.gov

Jobsub ID262940.105@dunegpschedd02.fnal.gov
Workflow ID11235
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-16 21:32:49
SiteUS_UChicago
EntryEngage_US_MWT2_iut2_condce_mcore
Last heartbeat2025-12-17 01:01:35
From worker nodeHostnameiut2-c330.iu.edu
cpuinfoAMD EPYC 7402 24-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit86400 (24 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-16 22:54:48
Input filesusertests:pdhd_prod_beam__261613_64_1_20251212T161445Z_gen_g4_IonScintPDExt.root_262728_163_1_20251215T220754Z_PDInt.root
JobscriptExit code0
Real time1h (6449s)
CPU time1h (6121s = 94%)
Max RSS bytes6619369472 (6312 MiB)
Outputting started2025-12-17 00:42:18
Output filesusertests:pdhd_prod_beam__261613_64_1_20251212T161445Z_gen_g4_IonScintPDExt.root_262728_163_1_20251215T220754Z_PDInt.root_262940_105_1_20251216T225502Z_detsim_reco1.root
Finished2025-12-17 01:01:35
Saved logsjustin-logs:262940.105-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

4] D [  glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=199 time=0 tick=512 with 45190 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":21477 [0] "wiener0":23713 [23713] ] cmm:[ bad:0 lf_noisy:94 ] input 1: frame: ident=199 time=0 tick=512 with 39005 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":16686 [0] "wiener1":22319 [22319] ] cmm:[ bad:0 ] input 2: frame: ident=199 time=0 tick=512 with 58572 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":23937 [0] "wiener2":34635 [34635] ] cmm:[ bad:0 ] input 3: frame: ident=199 time=0 tick=512 with 34282 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":14344 [0] "wiener3":19938 [19938] ] cmm:[ bad:0 ] output: frame: ident=199 time=0 tick=512 with 177049 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":21477 [0] "gauss1":16686 [0] "gauss2":23937 [0] "gauss3":14344 [0] "wiener0":23713 [23713] "wiener1":22319 [22319] "wiener2":34635 [34635] "wiener3":19938 [19938] ] cmm:[ bad:0 lf_noisy:94 ] 
Retagger: tagging trace set: wiener with 100605 traces, 100605 summary
Retagger: tagging trace set: gauss with 76444 traces, 0 summary
[19:41:27.568] D [  glue  ] sink frame: #199 @0 with 177049 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[19:41:27.569] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[19:41:27.569] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[19:41:27.569] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[19:41:27.569] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[19:41:27.569] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[19:41:27.569] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[19:41:27.569] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[19:41:27.569] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[19:41:27.569] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[19:41:27.569] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[19:41:27.569] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[19:41:27.569] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[19:41:27.569] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[19:41:27.569] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[19:41:27.569] D [  glue  ] frame sink sees EOS
[19:41:27.569] D [ pgraph ] <Pgrapher:> graph execution complete 
[19:41:27.569] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 14.43 sec
[19:41:27.569] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.28 sec
[19:41:27.569] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.91 sec
[19:41:27.569] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.47 sec
[19:41:27.569] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.13 sec
[19:41:27.569] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.82 sec
[19:41:27.569] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.73 sec
[19:41:27.569] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.72 sec
[19:41:27.569] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.03 sec
[19:41:27.569] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[19:41:27.569] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[19:41:27.569] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[19:41:27.569] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[19:41:27.569] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[19:41:27.569] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[19:41:27.569] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[19:41:27.569] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[19:41:27.569] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[19:41:27.569] I [ timer  ] Timer: Total node execution : 55.53000024706125 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 76444 traces tagged "gauss"
FrameSaver: q=2.60522e+06 n=1203495 tag=gauss
wclsFrameSaver: saving 100605 traces tagged "wiener"
FrameSaver: q=3.36422e+06 n=1184932 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 27790 XUs and 27554 XVs -> 759 XUVs
C:0 T:3 9 XUs and 6 XVs -> 0 XUVs
C:0 T:4 33 XUs and 25 XVs -> 0 XUVs
C:0 T:5 58881 XUs and 59967 XVs -> 3057 XUVs
C:0 T:6 12182 XUs and 17913 XVs -> 3228 XUVs
C:0 T:7 7 XUs and 14 XVs -> 0 XUVs
7044 XUVs total
2042 collection wire objects
7044 potential space points
Neighbour search...
357928 tests to find 220700 neighbours
Iterating with no regularization...
Begin: 2.99545e+07
0 2.73586e+07
1 2.71419e+07
2 2.71084e+07
3 2.70969e+07
Now with regularization...
Begin: 2.47837e+07
0 2.47752e+07
16-Dec-2025 19:42:16 EST  Closed output file "pdhd_prod_beam__261613_64_1_20251212T161445Z_gen_g4_IonScintPDExt.root_262728_163_1_20251215T220754Z_PDInt.root_262940_105_1_20251216T225502Z_detsim_reco1.root"
16-Dec-2025 19:42:16 EST  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           87.1574       102.914       116.458       105.799       9.80938        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.00149241    0.00455351     0.0067326    0.00483468    0.00179498       10     
reco:wclsdatahd:WireCellToolkit                      50.7756       55.8768       63.1108       55.6141       3.35751        10     
reco:wirefilter:WireFilter                          0.0153128     0.0304136     0.0565434     0.0205429     0.0158733       10     
reco:gaushit:GausHitFinder                          0.481079      0.678212      0.948599      0.668206       0.14788        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000279092   0.00038904    0.000600642   0.000358991   0.000104742      10     
reco:reco3d:SpacePointSolver                        0.271469      0.673237       1.73858      0.562033      0.400656        10     
reco:hitpdune:DisambigFromSpacePoints               0.273101      0.490344      0.792564      0.499889      0.169198        10     
reco:pandora:StandardPandora                         5.06114       9.65279       15.138        8.92313       3.04384        10     
reco:pandoraWriter:StandardPandora                  0.183548      0.238528      0.310007      0.229086      0.0456536       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.735043       0.98975       1.4817       0.997705      0.197517        10     
reco:pandoraShower:LArPandoraShowerCreation         0.630996       1.10432       1.58932       1.14808      0.354466        10     
reco:pandoracalo:Calorimetry                        0.354881      0.555262      0.667069      0.590427       0.10101        10     
reco:pandoracalonosce:Calorimetry                   0.344525       0.52376      0.699305      0.528462      0.114498        10     
reco:pandoraShowercalo:ShowerCalorimetry             0.49359       1.17086       1.75051       1.22014      0.430842        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.491862       1.08722       1.67821       1.08351      0.381315        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0352172     0.0555533     0.0762957     0.0556526     0.0142054       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0348415     0.0539584     0.0757317     0.0520308     0.0144058       10     
reco:emtrkmichelid:EmTrackMichelId                   20.4916       28.3096       40.2071       27.4255       5.07756        10     
[art]:TriggerResults:TriggerResultInserter         2.9547e-05    5.04884e-05   0.000103746   3.5211e-05    2.53061e-05      10     
end_path:out1:RootOutput                            7.153e-06    1.08084e-05    3.197e-05    8.4565e-06    7.11221e-06      10     
end_path:out1:RootOutput(write)                     0.944455       1.33304       1.64656       1.34737      0.225478        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5698.09 MB
  Peak resident set size usage (VmHWM): 3495.88 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12144492
-rw-r--r--. 1 dune osgvo     231302 Dec 16 19:42 jobscript.log
-rw-r--r--. 1 dune osgvo        201 Dec 16 19:42 justin-processed-pfns.txt
-rw-r--r--. 1 dune osgvo        519 Dec 16 19:42 reco_protodunehd_hist.root
-rw-r--r--. 1 dune osgvo       3574 Dec 16 19:42 Pandora_Geometry.xml
-rw-r--r--. 1 dune osgvo     139264 Dec 16 19:42 mem.db
-rw-r--r--. 1 dune osgvo      28672 Dec 16 19:42 time.db
-rw-r--r--. 1 dune osgvo 6299567644 Dec 16 19:42 pdhd_prod_beam__261613_64_1_20251212T161445Z_gen_g4_IonScintPDExt.root_262728_163_1_20251215T220754Z_PDInt.root_262940_105_1_20251216T225502Z_detsim_reco1.root
-rw-r--r--. 1 dune osgvo   38296960 Dec 16 19:41 Pandora_Events.pndr
-rw-r--r--. 1 dune osgvo          0 Dec 16 19:22 debugprod.log
-rw-r--r--. 1 dune osgvo        519 Dec 16 19:22 detsim_single_protodunehd_hist.root
-rw-r--r--. 1 dune osgvo 6097647728 Dec 16 19:22 temp_detsim.root
-rw-r--r--. 1 dune osgvo        122 Dec 16 17:55 all-input-dids.txt
justIN time: 2025-12-19 05:18:29 UTC       justIN version: 01.05.03