justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 241192.0@dunegpschedd01.fnal.gov

Jobsub ID241192.0@dunegpschedd01.fnal.gov
Workflow ID9455
Stage ID1
User namemwrobel@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-10-30 20:48:06
SiteUS_UChicago
EntryEngage_US_MWT2_uiuc_condce_mcore
Last heartbeat2025-10-30 22:08:46
From worker nodeHostnameiut2-c409.iu.edu
cpuinfoAMD EPYC 7443 24-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit86400 (24 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-10-30 20:49:00
Input filesusertests:pdhd_prod_beam__231691_75_1_20251008T183305Z_gen_g4_IonScintPDExt.root_227038_94_1_20251010T163702Z_PDInt.root
JobscriptExit code0
Real time1h (4672s)
CPU time1h (4414s = 94%)
Max RSS bytes7095422976 (6766 MiB)
Outputting started2025-10-30 22:06:53
Output filesusertests:pdhd_prod_beam__231691_75_1_20251008T183305Z_gen_g4_IonScintPDExt.root_227038_94_1_20251010T163702Z_PDInt.root_241192_0_1_20251030T204911Z_detsim_reco1.root
Finished2025-10-30 22:08:46
Saved logsjustin-logs:241192.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ener0":32447 [32447] ] cmm:[ bad:0 lf_noisy:198 ] 
[18:05:51.023] D [  glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=89 time=0 tick=512 with 61266 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":28819 [0] "wiener0":32447 [32447] ] cmm:[ bad:0 lf_noisy:198 ] input 1: frame: ident=89 time=0 tick=512 with 53255 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":22126 [0] "wiener1":31129 [31129] ] cmm:[ bad:0 ] input 2: frame: ident=89 time=0 tick=512 with 52742 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":21874 [0] "wiener2":30868 [30868] ] cmm:[ bad:0 ] input 3: frame: ident=89 time=0 tick=512 with 46572 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":19974 [0] "wiener3":26598 [26598] ] cmm:[ bad:0 ] output: frame: ident=89 time=0 tick=512 with 213835 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":28819 [0] "gauss1":22126 [0] "gauss2":21874 [0] "gauss3":19974 [0] "wiener0":32447 [32447] "wiener1":31129 [31129] "wiener2":30868 [30868] "wiener3":26598 [26598] ] cmm:[ bad:0 lf_noisy:198 ] 
Retagger: tagging trace set: wiener with 121042 traces, 121042 summary
Retagger: tagging trace set: gauss with 92793 traces, 0 summary
[18:05:51.041] D [  glue  ] sink frame: #89 @0 with 213835 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[18:05:51.041] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[18:05:51.041] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[18:05:51.041] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[18:05:51.041] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[18:05:51.041] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[18:05:51.041] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[18:05:51.041] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[18:05:51.041] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[18:05:51.041] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[18:05:51.042] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[18:05:51.042] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[18:05:51.042] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[18:05:51.042] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[18:05:51.042] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[18:05:51.042] D [  glue  ] frame sink sees EOS
[18:05:51.042] D [ pgraph ] <Pgrapher:> graph execution complete 
[18:05:51.042] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.91 sec
[18:05:51.042] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.78 sec
[18:05:51.042] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.69 sec
[18:05:51.042] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.35 sec
[18:05:51.042] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.98 sec
[18:05:51.042] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.93 sec
[18:05:51.042] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.91 sec
[18:05:51.042] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.88 sec
[18:05:51.042] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.02 sec
[18:05:51.042] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[18:05:51.042] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[18:05:51.042] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[18:05:51.042] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:05:51.042] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:05:51.042] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[18:05:51.042] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[18:05:51.042] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[18:05:51.042] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[18:05:51.042] I [ timer  ] Timer: Total node execution : 50.479999827221036 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 92793 traces tagged "gauss"
FrameSaver: q=3.16816e+06 n=1502045 tag=gauss
wclsFrameSaver: saving 121042 traces tagged "wiener"
FrameSaver: q=3.99019e+06 n=1432529 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 46956 XUs and 82784 XVs -> 3982 XUVs
C:0 T:3 12 XUs and 41 XVs -> 8 XUVs
C:0 T:4 71 XUs and 53 XVs -> 1 XUVs
C:0 T:5 76853 XUs and 46436 XVs -> 1652 XUVs
C:0 T:6 39372 XUs and 34167 XVs -> 894 XUVs
6537 XUVs total
2342 collection wire objects
6537 potential space points
Neighbour search...
359357 tests to find 215154 neighbours
Iterating with no regularization...
Begin: 5.00826e+07
0 4.49734e+07
1 4.47552e+07
2 4.47256e+07
Now with regularization...
Begin: 4.23792e+07
0 4.23678e+07
30-Oct-2025 18:06:52 EDT  Closed output file "pdhd_prod_beam__231691_75_1_20251008T183305Z_gen_g4_IonScintPDExt.root_227038_94_1_20251010T163702Z_PDInt.root_241192_0_1_20251030T204911Z_detsim_reco1.root"
30-Oct-2025 18:06:52 EDT  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           85.235        94.2064       112.255       90.289        8.79968        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.00136768    0.00468944    0.00723278    0.00530692    0.00211094       10     
reco:wclsdatahd:WireCellToolkit                      49.8484       52.0656       58.9452       51.4617       2.41416        10     
reco:wirefilter:WireFilter                          0.019109      0.0277205     0.0406549     0.0252865    0.00679848       10     
reco:gaushit:GausHitFinder                          0.517894      0.668678      0.969814      0.599816      0.132541        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000289063   0.000410206   0.000640434   0.000369751   0.000121733      10     
reco:reco3d:SpacePointSolver                         0.38505      0.672135       1.21646      0.563326       0.28299        10     
reco:hitpdune:DisambigFromSpacePoints               0.251882       0.45969      0.875381      0.399085      0.179279        10     
reco:pandora:StandardPandora                         6.88166        9.563        13.8053       8.37643       2.5414         10     
reco:pandoraWriter:StandardPandora                  0.228827      0.267163      0.360952       0.24565      0.0422548       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.509264       1.06492       1.88556      0.932922       0.3956         10     
reco:pandoraShower:LArPandoraShowerCreation          0.5629        1.06678       2.03798      0.854221      0.472019        10     
reco:pandoracalo:Calorimetry                        0.330973      0.601748      0.946986      0.551574      0.185803        10     
reco:pandoracalonosce:Calorimetry                   0.438452      0.583615      0.850147      0.550094      0.131218        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.575306       1.06121       1.95953      0.838986      0.418854        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.571742       1.04301       1.91901      0.883912      0.404617        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0438225     0.0664962     0.0893563     0.0671758     0.0122209       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0413061     0.0654066     0.0869223     0.0673944     0.0129865       10     
reco:emtrkmichelid:EmTrackMichelId                   21.1354       23.4797       30.9098       21.5019       3.31643        10     
[art]:TriggerResults:TriggerResultInserter         3.1819e-05    4.97366e-05   0.000119185   4.2605e-05    2.4374e-05       10     
end_path:out1:RootOutput                            9.007e-06    1.57666e-05   3.8993e-05    1.32045e-05   8.42996e-06      10     
end_path:out1:RootOutput(write)                      1.09297       1.35367       1.80583       1.29411      0.222071        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6120.87 MB
  Peak resident set size usage (VmHWM): 3919.87 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12638516
-rw-r--r-- 1 dune osgvo     230734 Oct 30 18:06 jobscript.log
-rw-r--r-- 1 dune osgvo        200 Oct 30 18:06 justin-processed-pfns.txt
-rw-r--r-- 1 dune osgvo        519 Oct 30 18:06 reco_protodunehd_hist.root
-rw-r--r-- 1 dune osgvo       3574 Oct 30 18:06 Pandora_Geometry.xml
-rw-r--r-- 1 dune osgvo     139264 Oct 30 18:06 mem.db
-rw-r--r-- 1 dune osgvo      28672 Oct 30 18:06 time.db
-rw-r--r-- 1 dune osgvo 6548868059 Oct 30 18:06 pdhd_prod_beam__231691_75_1_20251008T183305Z_gen_g4_IonScintPDExt.root_227038_94_1_20251010T163702Z_PDInt.root_241192_0_1_20251030T204911Z_detsim_reco1.root
-rw-r--r-- 1 dune osgvo   36591960 Oct 30 18:06 Pandora_Events.pndr
-rw-r--r-- 1 dune osgvo          0 Oct 30 17:49 debugprod.log
-rw-r--r-- 1 dune osgvo        519 Oct 30 17:48 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dune osgvo 6355932812 Oct 30 17:48 temp_detsim.root
-rw-r--r-- 1 dune osgvo        121 Oct 30 16:49 all-input-dids.txt
justIN time: 2025-11-04 18:20:12 UTC       justIN version: 01.05.01