justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 262940.16@dunegpschedd02.fnal.gov

Jobsub ID262940.16@dunegpschedd02.fnal.gov
Workflow ID11235
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-16 21:32:49
SiteUS_BNL
EntryDUNE_US_BNL_spoolce02
Last heartbeat2025-12-16 23:55:15
From worker nodeHostnamespool0691.sdcc.bnl.gov
cpuinfoIntel(R) Xeon(R) Gold 6150 CPU @ 2.70GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-16 22:00:16
Input filesusertests:pdhd_prod_beam__261613_145_1_20251212T162029Z_gen_g4_IonScintPDExt.root_262728_100_1_20251215T220322Z_PDInt.root
JobscriptExit code0
Real time1h (6764s)
CPU time1h (6530s = 96%)
Max RSS bytes7061954560 (6734 MiB)
Outputting started2025-12-16 23:53:01
Output filesusertests:pdhd_prod_beam__261613_145_1_20251212T162029Z_gen_g4_IonScintPDExt.root_262728_100_1_20251215T220322Z_PDInt.root_262940_16_1_20251216T220034Z_detsim_reco1.root
Finished2025-12-16 23:55:15
Saved logsjustin-logs:262940.16-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

with 66024 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":30657 [0] "wiener0":35367 [35367] ] cmm:[ bad:0 lf_noisy:120 ] input 1: frame: ident=199 time=0 tick=512 with 46108 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":19188 [0] "wiener1":26920 [26920] ] cmm:[ bad:0 ] input 2: frame: ident=199 time=0 tick=512 with 34716 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":15479 [0] "wiener2":19237 [19237] ] cmm:[ bad:0 ] input 3: frame: ident=199 time=0 tick=512 with 45130 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":19351 [0] "wiener3":25779 [25779] ] cmm:[ bad:0 ] output: frame: ident=199 time=0 tick=512 with 191978 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":30657 [0] "gauss1":19188 [0] "gauss2":15479 [0] "gauss3":19351 [0] "wiener0":35367 [35367] "wiener1":26920 [26920] "wiener2":19237 [19237] "wiener3":25779 [25779] ] cmm:[ bad:0 lf_noisy:120 ] 
Retagger: tagging trace set: wiener with 107303 traces, 107303 summary
Retagger: tagging trace set: gauss with 84675 traces, 0 summary
[18:52:01.335] D [  glue  ] sink frame: #199 @0 with 191978 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[18:52:01.335] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[18:52:01.335] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[18:52:01.335] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[18:52:01.335] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[18:52:01.335] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[18:52:01.335] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[18:52:01.335] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[18:52:01.335] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[18:52:01.335] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[18:52:01.335] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[18:52:01.335] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[18:52:01.335] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[18:52:01.335] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[18:52:01.335] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[18:52:01.335] D [  glue  ] frame sink sees EOS
[18:52:01.335] D [ pgraph ] <Pgrapher:> graph execution complete 
[18:52:01.335] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 17.96 sec
[18:52:01.336] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.56 sec
[18:52:01.336] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.3 sec
[18:52:01.336] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.83 sec
[18:52:01.336] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.7 sec
[18:52:01.336] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.51 sec
[18:52:01.336] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.39 sec
[18:52:01.336] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.37 sec
[18:52:01.336] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.03 sec
[18:52:01.336] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.02 sec
[18:52:01.336] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.02 sec
[18:52:01.336] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[18:52:01.336] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:52:01.336] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[18:52:01.336] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:52:01.336] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[18:52:01.336] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[18:52:01.336] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[18:52:01.336] I [ timer  ] Timer: Total node execution : 64.69999964535236 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 84675 traces tagged "gauss"
FrameSaver: q=2.77601e+06 n=1342369 tag=gauss
wclsFrameSaver: saving 107303 traces tagged "wiener"
FrameSaver: q=3.52273e+06 n=1307643 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 42315 XUs and 107336 XVs -> 11242 XUVs
C:0 T:3 107 XUs and 133 XVs -> 0 XUVs
C:0 T:4 3 XUs and 0 XVs -> 0 XUVs
C:0 T:5 26097 XUs and 19968 XVs -> 542 XUVs
C:0 T:6 54905 XUs and 25692 XVs -> 958 XUVs
C:0 T:7 83 XUs and 35 XVs -> 0 XUVs
12742 XUVs total
1914 collection wire objects
12742 potential space points
Neighbour search...
2518560 tests to find 1236960 neighbours
Iterating with no regularization...
Begin: 2.78357e+07
0 2.41428e+07
1 2.39678e+07
2 2.39292e+07
3 2.39125e+07
Now with regularization...
Begin: 2.23784e+07
0 2.23592e+07
16-Dec-2025 18:52:58 EST  Closed output file "pdhd_prod_beam__261613_145_1_20251212T162029Z_gen_g4_IonScintPDExt.root_262728_100_1_20251215T220322Z_PDInt.root_262940_16_1_20251216T220034Z_detsim_reco1.root"
16-Dec-2025 18:52:58 EST  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           109.75        145.152       199.426       136.411       28.8225        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.00117067     0.0100668     0.036274     0.00431053     0.0127771       10     
reco:wclsdatahd:WireCellToolkit                      59.0505       82.3674       100.292       84.7614       14.9265        10     
reco:wirefilter:WireFilter                          0.0190087     0.0287821     0.0441321     0.0261886    0.00851146       10     
reco:gaushit:GausHitFinder                          0.455565      0.907511       1.67617      0.850342      0.380857        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000221068   0.000369987   0.000709407   0.00034773    0.00014229       10     
reco:reco3d:SpacePointSolver                        0.288281      0.714468       1.71705      0.581231      0.390858        10     
reco:hitpdune:DisambigFromSpacePoints                0.31469      0.838481       2.00833      0.677557      0.517002        10     
reco:pandora:StandardPandora                         4.32866       12.1713       31.9884       8.97863       8.05394        10     
reco:pandoraWriter:StandardPandora                  0.192497      0.298613      0.512712      0.251863      0.099255        10     
reco:pandoraTrack:LArPandoraTrackCreation           0.754354       1.60376       3.50403       1.23414      0.863875        10     
reco:pandoraShower:LArPandoraShowerCreation         0.466429       1.58389       3.91951       1.19486      0.979391        10     
reco:pandoracalo:Calorimetry                        0.389489      0.817504       1.73732      0.661632      0.408932        10     
reco:pandoracalonosce:Calorimetry                   0.380619      0.793461       1.65513      0.648036      0.387739        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.375481       1.44352       3.19963       1.15804       0.80733        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.367514       1.44504       3.19397       1.20344      0.795319        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0439062     0.0812983      0.14551      0.0719647     0.0308689       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0446819     0.0825768     0.147007      0.0737971     0.030248        10     
reco:emtrkmichelid:EmTrackMichelId                   23.9316       38.1011       49.9225       40.0518       7.00177        10     
[art]:TriggerResults:TriggerResultInserter         2.3301e-05    5.38154e-05   0.000110909   5.1542e-05    2.17406e-05      10     
end_path:out1:RootOutput                            4.015e-06    8.4081e-06    2.5486e-05     7.406e-06    5.96995e-06      10     
end_path:out1:RootOutput(write)                      1.11102       1.81066       2.75177       1.92539      0.464506        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5827.26 MB
  Peak resident set size usage (VmHWM): 3634.38 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12348252
-rw-r--r-- 1 fermipro fermilab     231174 Dec 16 18:53 jobscript.log
-rw-r--r-- 1 fermipro fermilab        196 Dec 16 18:53 justin-processed-pfns.txt
-rw-r--r-- 1 fermipro fermilab        519 Dec 16 18:52 reco_protodunehd_hist.root
-rw-r--r-- 1 fermipro fermilab       3574 Dec 16 18:52 Pandora_Geometry.xml
-rw-r--r-- 1 fermipro fermilab     139264 Dec 16 18:52 mem.db
-rw-r--r-- 1 fermipro fermilab      28672 Dec 16 18:52 time.db
-rw-r--r-- 1 fermipro fermilab 6401620853 Dec 16 18:52 pdhd_prod_beam__261613_145_1_20251212T162029Z_gen_g4_IonScintPDExt.root_262728_100_1_20251215T220322Z_PDInt.root_262940_16_1_20251216T220034Z_detsim_reco1.root
-rw-r--r-- 1 fermipro fermilab   36385820 Dec 16 18:52 Pandora_Events.pndr
-rw-r--r-- 1 fermipro fermilab          0 Dec 16 18:26 debugprod.log
-rw-r--r-- 1 fermipro fermilab        519 Dec 16 18:25 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 fermipro fermilab 6206154773 Dec 16 18:25 temp_detsim.root
-rw-r--r-- 1 fermipro fermilab        123 Dec 16 17:00 all-input-dids.txt
justIN time: 2025-12-19 04:01:38 UTC       justIN version: 01.05.03