justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 269119.105@dunegpschedd01.fnal.gov

Jobsub ID269119.105@dunegpschedd01.fnal.gov
Workflow ID11264
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 19:29:50
SiteUS_UChicago
EntryEngage_US_MWT2_uct2_gk02_condce_mcore
Last heartbeat2025-12-18 23:21:18
From worker nodeHostnameiut2-c209.iu.edu
cpuinfoIntel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit86400 (24 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 20:48:45
Input filesusertests:pdhd_prod_beam__261612_64_1_20251212T155907Z_gen_g4_IonScintPDExt.root_262727_75_1_20251215T214820Z_PDInt.root
JobscriptExit code0
Real time2h (9014s)
CPU time2h (8726s = 96%)
Max RSS bytes7347556352 (7007 MiB)
Outputting started2025-12-18 23:18:59
Output filesusertests:pdhd_prod_beam__261612_64_1_20251212T155907Z_gen_g4_IonScintPDExt.root_262727_75_1_20251215T214820Z_PDInt.root_269119_105_1_20251218T204854Z_detsim_reco1.root
Finished2025-12-18 23:21:18
Saved logsjustin-logs:269119.105-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

13 [31513] ] cmm:[ bad:0 lf_noisy:210 ] 
[18:17:55.560] D [  glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=179 time=0 tick=512 with 59407 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":27894 [0] "wiener0":31513 [31513] ] cmm:[ bad:0 lf_noisy:210 ] input 1: frame: ident=179 time=0 tick=512 with 46945 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":20147 [0] "wiener1":26798 [26798] ] cmm:[ bad:0 ] input 2: frame: ident=179 time=0 tick=512 with 27159 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":12399 [0] "wiener2":14760 [14760] ] cmm:[ bad:0 ] input 3: frame: ident=179 time=0 tick=512 with 36289 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":16397 [0] "wiener3":19892 [19892] ] cmm:[ bad:0 ] output: frame: ident=179 time=0 tick=512 with 169800 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":27894 [0] "gauss1":20147 [0] "gauss2":12399 [0] "gauss3":16397 [0] "wiener0":31513 [31513] "wiener1":26798 [26798] "wiener2":14760 [14760] "wiener3":19892 [19892] ] cmm:[ bad:0 lf_noisy:210 ] 
Retagger: tagging trace set: wiener with 92963 traces, 92963 summary
Retagger: tagging trace set: gauss with 76837 traces, 0 summary
[18:17:55.592] D [  glue  ] sink frame: #179 @0 with 169800 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[18:17:55.593] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[18:17:55.593] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[18:17:55.593] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[18:17:55.593] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[18:17:55.593] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[18:17:55.593] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[18:17:55.593] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[18:17:55.593] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[18:17:55.593] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[18:17:55.593] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[18:17:55.593] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[18:17:55.593] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[18:17:55.593] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[18:17:55.593] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[18:17:55.593] D [  glue  ] frame sink sees EOS
[18:17:55.593] D [ pgraph ] <Pgrapher:> graph execution complete 
[18:17:55.593] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 29.28 sec
[18:17:55.593] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 19.4 sec
[18:17:55.593] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 19.36 sec
[18:17:55.593] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 19 sec
[18:17:55.593] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 4.59 sec
[18:17:55.593] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 4.55 sec
[18:17:55.593] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 4.18 sec
[18:17:55.593] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.53 sec
[18:17:55.593] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.05 sec
[18:17:55.593] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[18:17:55.593] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[18:17:55.593] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[18:17:55.593] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[18:17:55.593] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:17:55.593] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[18:17:55.593] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[18:17:55.593] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[18:17:55.593] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[18:17:55.593] I [ timer  ] Timer: Total node execution : 103.98000105842948 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 76837 traces tagged "gauss"
FrameSaver: q=2.36631e+06 n=1255905 tag=gauss
wclsFrameSaver: saving 92963 traces tagged "wiener"
FrameSaver: q=3.1081e+06 n=1220977 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 25904 XUs and 20330 XVs -> 805 XUVs
C:0 T:3 1 XUs and 7 XVs -> 1 XUVs
C:0 T:4 7 XUs and 19 XVs -> 4 XUVs
C:0 T:5 5836 XUs and 20777 XVs -> 845 XUVs
C:0 T:6 17228 XUs and 14287 XVs -> 507 XUVs
2162 XUVs total
1429 collection wire objects
2162 potential space points
Neighbour search...
96234 tests to find 51594 neighbours
Iterating with no regularization...
Begin: 1.62604e+07
0 1.533e+07
1 1.52945e+07
2 1.52897e+07
Now with regularization...
Begin: 1.43833e+07
0 1.43817e+07
18-Dec-2025 18:18:56 EST  Closed output file "pdhd_prod_beam__261612_64_1_20251212T155907Z_gen_g4_IonScintPDExt.root_262727_75_1_20251215T214820Z_PDInt.root_269119_105_1_20251218T204854Z_detsim_reco1.root"
18-Dec-2025 18:18:56 EST  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           155.199       191.014       245.596       185.092       27.7429        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.00131442    0.00596576     0.0102052    0.00645807    0.00294069       10     
reco:wclsdatahd:WireCellToolkit                      99.7863       107.427       114.203       107.517       4.63021        10     
reco:wirefilter:WireFilter                          0.0319592     0.0563725     0.0798006     0.0525529     0.015553        10     
reco:gaushit:GausHitFinder                          0.785421       1.12569       1.54639       1.09489      0.240096        10     
reco:nhitsfilter:NumberOfHitsFilter                0.00030632    0.000554063   0.000864817   0.000518484   0.000208209      10     
reco:reco3d:SpacePointSolver                        0.372886       1.16592       2.74405      0.848781      0.798144        10     
reco:hitpdune:DisambigFromSpacePoints               0.706973       1.22167       2.19423       1.17421      0.492691        10     
reco:pandora:StandardPandora                         8.01157       20.2809        43.63        17.4578       10.5762        10     
reco:pandoraWriter:StandardPandora                  0.204296      0.372342      0.512909      0.369842      0.0924357       10     
reco:pandoraTrack:LArPandoraTrackCreation            1.47962       2.7416        4.21313       2.58777      0.882905        10     
reco:pandoraShower:LArPandoraShowerCreation          1.16275       3.52706       6.85585       3.20821       2.0453         10     
reco:pandoracalo:Calorimetry                        0.671612       1.18864       1.73419       1.01568       0.38854        10     
reco:pandoracalonosce:Calorimetry                    0.6277        1.21063       1.93424      0.958278       0.51978        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.992393       2.94769       5.65522       2.80294       1.63354        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.995812       2.90204       6.22801       2.77251       1.78029        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.047841      0.0807821     0.151831      0.077571      0.0291862       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0442149     0.0870741      0.15271      0.0817315     0.0347949       10     
reco:emtrkmichelid:EmTrackMichelId                   32.6581       42.4552       55.5368       40.962        6.8645         10     
[art]:TriggerResults:TriggerResultInserter         3.6959e-05    7.48604e-05   0.000148047   6.9661e-05    2.97043e-05      10     
end_path:out1:RootOutput                            5.108e-06    1.03752e-05   3.3118e-05     7.331e-06    8.10631e-06      10     
end_path:out1:RootOutput(write)                      1.31322       2.1324        2.92281       1.99176      0.481426        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5988.61 MB
  Peak resident set size usage (VmHWM): 3824.83 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12581384
-rw-r--r--. 1 dune osgvo     231180 Dec 18 18:18 jobscript.log
-rw-r--r--. 1 dune osgvo        200 Dec 18 18:18 justin-processed-pfns.txt
-rw-r--r--. 1 dune osgvo        519 Dec 18 18:18 reco_protodunehd_hist.root
-rw-r--r--. 1 dune osgvo       3574 Dec 18 18:18 Pandora_Geometry.xml
-rw-r--r--. 1 dune osgvo     139264 Dec 18 18:18 mem.db
-rw-r--r--. 1 dune osgvo      28672 Dec 18 18:18 time.db
-rw-r--r--. 1 dune osgvo 6524762627 Dec 18 18:18 pdhd_prod_beam__261612_64_1_20251212T155907Z_gen_g4_IonScintPDExt.root_262727_75_1_20251215T214820Z_PDInt.root_269119_105_1_20251218T204854Z_detsim_reco1.root
-rw-r--r--. 1 dune osgvo   38641700 Dec 18 18:18 Pandora_Events.pndr
-rw-r--r--. 1 dune osgvo          0 Dec 18 17:43 debugprod.log
-rw-r--r--. 1 dune osgvo        519 Dec 18 17:43 detsim_single_protodunehd_hist.root
-rw-r--r--. 1 dune osgvo 6319474601 Dec 18 17:43 temp_detsim.root
-rw-r--r--. 1 dune osgvo        121 Dec 18 15:48 all-input-dids.txt
justIN time: 2025-12-19 13:10:59 UTC       justIN version: 01.05.03