justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 269119.88@dunegpschedd01.fnal.gov

Jobsub ID269119.88@dunegpschedd01.fnal.gov
Workflow ID11264
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 19:29:50
SiteUS_UChicago
EntryEngage_US_MWT2_iut2_condce_mcore
Last heartbeat2025-12-18 23:05:34
From worker nodeHostnameiut2-c287.iu.edu
cpuinfoIntel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit86400 (24 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 20:33:04
Input filesusertests:pdhd_prod_beam__261612_48_1_20251212T155712Z_gen_g4_IonScintPDExt.root_262727_104_1_20251215T215342Z_PDInt.root
JobscriptExit code0
Real time2h (8388s)
CPU time2h (8210s = 97%)
Max RSS bytes7069233152 (6741 MiB)
Outputting started2025-12-18 22:52:52
Output filesusertests:pdhd_prod_beam__261612_48_1_20251212T155712Z_gen_g4_IonScintPDExt.root_262727_104_1_20251215T215342Z_PDInt.root_269119_88_1_20251218T203311Z_detsim_reco1.root
Finished2025-12-18 23:05:34
Saved logsjustin-logs:269119.88-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

gauss0":17980 [0] "wiener0":20177 [20177] ] cmm:[ bad:0 lf_noisy:39 ] 
[17:51:54.427] D [  glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=179 time=0 tick=512 with 38157 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":17980 [0] "wiener0":20177 [20177] ] cmm:[ bad:0 lf_noisy:39 ] input 1: frame: ident=179 time=0 tick=512 with 39670 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":17476 [0] "wiener1":22194 [22194] ] cmm:[ bad:0 ] input 2: frame: ident=179 time=0 tick=512 with 33241 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":15058 [0] "wiener2":18183 [18183] ] cmm:[ bad:0 lf_noisy:1 ] input 3: frame: ident=179 time=0 tick=512 with 36909 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":16526 [0] "wiener3":20383 [20383] ] cmm:[ bad:0 ] output: frame: ident=179 time=0 tick=512 with 147977 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":17980 [0] "gauss1":17476 [0] "gauss2":15058 [0] "gauss3":16526 [0] "wiener0":20177 [20177] "wiener1":22194 [22194] "wiener2":18183 [18183] "wiener3":20383 [20383] ] cmm:[ bad:0 lf_noisy:40 ] 
Retagger: tagging trace set: wiener with 80937 traces, 80937 summary
Retagger: tagging trace set: gauss with 67040 traces, 0 summary
[17:51:54.453] D [  glue  ] sink frame: #179 @0 with 147977 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[17:51:54.453] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[17:51:54.453] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[17:51:54.453] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[17:51:54.453] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[17:51:54.453] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[17:51:54.454] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[17:51:54.454] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[17:51:54.454] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[17:51:54.454] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[17:51:54.454] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[17:51:54.454] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[17:51:54.454] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[17:51:54.454] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[17:51:54.454] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[17:51:54.454] D [  glue  ] frame sink sees EOS
[17:51:54.454] D [ pgraph ] <Pgrapher:> graph execution complete 
[17:51:54.454] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 19.54 sec
[17:51:54.454] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 13.76 sec
[17:51:54.454] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 13.61 sec
[17:51:54.454] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 13.5 sec
[17:51:54.454] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 4.6 sec
[17:51:54.454] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.57 sec
[17:51:54.454] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.31 sec
[17:51:54.454] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.07 sec
[17:51:54.454] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.04 sec
[17:51:54.454] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[17:51:54.454] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[17:51:54.454] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:51:54.454] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[17:51:54.454] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:51:54.454] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[17:51:54.454] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[17:51:54.454] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[17:51:54.454] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[17:51:54.454] I [ timer  ] Timer: Total node execution : 75.02000051364303 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 67040 traces tagged "gauss"
FrameSaver: q=1.7471e+06 n=1049037 tag=gauss
wclsFrameSaver: saving 80937 traces tagged "wiener"
FrameSaver: q=2.17191e+06 n=1015238 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 24963 XUs and 26886 XVs -> 681 XUVs
C:0 T:4 9 XUs and 13 XVs -> 1 XUVs
C:0 T:5 14509 XUs and 23177 XVs -> 405 XUVs
C:0 T:6 17286 XUs and 19847 XVs -> 391 XUVs
1478 XUVs total
943 collection wire objects
1478 potential space points
Neighbour search...
24852 tests to find 14432 neighbours
Iterating with no regularization...
Begin: 1.80302e+07
0 1.75712e+07
1 1.75495e+07
2 1.75466e+07
Now with regularization...
Begin: 1.66021e+07
0 1.66005e+07
18-Dec-2025 17:52:50 EST  Closed output file "pdhd_prod_beam__261612_48_1_20251212T155712Z_gen_g4_IonScintPDExt.root_262727_104_1_20251215T215342Z_PDInt.root_269119_88_1_20251218T203311Z_detsim_reco1.root"
18-Dec-2025 17:52:50 EST  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           129.54        152.375       183.708       152.271       17.8265        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.00135375    0.00479569     0.0068237    0.00507019    0.00178032       10     
reco:wclsdatahd:WireCellToolkit                      77.7127       88.158        98.0649       87.9979       5.52877        10     
reco:wirefilter:WireFilter                          0.040345      0.0511177     0.0608413     0.0508704    0.00602553       10     
reco:gaushit:GausHitFinder                          0.808722       1.08142       1.50534       1.04231      0.197278        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000435978   0.000543719   0.000892045   0.000506454   0.000130633      10     
reco:reco3d:SpacePointSolver                         0.25311       1.06027       1.89421      0.938046      0.504944        10     
reco:hitpdune:DisambigFromSpacePoints               0.326943      0.867877        1.647       0.814174      0.412607        10     
reco:pandora:StandardPandora                         7.76751       14.3604       21.8563       13.3407       4.68312        10     
reco:pandoraWriter:StandardPandora                  0.269781      0.376332      0.499663      0.362125      0.0676665       10     
reco:pandoraTrack:LArPandoraTrackCreation            1.44434       2.74258       3.97824       2.75713      0.665255        10     
reco:pandoraShower:LArPandoraShowerCreation          1.24219       2.61956       4.08772       2.62949      0.929628        10     
reco:pandoracalo:Calorimetry                        0.634371       1.3591        2.16085       1.25097      0.413973        10     
reco:pandoracalonosce:Calorimetry                   0.748579       1.34249       1.73171       1.42398      0.329021        10     
reco:pandoraShowercalo:ShowerCalorimetry             1.12484       2.62001       4.20037       2.81072       1.06095        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        1.28873       2.42617       4.15079       2.07169      0.903136        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0534109     0.096869      0.135061      0.0968783     0.0290792       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0521553     0.0839823     0.124427      0.084142      0.0256248       10     
reco:emtrkmichelid:EmTrackMichelId                   22.646        31.2933       42.5882       31.853        5.70278        10     
[art]:TriggerResults:TriggerResultInserter         5.3011e-05    8.6057e-05    0.00016157    8.1771e-05    2.86873e-05      10     
end_path:out1:RootOutput                            8.253e-06    1.48401e-05    3.535e-05    1.13715e-05   7.77657e-06      10     
end_path:out1:RootOutput(write)                      1.3524        1.72638       2.51975       1.51314      0.421799        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5782.1 MB
  Peak resident set size usage (VmHWM): 3605.3 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12387812
-rw-r--r--. 1 dune osgvo     231019 Dec 18 17:52 jobscript.log
-rw-r--r--. 1 dune osgvo        201 Dec 18 17:52 justin-processed-pfns.txt
-rw-r--r--. 1 dune osgvo        519 Dec 18 17:52 reco_protodunehd_hist.root
-rw-r--r--. 1 dune osgvo       3574 Dec 18 17:52 Pandora_Geometry.xml
-rw-r--r--. 1 dune osgvo     139264 Dec 18 17:52 mem.db
-rw-r--r--. 1 dune osgvo      28672 Dec 18 17:52 time.db
-rw-r--r--. 1 dune osgvo 6418626413 Dec 18 17:52 pdhd_prod_beam__261612_48_1_20251212T155712Z_gen_g4_IonScintPDExt.root_262727_104_1_20251215T215342Z_PDInt.root_269119_88_1_20251218T203311Z_detsim_reco1.root
-rw-r--r--. 1 dune osgvo   33892010 Dec 18 17:52 Pandora_Events.pndr
-rw-r--r--. 1 dune osgvo          0 Dec 18 17:24 debugprod.log
-rw-r--r--. 1 dune osgvo        519 Dec 18 17:24 detsim_single_protodunehd_hist.root
-rw-r--r--. 1 dune osgvo 6232158118 Dec 18 17:24 temp_detsim.root
-rw-r--r--. 1 dune osgvo        122 Dec 18 15:33 all-input-dids.txt
justIN time: 2025-12-19 13:10:55 UTC       justIN version: 01.05.03