justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 269119.108@dunegpschedd01.fnal.gov

Jobsub ID269119.108@dunegpschedd01.fnal.gov
Workflow ID11264
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 19:29:50
SiteUS_UChicago
EntryEngage_US_MWT2_uct2_condce_mcore
Last heartbeat2025-12-18 22:24:32
From worker nodeHostnameuct2-c646.mwt2.org
cpuinfoAMD EPYC 7302 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit86400 (24 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 20:49:04
Input filesusertests:pdhd_prod_beam__261612_102_1_20251212T160405Z_gen_g4_IonScintPDExt.root_268535_13_1_20251215T214450Z_PDInt.root
JobscriptExit code0
Real time1h (5614s)
CPU time1h (5427s = 96%)
Max RSS bytes7244529664 (6908 MiB)
Outputting started2025-12-18 22:22:39
Output filesusertests:pdhd_prod_beam__261612_102_1_20251212T160405Z_gen_g4_IonScintPDExt.root_268535_13_1_20251215T214450Z_PDInt.root_269119_108_1_20251218T204914Z_detsim_reco1.root
Finished2025-12-18 22:24:32
Saved logsjustin-logs:269119.108-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

9923 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":23111 [0] "wiener0":26812 [26812] ] cmm:[ bad:0 lf_noisy:52 ] input 1: frame: ident=179 time=0 tick=512 with 61727 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":25391 [0] "wiener1":36336 [36336] ] cmm:[ bad:0 ] input 2: frame: ident=179 time=0 tick=512 with 50447 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":20902 [0] "wiener2":29545 [29545] ] cmm:[ bad:0 ] input 3: frame: ident=179 time=0 tick=512 with 32415 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":14279 [0] "wiener3":18136 [18136] ] cmm:[ bad:0 lf_noisy:1 ] output: frame: ident=179 time=0 tick=512 with 194512 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":23111 [0] "gauss1":25391 [0] "gauss2":20902 [0] "gauss3":14279 [0] "wiener0":26812 [26812] "wiener1":36336 [36336] "wiener2":29545 [29545] "wiener3":18136 [18136] ] cmm:[ bad:0 lf_noisy:53 ] 
Retagger: tagging trace set: wiener with 110829 traces, 110829 summary
Retagger: tagging trace set: gauss with 83683 traces, 0 summary
[16:21:40.101] D [  glue  ] sink frame: #179 @0 with 194512 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[16:21:40.101] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[16:21:40.101] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[16:21:40.101] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[16:21:40.101] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[16:21:40.101] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[16:21:40.101] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[16:21:40.101] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[16:21:40.101] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[16:21:40.101] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[16:21:40.101] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[16:21:40.101] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[16:21:40.101] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[16:21:40.101] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[16:21:40.101] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[16:21:40.101] D [  glue  ] frame sink sees EOS
[16:21:40.101] D [ pgraph ] <Pgrapher:> graph execution complete 
[16:21:40.101] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 13.05 sec
[16:21:40.101] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.51 sec
[16:21:40.101] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.81 sec
[16:21:40.101] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.97 sec
[16:21:40.101] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.78 sec
[16:21:40.101] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.78 sec
[16:21:40.101] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.78 sec
[16:21:40.101] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.69 sec
[16:21:40.101] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.02 sec
[16:21:40.101] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[16:21:40.101] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0.01 sec
[16:21:40.101] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:21:40.101] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:21:40.101] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:21:40.101] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[16:21:40.101] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[16:21:40.101] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[16:21:40.101] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[16:21:40.102] I [ timer  ] Timer: Total node execution : 50.41000059992075 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 83683 traces tagged "gauss"
FrameSaver: q=2.87046e+06 n=1331234 tag=gauss
wclsFrameSaver: saving 110829 traces tagged "wiener"
FrameSaver: q=3.55431e+06 n=1274668 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 66727 XUs and 117047 XVs -> 2808 XUVs
C:0 T:3 29 XUs and 22 XVs -> 1 XUVs
C:0 T:4 8 XUs and 20 XVs -> 1 XUVs
C:0 T:5 32317 XUs and 37243 XVs -> 770 XUVs
C:0 T:6 22956 XUs and 53338 XVs -> 1919 XUVs
C:0 T:7 35 XUs and 73 XVs -> 3 XUVs
5502 XUVs total
2581 collection wire objects
5502 potential space points
Neighbour search...
278750 tests to find 125760 neighbours
Iterating with no regularization...
Begin: 4.27437e+07
0 3.83731e+07
1 3.80661e+07
2 3.80208e+07
3 3.80083e+07
Now with regularization...
Begin: 3.58405e+07
0 3.58334e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
18-Dec-2025 16:22:37 CST  Closed output file "pdhd_prod_beam__261612_102_1_20251212T160405Z_gen_g4_IonScintPDExt.root_268535_13_1_20251215T214450Z_PDInt.root_269119_108_1_20251218T204914Z_detsim_reco1.root"
18-Dec-2025 16:22:37 CST  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           71.1038       90.7851       108.751       88.6238       11.0318        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.00131291     0.0038831    0.00717752    0.00384688    0.00154869       10     
reco:wclsdatahd:WireCellToolkit                      47.0492       50.4958       57.8257       49.3413       3.13583        10     
reco:wirefilter:WireFilter                          0.0114195     0.0168004     0.0230094     0.0167292    0.00344591       10     
reco:gaushit:GausHitFinder                          0.404854      0.558109      0.641229      0.570129      0.0706301       10     
reco:nhitsfilter:NumberOfHitsFilter                0.00024973    0.000337495   0.000591072   0.000308731   9.17724e-05      10     
reco:reco3d:SpacePointSolver                        0.162319      0.745811       1.40367      0.638095      0.373755        10     
reco:hitpdune:DisambigFromSpacePoints               0.179802       0.46641      0.664058      0.495214      0.142867        10     
reco:pandora:StandardPandora                         4.18081       8.57075       12.5246       8.18004       2.8475         10     
reco:pandoraWriter:StandardPandora                   0.16638      0.222449      0.288095       0.22637      0.0390225       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.517736      0.908449       1.39528      0.863264      0.249503        10     
reco:pandoraShower:LArPandoraShowerCreation         0.473482      0.909503       1.41744      0.909664      0.316754        10     
reco:pandoracalo:Calorimetry                        0.257896      0.478806      0.622709      0.463373      0.108795        10     
reco:pandoracalonosce:Calorimetry                   0.269196      0.426017      0.596514      0.407557      0.103474        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.524307      0.852623       1.20594      0.891595      0.207012        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.479619      0.860218       1.42129      0.860132      0.274276        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0347554     0.0518207     0.0680629     0.0498993     0.0114283       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0345132     0.0555371     0.0873775     0.0516997     0.015498        10     
reco:emtrkmichelid:EmTrackMichelId                   15.2676       23.8579       35.3631       23.7146       5.69055        10     
[art]:TriggerResults:TriggerResultInserter          1.602e-05    4.02345e-05   0.000131026   3.22305e-05   3.12059e-05      10     
end_path:out1:RootOutput                            7.684e-06    1.30754e-05   3.2822e-05    1.05045e-05   7.27197e-06      10     
end_path:out1:RootOutput(write)                      1.01734       1.24057       1.48446       1.16665      0.173254        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6142.85 MB
  Peak resident set size usage (VmHWM): 3962.93 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12157316
-rw-r--r--. 1 dune osgvo     231655 Dec 18 16:22 jobscript.log
-rw-r--r--. 1 dune osgvo        201 Dec 18 16:22 justin-processed-pfns.txt
-rw-r--r--. 1 dune osgvo        519 Dec 18 16:22 reco_protodunehd_hist.root
-rw-r--r--. 1 dune osgvo       3574 Dec 18 16:22 Pandora_Geometry.xml
-rw-r--r--. 1 dune osgvo     139264 Dec 18 16:22 mem.db
-rw-r--r--. 1 dune osgvo      28672 Dec 18 16:22 time.db
-rw-r--r--. 1 dune osgvo 6305186567 Dec 18 16:22 pdhd_prod_beam__261612_102_1_20251212T160405Z_gen_g4_IonScintPDExt.root_268535_13_1_20251215T214450Z_PDInt.root_269119_108_1_20251218T204914Z_detsim_reco1.root
-rw-r--r--. 1 dune osgvo   38873580 Dec 18 16:21 Pandora_Events.pndr
-rw-r--r--. 1 dune osgvo          0 Dec 18 16:05 debugprod.log
-rw-r--r--. 1 dune osgvo        519 Dec 18 16:05 detsim_single_protodunehd_hist.root
-rw-r--r--. 1 dune osgvo 6104582764 Dec 18 16:05 temp_detsim.root
-rw-r--r--. 1 dune osgvo        122 Dec 18 14:49 all-input-dids.txt
justIN time: 2025-12-19 14:57:09 UTC       justIN version: 01.05.03