justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 263305.11@dunegpschedd02.fnal.gov

Jobsub ID263305.11@dunegpschedd02.fnal.gov
Workflow ID11264
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 19:31:07
SiteUS_UCSD
EntryCMSHTPC_T2_US_UCSD_gw7
Last heartbeat2025-12-18 22:48:23
From worker nodeHostnamemh-7662-3.t2.ucsd.edu
cpuinfoAMD EPYC 7662 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 20:27:28
Input filesusertests:pdhd_prod_beam__261612_16_1_20251212T155514Z_gen_g4_IonScintPDExt.root_262727_16_1_20251215T214304Z_PDInt.root
JobscriptExit code0
Real time2h (8196s)
CPU time2h (7473s = 91%)
Max RSS bytes6790455296 (6475 MiB)
Outputting started2025-12-18 22:44:04
Output filesusertests:pdhd_prod_beam__261612_16_1_20251212T155514Z_gen_g4_IonScintPDExt.root_262727_16_1_20251215T214304Z_PDInt.root_263305_11_1_20251218T202736Z_detsim_reco1.root
Finished2025-12-18 22:48:23
Saved logsjustin-logs:263305.11-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

sy:272 ] 
[14:43:16.810] D [  glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=179 time=0 tick=512 with 60912 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":28460 [0] "wiener0":32452 [32452] ] cmm:[ bad:0 lf_noisy:272 ] input 1: frame: ident=179 time=0 tick=512 with 32033 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":14127 [0] "wiener1":17906 [17906] ] cmm:[ bad:0 ] input 2: frame: ident=179 time=0 tick=512 with 41557 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":18110 [0] "wiener2":23447 [23447] ] cmm:[ bad:0 lf_noisy:10 ] input 3: frame: ident=179 time=0 tick=512 with 36298 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":15592 [0] "wiener3":20706 [20706] ] cmm:[ bad:0 ] output: frame: ident=179 time=0 tick=512 with 170800 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":28460 [0] "gauss1":14127 [0] "gauss2":18110 [0] "gauss3":15592 [0] "wiener0":32452 [32452] "wiener1":17906 [17906] "wiener2":23447 [23447] "wiener3":20706 [20706] ] cmm:[ bad:0 lf_noisy:282 ] 
Retagger: tagging trace set: wiener with 94511 traces, 94511 summary
Retagger: tagging trace set: gauss with 76289 traces, 0 summary
[14:43:16.829] D [  glue  ] sink frame: #179 @0 with 170800 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[14:43:16.829] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[14:43:16.829] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[14:43:16.829] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[14:43:16.829] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[14:43:16.829] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[14:43:16.829] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[14:43:16.829] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[14:43:16.829] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[14:43:16.829] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[14:43:16.830] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[14:43:16.830] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[14:43:16.830] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[14:43:16.830] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[14:43:16.830] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[14:43:16.830] D [  glue  ] frame sink sees EOS
[14:43:16.830] D [ pgraph ] <Pgrapher:> graph execution complete 
[14:43:16.830] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 15.94 sec
[14:43:16.830] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.36 sec
[14:43:16.830] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.8 sec
[14:43:16.830] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.8 sec
[14:43:16.830] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 4.28 sec
[14:43:16.830] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.72 sec
[14:43:16.830] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.63 sec
[14:43:16.830] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.39 sec
[14:43:16.830] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.03 sec
[14:43:16.830] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.02 sec
[14:43:16.830] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:43:16.830] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[14:43:16.830] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:43:16.830] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:43:16.830] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[14:43:16.830] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[14:43:16.830] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[14:43:16.830] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[14:43:16.830] I [ timer  ] Timer: Total node execution : 64.97000007517636 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 76289 traces tagged "gauss"
FrameSaver: q=2.34939e+06 n=1187746 tag=gauss
wclsFrameSaver: saving 94511 traces tagged "wiener"
FrameSaver: q=2.94234e+06 n=1142845 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 11450 XUs and 25738 XVs -> 561 XUVs
C:0 T:3 13 XUs and 5 XVs -> 0 XUVs
C:0 T:4 1 XUs and 13 XVs -> 1 XUVs
C:0 T:5 31258 XUs and 30860 XVs -> 1973 XUVs
C:0 T:6 12340 XUs and 18414 XVs -> 628 XUVs
3163 XUVs total
1835 collection wire objects
3163 potential space points
Neighbour search...
275977 tests to find 88826 neighbours
Iterating with no regularization...
Begin: 2.84833e+07
0 2.55347e+07
1 2.52713e+07
2 2.52388e+07
3 2.52313e+07
Now with regularization...
Begin: 2.2873e+07
0 2.2854e+07
18-Dec-2025 14:44:02 PST  Closed output file "pdhd_prod_beam__261612_16_1_20251212T155514Z_gen_g4_IonScintPDExt.root_262727_16_1_20251215T214304Z_PDInt.root_263305_11_1_20251218T202736Z_detsim_reco1.root"
18-Dec-2025 14:44:02 PST  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           99.3413       120.991       149.589       116.42        16.3311        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                              0.001651     0.00450945    0.00753622    0.00432034    0.00175157       10     
reco:wclsdatahd:WireCellToolkit                      61.8138       68.623        77.074        67.9776       5.23866        10     
reco:wirefilter:WireFilter                          0.0173623     0.0270176     0.0484884     0.0252092    0.00939141       10     
reco:gaushit:GausHitFinder                           0.53111      0.820054       1.61456      0.736316      0.292468        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000192394   0.000342545   0.000598982   0.000307591   0.00011811       10     
reco:reco3d:SpacePointSolver                        0.193818      0.736395       1.93326      0.547542      0.514595        10     
reco:hitpdune:DisambigFromSpacePoints               0.173511      0.604146       1.42915      0.544427      0.347309        10     
reco:pandora:StandardPandora                         5.04933       11.9374       22.2983       10.5894       5.2859         10     
reco:pandoraWriter:StandardPandora                  0.155464      0.252309       0.40462      0.241765      0.0659469       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.696913       1.26355       2.16114       1.14648       0.50286        10     
reco:pandoraShower:LArPandoraShowerCreation         0.468953       1.22756       2.62023      0.954698      0.737333        10     
reco:pandoracalo:Calorimetry                        0.391964       0.7294        1.22844      0.652201      0.258195        10     
reco:pandoracalonosce:Calorimetry                    0.37976      0.693199       1.16508      0.617104      0.253074        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.409805       1.23086       2.59055       1.07753       0.67498        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.409528       1.16331       2.6125       0.976654       0.64609        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0408959     0.0665354     0.109852      0.0558164     0.0213993       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0409318     0.0651398     0.102381      0.0556837     0.0198513       10     
reco:emtrkmichelid:EmTrackMichelId                   20.1644       29.8099       38.8111       29.7417       7.14802        10     
[art]:TriggerResults:TriggerResultInserter         2.6011e-05    5.26232e-05   0.000144803   3.92555e-05   3.40478e-05      10     
end_path:out1:RootOutput                            4.381e-06    9.0431e-06     4.359e-05     4.92e-06     1.15629e-05      10     
end_path:out1:RootOutput(write)                      1.28186       1.66524       2.2175        1.62166      0.332129        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6104.2 MB
  Peak resident set size usage (VmHWM): 3893.8 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12053024
-rw-r--r--. 1 cuser cuser     231632 Dec 18 14:44 jobscript.log
-rw-r--r--. 1 cuser cuser        200 Dec 18 14:44 justin-processed-pfns.txt
-rw-r--r--. 1 cuser cuser        519 Dec 18 14:44 reco_protodunehd_hist.root
-rw-r--r--. 1 cuser cuser       3574 Dec 18 14:44 Pandora_Geometry.xml
-rw-r--r--. 1 cuser cuser     139264 Dec 18 14:44 mem.db
-rw-r--r--. 1 cuser cuser      28672 Dec 18 14:44 time.db
-rw-r--r--. 1 cuser cuser 6247071926 Dec 18 14:44 pdhd_prod_beam__261612_16_1_20251212T155514Z_gen_g4_IonScintPDExt.root_262727_16_1_20251215T214304Z_PDInt.root_263305_11_1_20251218T202736Z_detsim_reco1.root
-rw-r--r--. 1 cuser cuser   34782130 Dec 18 14:43 Pandora_Events.pndr
-rw-r--r--. 1 cuser cuser          0 Dec 18 14:21 debugprod.log
-rw-r--r--. 1 cuser cuser        519 Dec 18 14:21 detsim_single_protodunehd_hist.root
-rw-r--r--. 1 cuser cuser 6059916686 Dec 18 14:21 temp_detsim.root
-rw-r--r--. 1 cuser cuser        121 Dec 18 12:27 all-input-dids.txt
justIN time: 2025-12-18 23:05:59 UTC       justIN version: 01.05.03