justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 263048.0@dunegpschedd02.fnal.gov

Jobsub ID263048.0@dunegpschedd02.fnal.gov
Workflow ID11235
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-17 09:55:24
SiteUS_UCSD
EntryCMSHTPC_T2_US_UCSD_gw6
Last heartbeat2025-12-17 11:56:04
From worker nodeHostnamemh-7662-3.t2.ucsd.edu
cpuinfoAMD EPYC 7662 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-17 09:55:59
Input filesusertests:pdhd_prod_beam__261613_37_1_20251212T161341Z_gen_g4_IonScintPDExt.root_262728_42_1_20251215T215847Z_PDInt.root
JobscriptExit code0
Real time1h (6993s)
CPU time1h (6387s = 91%)
Max RSS bytes6380765184 (6085 MiB)
Outputting started2025-12-17 11:52:33
Output filesusertests:pdhd_prod_beam__261613_37_1_20251212T161341Z_gen_g4_IonScintPDExt.root_262728_42_1_20251215T215847Z_PDInt.root_263048_0_1_20251217T095607Z_detsim_reco1.root
Finished2025-12-17 11:56:02
Saved logsjustin-logs:263048.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ener0":19159 [19159] ] cmm:[ bad:0 lf_noisy:56 ] 
[03:51:35.132] D [  glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=199 time=0 tick=512 with 37309 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":18150 [0] "wiener0":19159 [19159] ] cmm:[ bad:0 lf_noisy:56 ] input 1: frame: ident=199 time=0 tick=512 with 32419 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":14864 [0] "wiener1":17555 [17555] ] cmm:[ bad:0 ] input 2: frame: ident=199 time=0 tick=512 with 35953 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":15614 [0] "wiener2":20339 [20339] ] cmm:[ bad:0 ] input 3: frame: ident=199 time=0 tick=512 with 42754 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":18262 [0] "wiener3":24492 [24492] ] cmm:[ bad:0 ] output: frame: ident=199 time=0 tick=512 with 148435 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":18150 [0] "gauss1":14864 [0] "gauss2":15614 [0] "gauss3":18262 [0] "wiener0":19159 [19159] "wiener1":17555 [17555] "wiener2":20339 [20339] "wiener3":24492 [24492] ] cmm:[ bad:0 lf_noisy:56 ] 
Retagger: tagging trace set: wiener with 81545 traces, 81545 summary
Retagger: tagging trace set: gauss with 66890 traces, 0 summary
[03:51:35.142] D [  glue  ] sink frame: #199 @0 with 148435 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[03:51:35.142] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[03:51:35.142] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[03:51:35.142] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[03:51:35.142] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[03:51:35.142] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[03:51:35.142] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[03:51:35.142] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[03:51:35.142] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[03:51:35.142] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[03:51:35.142] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[03:51:35.142] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[03:51:35.142] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[03:51:35.142] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[03:51:35.143] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[03:51:35.143] D [  glue  ] frame sink sees EOS
[03:51:35.143] D [ pgraph ] <Pgrapher:> graph execution complete 
[03:51:35.143] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.98 sec
[03:51:35.143] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.02 sec
[03:51:35.143] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.1 sec
[03:51:35.143] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.07 sec
[03:51:35.143] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.58 sec
[03:51:35.143] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.47 sec
[03:51:35.143] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.32 sec
[03:51:35.143] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.09 sec
[03:51:35.143] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[03:51:35.143] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[03:51:35.143] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[03:51:35.143] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[03:51:35.143] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[03:51:35.143] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[03:51:35.143] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[03:51:35.143] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[03:51:35.143] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[03:51:35.143] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[03:51:35.143] I [ timer  ] Timer: Total node execution : 52.65999987535179 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 66890 traces tagged "gauss"
FrameSaver: q=1.99907e+06 n=1080482 tag=gauss
wclsFrameSaver: saving 81545 traces tagged "wiener"
FrameSaver: q=2.44669e+06 n=1046891 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 12277 XUs and 18874 XVs -> 467 XUVs
C:0 T:4 10 XUs and 5 XVs -> 1 XUVs
C:0 T:5 11442 XUs and 10564 XVs -> 289 XUVs
C:0 T:6 31447 XUs and 21650 XVs -> 765 XUVs
C:0 T:7 17 XUs and 12 XVs -> 4 XUVs
1526 XUVs total
1088 collection wire objects
1526 potential space points
Neighbour search...
17622 tests to find 10060 neighbours
Iterating with no regularization...
Begin: 2.1249e+07
0 2.07528e+07
1 2.07336e+07
Now with regularization...
Begin: 1.96572e+07
0 1.96547e+07
17-Dec-2025 03:52:32 PST  Closed output file "pdhd_prod_beam__261613_37_1_20251212T161341Z_gen_g4_IonScintPDExt.root_262728_42_1_20251215T215847Z_PDInt.root_263048_0_1_20251217T095607Z_detsim_reco1.root"
17-Dec-2025 03:52:32 PST  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           80.7963       100.813       115.898       105.098       11.5912        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.000828427   0.00356433     0.0052212    0.00369874    0.00146904       10     
reco:wclsdatahd:WireCellToolkit                      53.6282       56.6297       59.8007       56.5179       1.77618        10     
reco:wirefilter:WireFilter                          0.013442      0.018036      0.0271506     0.0175578    0.00374559       10     
reco:gaushit:GausHitFinder                          0.452578      0.662348      0.830537       0.67747      0.135101        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000149573   0.000272853   0.000493391   0.000276126   9.3215e-05       10     
reco:reco3d:SpacePointSolver                        0.195344      0.597551       1.42611      0.519164      0.369934        10     
reco:hitpdune:DisambigFromSpacePoints               0.222599      0.423352      0.650053      0.419486      0.151203        10     
reco:pandora:StandardPandora                         3.44588       10.1131       22.1313       10.5466       5.00984        10     
reco:pandoraWriter:StandardPandora                  0.134996       0.2081       0.281271      0.208919      0.0478969       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.488725      0.812097       1.35104      0.782399       0.28637        10     
reco:pandoraShower:LArPandoraShowerCreation         0.411622       1.01176       1.77869       0.94052      0.444414        10     
reco:pandoracalo:Calorimetry                         0.25618      0.459625      0.662489      0.471341      0.135146        10     
reco:pandoracalonosce:Calorimetry                   0.263903      0.436721      0.641114      0.442834      0.123547        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.384408      0.936601       1.71333      0.861035      0.435448        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.391722      0.969821       1.64734       0.93145      0.407397        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0391462     0.0611042     0.0856728     0.0629293     0.0163571       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0382746     0.0586293     0.0810603     0.0590191     0.0144691       10     
reco:emtrkmichelid:EmTrackMichelId                   17.8626       25.9362       35.3947       26.9238       4.88265        10     
[art]:TriggerResults:TriggerResultInserter          2.453e-05    4.5402e-05    9.0042e-05    3.6376e-05    2.31017e-05      10     
end_path:out1:RootOutput                            9.35e-06     1.19601e-05    2.797e-05    1.01005e-05   5.38301e-06      10     
end_path:out1:RootOutput(write)                      1.04545       1.42175       1.68199       1.48032      0.207907        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5770.77 MB
  Peak resident set size usage (VmHWM): 3521.65 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11177212
-rw-r--r--. 1 cuser cuser     230979 Dec 17 03:52 jobscript.log
-rw-r--r--. 1 cuser cuser        200 Dec 17 03:52 justin-processed-pfns.txt
-rw-r--r--. 1 cuser cuser        519 Dec 17 03:52 reco_protodunehd_hist.root
-rw-r--r--. 1 cuser cuser       3574 Dec 17 03:52 Pandora_Geometry.xml
-rw-r--r--. 1 cuser cuser     139264 Dec 17 03:52 mem.db
-rw-r--r--. 1 cuser cuser      28672 Dec 17 03:52 time.db
-rw-r--r--. 1 cuser cuser 5797830162 Dec 17 03:52 pdhd_prod_beam__261613_37_1_20251212T161341Z_gen_g4_IonScintPDExt.root_262728_42_1_20251215T215847Z_PDInt.root_263048_0_1_20251217T095607Z_detsim_reco1.root
-rw-r--r--. 1 cuser cuser   35458740 Dec 17 03:51 Pandora_Events.pndr
-rw-r--r--. 1 cuser cuser          0 Dec 17 03:34 debugprod.log
-rw-r--r--. 1 cuser cuser        519 Dec 17 03:33 detsim_single_protodunehd_hist.root
-rw-r--r--. 1 cuser cuser 5611717635 Dec 17 03:33 temp_detsim.root
-rw-r--r--. 1 cuser cuser        121 Dec 17 01:56 all-input-dids.txt
justIN time: 2025-12-18 20:20:52 UTC       justIN version: 01.05.03