justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 269119.5@dunegpschedd01.fnal.gov

Jobsub ID269119.5@dunegpschedd01.fnal.gov
Workflow ID11264
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 19:29:50
SiteUS_UCSD
EntryCMSHTPC_T2_US_UCSD_gw6
Last heartbeat2025-12-18 22:17:22
From worker nodeHostnamemh-7763-13.t2.ucsd.edu
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 19:31:20
Input filesusertests:pdhd_prod_beam__261612_86_1_20251212T160221Z_gen_g4_IonScintPDExt.root_262727_62_1_20251215T214729Z_PDInt.root
JobscriptExit code0
Real time2h (9752s)
CPU time1h (5486s = 56%)
Max RSS bytes6912679936 (6592 MiB)
Outputting started2025-12-18 22:13:53
Output filesusertests:pdhd_prod_beam__261612_86_1_20251212T160221Z_gen_g4_IonScintPDExt.root_262727_62_1_20251215T214729Z_PDInt.root_269119_5_1_20251218T193126Z_detsim_reco1.root
Finished2025-12-18 22:17:22
Saved logsjustin-logs:269119.5-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

uss0":24827 [0] "wiener0":27507 [27507] ] cmm:[ bad:0 lf_noisy:117 ] 
[14:13:32.250] D [  glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=179 time=0 tick=512 with 52334 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":24827 [0] "wiener0":27507 [27507] ] cmm:[ bad:0 lf_noisy:117 ] input 1: frame: ident=179 time=0 tick=512 with 30883 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":13885 [0] "wiener1":16998 [16998] ] cmm:[ bad:0 lf_noisy:1 ] input 2: frame: ident=179 time=0 tick=512 with 34453 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":15011 [0] "wiener2":19442 [19442] ] cmm:[ bad:0 ] input 3: frame: ident=179 time=0 tick=512 with 26580 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":12239 [0] "wiener3":14341 [14341] ] cmm:[ bad:0 ] output: frame: ident=179 time=0 tick=512 with 144250 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":24827 [0] "gauss1":13885 [0] "gauss2":15011 [0] "gauss3":12239 [0] "wiener0":27507 [27507] "wiener1":16998 [16998] "wiener2":19442 [19442] "wiener3":14341 [14341] ] cmm:[ bad:0 lf_noisy:118 ] 
Retagger: tagging trace set: wiener with 78288 traces, 78288 summary
Retagger: tagging trace set: gauss with 65962 traces, 0 summary
[14:13:32.257] D [  glue  ] sink frame: #179 @0 with 144250 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[14:13:32.257] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[14:13:32.257] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[14:13:32.257] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[14:13:32.257] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[14:13:32.257] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[14:13:32.257] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[14:13:32.257] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[14:13:32.257] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[14:13:32.257] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[14:13:32.257] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[14:13:32.257] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[14:13:32.257] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[14:13:32.257] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[14:13:32.257] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[14:13:32.257] D [  glue  ] frame sink sees EOS
[14:13:32.257] D [ pgraph ] <Pgrapher:> graph execution complete 
[14:13:32.257] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.35 sec
[14:13:32.257] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.47 sec
[14:13:32.257] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.97 sec
[14:13:32.257] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.92 sec
[14:13:32.257] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.56 sec
[14:13:32.257] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.54 sec
[14:13:32.257] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.52 sec
[14:13:32.257] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.51 sec
[14:13:32.257] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[14:13:32.257] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[14:13:32.257] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:13:32.257] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:13:32.257] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:13:32.257] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:13:32.257] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[14:13:32.257] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[14:13:32.257] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[14:13:32.257] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[14:13:32.257] I [ timer  ] Timer: Total node execution : 41.849999913945794 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 65962 traces tagged "gauss"
FrameSaver: q=1.84188e+06 n=996497 tag=gauss
wclsFrameSaver: saving 78288 traces tagged "wiener"
FrameSaver: q=2.3876e+06 n=993932 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 11332 XUs and 13057 XVs -> 498 XUVs
C:0 T:3 6 XUs and 22 XVs -> 4 XUVs
C:0 T:5 12296 XUs and 12129 XVs -> 519 XUVs
C:0 T:6 4509 XUs and 6540 XVs -> 435 XUVs
1456 XUVs total
972 collection wire objects
1456 potential space points
Neighbour search...
27998 tests to find 15280 neighbours
Iterating with no regularization...
Begin: 1.34189e+07
0 1.23584e+07
1 1.22935e+07
2 1.22876e+07
Now with regularization...
Begin: 1.13666e+07
0 1.13646e+07
18-Dec-2025 14:13:51 PST  Closed output file "pdhd_prod_beam__261612_86_1_20251212T160221Z_gen_g4_IonScintPDExt.root_262727_62_1_20251215T214729Z_PDInt.root_269119_5_1_20251218T193126Z_detsim_reco1.root"
18-Dec-2025 14:13:51 PST  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           60.6634       80.6561       97.1887       76.3433       12.3853        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.00103027    0.00349388    0.00530456    0.00353341    0.00131448       10     
reco:wclsdatahd:WireCellToolkit                      42.4882       46.3121       48.239        46.7089       1.79257        10     
reco:wirefilter:WireFilter                          0.0107144     0.0153325     0.0258861     0.0132069     0.0047995       10     
reco:gaushit:GausHitFinder                          0.314529      0.562654      0.983927      0.552521      0.212255        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000136298   0.000280145   0.000914316   0.000197787   0.000223628      10     
reco:reco3d:SpacePointSolver                        0.0979694     0.339718      0.640468      0.284026      0.178512        10     
reco:hitpdune:DisambigFromSpacePoints               0.112014      0.343252      0.797756      0.200858      0.242811        10     
reco:pandora:StandardPandora                         2.63445        6.587        11.9651       5.00391       3.34909        10     
reco:pandoraWriter:StandardPandora                  0.0883523     0.143974      0.226265      0.127672      0.0473837       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.352337      0.667348       1.42912      0.471223      0.339547        10     
reco:pandoraShower:LArPandoraShowerCreation         0.288364      0.669699       1.22997      0.521407      0.347737        10     
reco:pandoracalo:Calorimetry                        0.234854      0.411927      0.790378       0.30193       0.1833         10     
reco:pandoracalonosce:Calorimetry                    0.22217      0.380829      0.734181      0.290654      0.170046        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.258734      0.713897       1.31659      0.494725      0.424358        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.258701      0.685066       1.3431       0.489594      0.401836        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0229596     0.0471225     0.0813701     0.0430969     0.0208973       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0233441     0.0478254     0.0817156     0.0401977     0.0227588       10     
reco:emtrkmichelid:EmTrackMichelId                   12.422        21.6441       29.0266       20.4442       5.62511        10     
[art]:TriggerResults:TriggerResultInserter          2.186e-05    3.74863e-05   9.0698e-05    3.0755e-05    1.91258e-05      10     
end_path:out1:RootOutput                            3.36e-06      7.149e-06     3.058e-05     4.47e-06     7.84961e-06      10     
end_path:out1:RootOutput(write)                      0.6732        1.03878       1.58554       1.02607      0.298821        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6045.28 MB
  Peak resident set size usage (VmHWM): 3811.2 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11264404
-rw-r--r--. 1 cuser cuser     230968 Dec 18 14:13 jobscript.log
-rw-r--r--. 1 cuser cuser        200 Dec 18 14:13 justin-processed-pfns.txt
-rw-r--r--. 1 cuser cuser        519 Dec 18 14:13 reco_protodunehd_hist.root
-rw-r--r--. 1 cuser cuser       3574 Dec 18 14:13 Pandora_Geometry.xml
-rw-r--r--. 1 cuser cuser     139264 Dec 18 14:13 mem.db
-rw-r--r--. 1 cuser cuser      28672 Dec 18 14:13 time.db
-rw-r--r--. 1 cuser cuser 5840345742 Dec 18 14:13 pdhd_prod_beam__261612_86_1_20251212T160221Z_gen_g4_IonScintPDExt.root_262727_62_1_20251215T214729Z_PDInt.root_269119_5_1_20251218T193126Z_detsim_reco1.root
-rw-r--r--. 1 cuser cuser   33737680 Dec 18 14:13 Pandora_Events.pndr
-rw-r--r--. 1 cuser cuser          0 Dec 18 13:59 debugprod.log
-rw-r--r--. 1 cuser cuser        519 Dec 18 13:59 detsim_single_protodunehd_hist.root
-rw-r--r--. 1 cuser cuser 5660209603 Dec 18 13:59 temp_detsim.root
-rw-r--r--. 1 cuser cuser        121 Dec 18 11:31 all-input-dids.txt
justIN time: 2025-12-18 22:58:07 UTC       justIN version: 01.05.03