justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 268935.0@dunegpschedd01.fnal.gov

Jobsub ID268935.0@dunegpschedd01.fnal.gov
Workflow ID11226
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-17 21:42:01
SiteUS_UCSD
EntryCMSHTPC_T2_US_UCSD_gw7
Last heartbeat2025-12-18 00:59:31
From worker nodeHostnamemh-7763-7.t2.ucsd.edu
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-17 21:42:33
Input filesusertests:pdhd_prod_beam__265010_24_1_20251208T181728Z_gen_g4_IonScintPDExt.root_260261_119_1_20251210T191025Z_PDInt.root
JobscriptExit code0
Real time3h (11609s)
CPU time1h (4215s = 36%)
Max RSS bytes7103283200 (6774 MiB)
Outputting started2025-12-18 00:56:03
Output filesusertests:pdhd_prod_beam__265010_24_1_20251208T181728Z_gen_g4_IonScintPDExt.root_260261_119_1_20251210T191025Z_PDInt.root_268935_0_1_20251217T214238Z_detsim_reco1.root
Finished2025-12-18 00:59:31
Saved logsjustin-logs:268935.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

[0] "wiener0":33732 [33732] ] cmm:[ bad:0 lf_noisy:161 ] 
[16:55:26.212] D [  glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=149 time=0 tick=512 with 63697 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":29965 [0] "wiener0":33732 [33732] ] cmm:[ bad:0 lf_noisy:161 ] input 1: frame: ident=149 time=0 tick=512 with 43477 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":19084 [0] "wiener1":24393 [24393] ] cmm:[ bad:0 lf_noisy:2 ] input 2: frame: ident=149 time=0 tick=512 with 65641 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":26798 [0] "wiener2":38843 [38843] ] cmm:[ bad:0 ] input 3: frame: ident=149 time=0 tick=512 with 36263 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":15544 [0] "wiener3":20719 [20719] ] cmm:[ bad:0 ] output: frame: ident=149 time=0 tick=512 with 209078 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":29965 [0] "gauss1":19084 [0] "gauss2":26798 [0] "gauss3":15544 [0] "wiener0":33732 [33732] "wiener1":24393 [24393] "wiener2":38843 [38843] "wiener3":20719 [20719] ] cmm:[ bad:0 lf_noisy:163 ] 
Retagger: tagging trace set: wiener with 117687 traces, 117687 summary
Retagger: tagging trace set: gauss with 91391 traces, 0 summary
[16:55:26.220] D [  glue  ] sink frame: #149 @0 with 209078 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[16:55:26.220] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[16:55:26.220] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[16:55:26.220] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[16:55:26.220] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[16:55:26.220] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[16:55:26.220] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[16:55:26.220] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[16:55:26.220] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[16:55:26.220] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[16:55:26.220] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[16:55:26.220] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[16:55:26.220] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[16:55:26.220] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[16:55:26.220] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[16:55:26.220] D [  glue  ] frame sink sees EOS
[16:55:26.220] D [ pgraph ] <Pgrapher:> graph execution complete 
[16:55:26.220] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.01 sec
[16:55:26.220] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.18 sec
[16:55:26.220] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.97 sec
[16:55:26.220] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.76 sec
[16:55:26.220] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.24 sec
[16:55:26.220] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.21 sec
[16:55:26.220] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.19 sec
[16:55:26.220] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.19 sec
[16:55:26.220] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[16:55:26.220] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:55:26.220] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[16:55:26.220] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:55:26.220] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:55:26.220] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[16:55:26.220] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[16:55:26.220] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[16:55:26.220] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[16:55:26.220] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[16:55:26.220] I [ timer  ] Timer: Total node execution : 35.76000023819506 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 91391 traces tagged "gauss"
FrameSaver: q=3.03745e+06 n=1464935 tag=gauss
wclsFrameSaver: saving 117687 traces tagged "wiener"
FrameSaver: q=3.86338e+06 n=1396577 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 23540 XUs and 35133 XVs -> 1665 XUVs
C:0 T:4 43 XUs and 29 XVs -> 0 XUVs
C:0 T:5 79645 XUs and 48317 XVs -> 1634 XUVs
C:0 T:6 11940 XUs and 16288 XVs -> 387 XUVs
3686 XUVs total
2111 collection wire objects
3686 potential space points
Neighbour search...
78864 tests to find 47004 neighbours
Iterating with no regularization...
Begin: 3.07675e+07
0 2.94059e+07
1 2.9309e+07
2 2.92982e+07
Now with regularization...
Begin: 2.72928e+07
0 2.72887e+07
17-Dec-2025 16:56:02 PST  Closed output file "pdhd_prod_beam__265010_24_1_20251208T181728Z_gen_g4_IonScintPDExt.root_260261_119_1_20251210T191025Z_PDInt.root_268935_0_1_20251217T214238Z_detsim_reco1.root"
17-Dec-2025 16:56:02 PST  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           55.2688       66.0104       74.5093       66.4704       6.8846         10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.000621998   0.00271359    0.00459627    0.00278856    0.00115399       10     
reco:wclsdatahd:WireCellToolkit                      37.3063       37.9875       38.7296       38.0432      0.460428        10     
reco:wirefilter:WireFilter                         0.00987601     0.012169      0.0142426     0.0121881    0.00165936       10     
reco:gaushit:GausHitFinder                          0.279049      0.437101      0.546599      0.468792      0.0927527       10     
reco:nhitsfilter:NumberOfHitsFilter                0.000130479   0.000180376   0.000294479   0.000173185   4.82691e-05      10     
reco:reco3d:SpacePointSolver                        0.199274      0.334777      0.484914      0.365495      0.0948053       10     
reco:hitpdune:DisambigFromSpacePoints               0.116909      0.292972      0.478492      0.241631      0.138125        10     
reco:pandora:StandardPandora                         2.98944       5.78596       8.12012       6.03598       1.73927        10     
reco:pandoraWriter:StandardPandora                  0.0893929     0.125433      0.159396      0.132194      0.0201759       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.329072      0.583066       1.10191      0.518413       0.20892        10     
reco:pandoraShower:LArPandoraShowerCreation         0.267103      0.662711       1.04839       0.62956       0.26868        10     
reco:pandoracalo:Calorimetry                        0.232283      0.357299      0.584103      0.326604      0.0981396       10     
reco:pandoracalonosce:Calorimetry                   0.222021      0.347233      0.563277      0.320534      0.0950778       10     
reco:pandoraShowercalo:ShowerCalorimetry            0.255292      0.696188       1.03726      0.681409      0.267629        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.252447      0.687616       1.03676      0.673064      0.267671        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0233016     0.034418      0.0452777     0.0347679    0.00669911       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0232115     0.0337709     0.0442088     0.0332167    0.00666129       10     
reco:emtrkmichelid:EmTrackMichelId                   11.0387       16.7555       21.1448       17.1832       3.65825        10     
[art]:TriggerResults:TriggerResultInserter          1.375e-05    2.60277e-05   6.0039e-05    2.2745e-05    1.26146e-05      10     
end_path:out1:RootOutput                            3.92e-06      9.734e-06     2.001e-05     7.985e-06    4.70525e-06      10     
end_path:out1:RootOutput(write)                     0.597805      0.841934       1.02529      0.844489      0.144882        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5982.91 MB
  Peak resident set size usage (VmHWM): 3702.98 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12203716
-rw-r--r--. 1 cuser cuser     231504 Dec 17 16:56 jobscript.log
-rw-r--r--. 1 cuser cuser        201 Dec 17 16:56 justin-processed-pfns.txt
-rw-r--r--. 1 cuser cuser        519 Dec 17 16:56 reco_protodunehd_hist.root
-rw-r--r--. 1 cuser cuser       3574 Dec 17 16:56 Pandora_Geometry.xml
-rw-r--r--. 1 cuser cuser     139264 Dec 17 16:56 mem.db
-rw-r--r--. 1 cuser cuser      28672 Dec 17 16:56 time.db
-rw-r--r--. 1 cuser cuser 6327016301 Dec 17 16:56 pdhd_prod_beam__265010_24_1_20251208T181728Z_gen_g4_IonScintPDExt.root_260261_119_1_20251210T191025Z_PDInt.root_268935_0_1_20251217T214238Z_detsim_reco1.root
-rw-r--r--. 1 cuser cuser   35968590 Dec 17 16:55 Pandora_Events.pndr
-rw-r--r--. 1 cuser cuser          0 Dec 17 16:43 debugprod.log
-rw-r--r--. 1 cuser cuser        519 Dec 17 16:43 detsim_single_protodunehd_hist.root
-rw-r--r--. 1 cuser cuser 6133152507 Dec 17 16:43 temp_detsim.root
-rw-r--r--. 1 cuser cuser        122 Dec 17 13:42 all-input-dids.txt
justIN time: 2025-12-18 20:21:58 UTC       justIN version: 01.05.03