justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 263135.0@dunegpschedd02.fnal.gov

Jobsub ID263135.0@dunegpschedd02.fnal.gov
Workflow ID11239
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-17 21:40:01
SiteCA_SFU
EntryDUNE_CA_SFU_lcg-ce3
Last heartbeat2025-12-17 22:27:01
From worker nodeHostnamefc30557.int.fir.alliancecan.ca
cpuinfoAMD EPYC 9655 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit84598 (23 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-17 21:41:39
Input filesusertests:pdhd_prod_beam__267379_8_1_20251212T160751Z_gen_g4_IonScintPDExt.root_262726_75_1_20251215T212431Z_PDInt.root
JobscriptExit code0
Real time44m (2668s)
CPU time33m (1999s = 74%)
Max RSS bytes5999292416 (5721 MiB)
Outputting started2025-12-17 22:26:08
Output filesusertests:pdhd_prod_beam__267379_8_1_20251212T160751Z_gen_g4_IonScintPDExt.root_262726_75_1_20251215T212431Z_PDInt.root_263135_0_1_20251217T214142Z_detsim_reco1.root
Finished2025-12-17 22:27:01
Saved logsjustin-logs:263135.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ged trace sets:[ "gauss0":25598 [0] "wiener0":29622 [29622] ] cmm:[ bad:0 lf_noisy:243 ] 
[14:25:56.164] D [  glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=189 time=0 tick=512 with 55220 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":25598 [0] "wiener0":29622 [29622] ] cmm:[ bad:0 lf_noisy:243 ] input 1: frame: ident=189 time=0 tick=512 with 30320 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":13466 [0] "wiener1":16854 [16854] ] cmm:[ bad:0 ] input 2: frame: ident=189 time=0 tick=512 with 31010 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":13823 [0] "wiener2":17187 [17187] ] cmm:[ bad:0 ] input 3: frame: ident=189 time=0 tick=512 with 45585 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":19733 [0] "wiener3":25852 [25852] ] cmm:[ bad:0 ] output: frame: ident=189 time=0 tick=512 with 162135 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":25598 [0] "gauss1":13466 [0] "gauss2":13823 [0] "gauss3":19733 [0] "wiener0":29622 [29622] "wiener1":16854 [16854] "wiener2":17187 [17187] "wiener3":25852 [25852] ] cmm:[ bad:0 lf_noisy:243 ] 
Retagger: tagging trace set: wiener with 89515 traces, 89515 summary
Retagger: tagging trace set: gauss with 72620 traces, 0 summary
[14:25:56.170] D [  glue  ] sink frame: #189 @0 with 162135 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[14:25:56.170] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[14:25:56.170] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[14:25:56.170] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[14:25:56.170] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[14:25:56.170] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[14:25:56.170] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[14:25:56.170] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[14:25:56.170] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[14:25:56.170] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[14:25:56.170] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[14:25:56.170] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[14:25:56.170] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[14:25:56.170] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[14:25:56.170] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[14:25:56.170] D [  glue  ] frame sink sees EOS
[14:25:56.170] D [ pgraph ] <Pgrapher:> graph execution complete 
[14:25:56.170] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.69 sec
[14:25:56.170] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 4.56 sec
[14:25:56.170] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 4.34 sec
[14:25:56.170] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 4.06 sec
[14:25:56.170] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.71 sec
[14:25:56.170] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.7 sec
[14:25:56.170] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.68 sec
[14:25:56.170] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.68 sec
[14:25:56.170] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[14:25:56.170] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:25:56.170] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:25:56.170] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:25:56.170] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[14:25:56.170] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[14:25:56.170] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[14:25:56.170] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[14:25:56.170] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[14:25:56.170] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[14:25:56.170] I [ timer  ] Timer: Total node execution : 26.430000076070428 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 72620 traces tagged "gauss"
FrameSaver: q=2.13567e+06 n=1109591 tag=gauss
wclsFrameSaver: saving 89515 traces tagged "wiener"
FrameSaver: q=2.85065e+06 n=1066883 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 43947 XUs and 20769 XVs -> 783 XUVs
C:0 T:4 7 XUs and 11 XVs -> 0 XUVs
C:0 T:5 41327 XUs and 19697 XVs -> 548 XUVs
C:0 T:6 59206 XUs and 34562 XVs -> 1256 XUVs
2587 XUVs total
1302 collection wire objects
2587 potential space points
Neighbour search...
61867 tests to find 36160 neighbours
Iterating with no regularization...
Begin: 1.83642e+07
0 1.76383e+07
1 1.75936e+07
2 1.75886e+07
Now with regularization...
Begin: 1.66225e+07
0 1.6621e+07
17-Dec-2025 14:26:07 PST  Closed output file "pdhd_prod_beam__267379_8_1_20251212T160751Z_gen_g4_IonScintPDExt.root_262726_75_1_20251215T212431Z_PDInt.root_263135_0_1_20251217T214142Z_detsim_reco1.root"
17-Dec-2025 14:26:07 PST  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           33.8927       37.4209       40.3571       37.8789       2.43912        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.000605675   0.00433408    0.00765917    0.00459841    0.00223915       10     
reco:wclsdatahd:WireCellToolkit                      25.5644       27.1345       28.9743       27.1218      0.856312        10     
reco:wirefilter:WireFilter                         0.00811938     0.012059      0.0196508     0.0108079     0.0035856       10     
reco:gaushit:GausHitFinder                          0.169369      0.221843      0.266295      0.217539      0.0276461       10     
reco:nhitsfilter:NumberOfHitsFilter                 9.21e-05     0.000173903   0.000371457   0.000126229   9.44812e-05      10     
reco:reco3d:SpacePointSolver                        0.0692188     0.239278      0.761136      0.192258      0.196022        10     
reco:hitpdune:DisambigFromSpacePoints               0.0507694     0.110898      0.172461       0.11067      0.0391395       10     
reco:pandora:StandardPandora                         1.69669       2.92724       4.03523       3.09278      0.772795        10     
reco:pandoraWriter:StandardPandora                  0.0601782     0.105889      0.137669      0.102371      0.0228184       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.209499      0.365013      0.536107      0.334893      0.101346        10     
reco:pandoraShower:LArPandoraShowerCreation         0.0968462     0.296882      0.468986      0.299176      0.124708        10     
reco:pandoracalo:Calorimetry                        0.100338      0.175887      0.244722      0.187852      0.0469954       10     
reco:pandoracalonosce:Calorimetry                   0.102566      0.174081      0.255164      0.176995      0.0464565       10     
reco:pandoraShowercalo:ShowerCalorimetry            0.0728007     0.277775      0.445033      0.288984      0.118632        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.06941      0.269725      0.443857      0.282052      0.114222        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.013647      0.0196775     0.0277178     0.0198702    0.00397863       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0141205     0.019655      0.0249244     0.0198061     0.0033632       10     
reco:emtrkmichelid:EmTrackMichelId                   3.14962       4.48018       5.32859       4.83335      0.794398        10     
[art]:TriggerResults:TriggerResultInserter          1.459e-05    2.1651e-05     5.922e-05     1.647e-05    1.27966e-05      10     
end_path:out1:RootOutput                            5.19e-06     8.5969e-06    1.8629e-05     6.98e-06     3.87751e-06      10     
end_path:out1:RootOutput(write)                     0.422946      0.526898      0.641765      0.552096      0.0693512       10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6109.79 MB
  Peak resident set size usage (VmHWM): 3711.26 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 10292592
-rw-r--r-- 1 nilic nilic     232847 Dec 17 14:26 jobscript.log
-rw-r--r-- 1 nilic nilic        193 Dec 17 14:26 justin-processed-pfns.txt
-rw-r--r-- 1 nilic nilic        519 Dec 17 14:26 reco_protodunehd_hist.root
-rw-r--r-- 1 nilic nilic       3574 Dec 17 14:26 Pandora_Geometry.xml
-rw-r--r-- 1 nilic nilic     139264 Dec 17 14:26 mem.db
-rw-r--r-- 1 nilic nilic      28672 Dec 17 14:26 time.db
-rw-r--r-- 1 nilic nilic 5335832461 Dec 17 14:26 pdhd_prod_beam__267379_8_1_20251212T160751Z_gen_g4_IonScintPDExt.root_262726_75_1_20251215T212431Z_PDInt.root_263135_0_1_20251217T214142Z_detsim_reco1.root
-rw-r--r-- 1 nilic nilic   28112610 Dec 17 14:25 Pandora_Events.pndr
-rw-r--r-- 1 nilic nilic          0 Dec 17 14:19 debugprod.log
-rw-r--r-- 1 nilic nilic        519 Dec 17 14:19 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 nilic nilic 5175210853 Dec 17 14:19 temp_detsim.root
-rw-r--r-- 1 nilic nilic        120 Dec 17 13:41 all-input-dids.txt
justIN time: 2025-12-19 04:00:56 UTC       justIN version: 01.05.03