justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 263305.28@dunegpschedd02.fnal.gov

Jobsub ID263305.28@dunegpschedd02.fnal.gov
Workflow ID11264
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 19:31:07
SiteCA_SFU
EntryDUNE_CA_SFU_lcg-ce3
Last heartbeat2025-12-18 21:41:02
From worker nodeHostnamefc30559.int.fir.alliancecan.ca
cpuinfoAMD EPYC 9655 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit84598 (23 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 20:56:05
Input filesusertests:pdhd_prod_beam__267378_39_1_20251212T160746Z_gen_g4_IonScintPDExt.root_262727_44_1_20251215T214435Z_PDInt.root
JobscriptExit code0
Real time44m (2641s)
CPU time40m (2417s = 91%)
Max RSS bytes7038599168 (6712 MiB)
Outputting started2025-12-18 21:40:07
Output filesusertests:pdhd_prod_beam__267378_39_1_20251212T160746Z_gen_g4_IonScintPDExt.root_262727_44_1_20251215T214435Z_PDInt.root_263305_28_1_20251218T205609Z_detsim_reco1.root
Finished2025-12-18 21:41:02
Saved logsjustin-logs:263305.28-dunegpschedd02.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

51.749] D [  glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=179 time=0 tick=512 with 49240 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":23384 [0] "wiener0":25856 [25856] ] cmm:[ bad:0 lf_noisy:64 ] input 1: frame: ident=179 time=0 tick=512 with 35063 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":14995 [0] "wiener1":20068 [20068] ] cmm:[ bad:0 ] input 2: frame: ident=179 time=0 tick=512 with 51050 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":21610 [0] "wiener2":29440 [29440] ] cmm:[ bad:0 ] input 3: frame: ident=179 time=0 tick=512 with 41639 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":17649 [0] "wiener3":23990 [23990] ] cmm:[ bad:0 ] output: frame: ident=179 time=0 tick=512 with 176992 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":23384 [0] "gauss1":14995 [0] "gauss2":21610 [0] "gauss3":17649 [0] "wiener0":25856 [25856] "wiener1":20068 [20068] "wiener2":29440 [29440] "wiener3":23990 [23990] ] cmm:[ bad:0 lf_noisy:64 ] 
Retagger: tagging trace set: wiener with 99354 traces, 99354 summary
Retagger: tagging trace set: gauss with 77638 traces, 0 summary
[13:39:51.755] D [  glue  ] sink frame: #179 @0 with 176992 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[13:39:51.755] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[13:39:51.755] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[13:39:51.755] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[13:39:51.755] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[13:39:51.755] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[13:39:51.755] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[13:39:51.755] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[13:39:51.755] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[13:39:51.755] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[13:39:51.755] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[13:39:51.755] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[13:39:51.755] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[13:39:51.755] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[13:39:51.755] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[13:39:51.756] D [  glue  ] frame sink sees EOS
[13:39:51.756] D [ pgraph ] <Pgrapher:> graph execution complete 
[13:39:51.756] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.05 sec
[13:39:51.756] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 4.85 sec
[13:39:51.756] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 4.68 sec
[13:39:51.756] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 4.63 sec
[13:39:51.756] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.99 sec
[13:39:51.756] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.97 sec
[13:39:51.756] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.96 sec
[13:39:51.756] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.96 sec
[13:39:51.756] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[13:39:51.756] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[13:39:51.756] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:39:51.756] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0 sec
[13:39:51.756] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[13:39:51.756] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:39:51.756] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:39:51.756] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[13:39:51.756] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[13:39:51.756] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[13:39:51.756] I [ timer  ] Timer: Total node execution : 29.110000152140856 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 77638 traces tagged "gauss"
FrameSaver: q=2.56526e+06 n=1182781 tag=gauss
wclsFrameSaver: saving 99354 traces tagged "wiener"
FrameSaver: q=3.49324e+06 n=1162843 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 15897 XUs and 30296 XVs -> 576 XUVs
C:0 T:3 24 XUs and 0 XVs -> 0 XUVs
C:0 T:4 20 XUs and 38 XVs -> 14 XUVs
C:0 T:5 108295 XUs and 33888 XVs -> 2882 XUVs
C:0 T:6 42173 XUs and 51476 XVs -> 2130 XUVs
C:0 T:7 7 XUs and 92 XVs -> 1 XUVs
5603 XUVs total
2075 collection wire objects
5603 potential space points
Neighbour search...
392091 tests to find 192348 neighbours
Iterating with no regularization...
Begin: 1.02869e+08
0 9.19862e+07
1 9.12712e+07
2 9.11543e+07
3 9.11317e+07
Now with regularization...
Begin: 8.70732e+07
0 8.70526e+07
18-Dec-2025 13:40:06 PST  Closed output file "pdhd_prod_beam__267378_39_1_20251212T160746Z_gen_g4_IonScintPDExt.root_262727_44_1_20251215T214435Z_PDInt.root_263305_28_1_20251218T205609Z_detsim_reco1.root"
18-Dec-2025 13:40:06 PST  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           39.7126       45.1538       52.4232       44.9889       3.73261        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.000442927   0.00400327    0.00759947    0.00410591     0.0021243       10     
reco:wclsdatahd:WireCellToolkit                      29.2648       29.7113       30.3223       29.7253      0.319799        10     
reco:wirefilter:WireFilter                          0.010822      0.011725      0.0131851     0.0113955    0.000888471      10     
reco:gaushit:GausHitFinder                           0.2615        0.33243       0.43135      0.329053      0.0603163       10     
reco:nhitsfilter:NumberOfHitsFilter                 8.266e-05    0.000132467   0.000234779   0.000123424   3.99359e-05      10     
reco:reco3d:SpacePointSolver                        0.156085      0.363358      0.772261      0.378128      0.172566        10     
reco:hitpdune:DisambigFromSpacePoints                0.13488      0.230583      0.359469       0.21194      0.0704394       10     
reco:pandora:StandardPandora                         2.72985       5.14897       7.74086       4.9618        1.4485         10     
reco:pandoraWriter:StandardPandora                  0.0886563     0.117459      0.162113      0.114851      0.0214263       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.204398      0.499804      0.878816       0.43365      0.212023        10     
reco:pandoraShower:LArPandoraShowerCreation         0.284746      0.530596      0.932977      0.474227      0.189942        10     
reco:pandoracalo:Calorimetry                        0.135836      0.260722      0.405549      0.233872      0.0882396       10     
reco:pandoracalonosce:Calorimetry                   0.126734      0.255246      0.393733      0.226401      0.0853659       10     
reco:pandoraShowercalo:ShowerCalorimetry            0.282019      0.506877      0.813211      0.478026      0.153391        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.279347       0.50298      0.808124      0.471848      0.154552        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0188741     0.0257616     0.0333924     0.0258744    0.00464534       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0182901     0.0253927     0.0330049     0.0254223    0.00463344       10     
reco:emtrkmichelid:EmTrackMichelId                   4.44114       5.84979       7.78372       5.85996      0.950159        10     
[art]:TriggerResults:TriggerResultInserter         1.3939e-05    1.91729e-05    4.97e-05     1.5305e-05    1.02675e-05      10     
end_path:out1:RootOutput                            3.75e-06     5.6629e-06     1.651e-05    4.2495e-06    3.7001e-06       10     
end_path:out1:RootOutput(write)                     0.604235       0.74512      0.959376      0.741013      0.105218        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6497.97 MB
  Peak resident set size usage (VmHWM): 4118.79 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11480492
-rw-r--r-- 1 nilic nilic     232748 Dec 18 13:40 jobscript.log
-rw-r--r-- 1 nilic nilic        200 Dec 18 13:40 justin-processed-pfns.txt
-rw-r--r-- 1 nilic nilic        519 Dec 18 13:40 reco_protodunehd_hist.root
-rw-r--r-- 1 nilic nilic       3574 Dec 18 13:40 Pandora_Geometry.xml
-rw-r--r-- 1 nilic nilic     139264 Dec 18 13:40 mem.db
-rw-r--r-- 1 nilic nilic      28672 Dec 18 13:40 time.db
-rw-r--r-- 1 nilic nilic 5956636177 Dec 18 13:40 pdhd_prod_beam__267378_39_1_20251212T160746Z_gen_g4_IonScintPDExt.root_262727_44_1_20251215T214435Z_PDInt.root_263305_28_1_20251218T205609Z_detsim_reco1.root
-rw-r--r-- 1 nilic nilic   36729350 Dec 18 13:39 Pandora_Events.pndr
-rw-r--r-- 1 nilic nilic          0 Dec 18 13:31 debugprod.log
-rw-r--r-- 1 nilic nilic        519 Dec 18 13:31 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 nilic nilic 5762201460 Dec 18 13:31 temp_detsim.root
-rw-r--r-- 1 nilic nilic        121 Dec 18 12:56 all-input-dids.txt
justIN time: 2025-12-18 23:06:00 UTC       justIN version: 01.05.03