justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 269119.37@dunegpschedd01.fnal.gov

Jobsub ID269119.37@dunegpschedd01.fnal.gov
Workflow ID11264
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 19:29:50
SiteCA_SFU
EntryDUNE_CA_SFU_lcg-ce3
Last heartbeat2025-12-18 20:32:08
From worker nodeHostnamefc30554.int.fir.alliancecan.ca
cpuinfoAMD EPYC 9655 96-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit84598 (23 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 19:49:57
Input filesusertests:pdhd_prod_beam__267378_58_1_20251212T160754Z_gen_g4_IonScintPDExt.root_268535_31_1_20251215T214751Z_PDInt.root
JobscriptExit code0
Real time40m (2415s)
CPU time31m (1893s = 78%)
Max RSS bytes7148335104 (6817 MiB)
Outputting started2025-12-18 20:30:13
Output filesusertests:pdhd_prod_beam__267378_58_1_20251212T160754Z_gen_g4_IonScintPDExt.root_268535_31_1_20251215T214751Z_PDInt.root_269119_37_1_20251218T195000Z_detsim_reco1.root
Finished2025-12-18 20:32:08
Saved logsjustin-logs:269119.37-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

9.869] D [  glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=179 time=0 tick=512 with 73915 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":35211 [0] "wiener0":38704 [38704] ] cmm:[ bad:0 lf_noisy:175 ] input 1: frame: ident=179 time=0 tick=512 with 44908 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":19262 [0] "wiener1":25646 [25646] ] cmm:[ bad:0 ] input 2: frame: ident=179 time=0 tick=512 with 37050 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":16215 [0] "wiener2":20835 [20835] ] cmm:[ bad:0 ] input 3: frame: ident=179 time=0 tick=512 with 51104 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":22304 [0] "wiener3":28800 [28800] ] cmm:[ bad:0 ] output: frame: ident=179 time=0 tick=512 with 206977 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":35211 [0] "gauss1":19262 [0] "gauss2":16215 [0] "gauss3":22304 [0] "wiener0":38704 [38704] "wiener1":25646 [25646] "wiener2":20835 [20835] "wiener3":28800 [28800] ] cmm:[ bad:0 lf_noisy:175 ] 
Retagger: tagging trace set: wiener with 113985 traces, 113985 summary
Retagger: tagging trace set: gauss with 92992 traces, 0 summary
[12:29:59.875] D [  glue  ] sink frame: #179 @0 with 206977 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[12:29:59.875] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[12:29:59.875] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[12:29:59.875] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[12:29:59.875] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[12:29:59.875] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[12:29:59.875] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[12:29:59.875] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[12:29:59.875] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[12:29:59.875] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[12:29:59.875] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[12:29:59.875] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[12:29:59.876] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[12:29:59.876] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[12:29:59.876] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[12:29:59.876] D [  glue  ] frame sink sees EOS
[12:29:59.876] D [ pgraph ] <Pgrapher:> graph execution complete 
[12:29:59.876] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.45 sec
[12:29:59.876] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 3.49 sec
[12:29:59.876] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 3.45 sec
[12:29:59.876] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 3.37 sec
[12:29:59.876] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.43 sec
[12:29:59.876] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.43 sec
[12:29:59.876] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.41 sec
[12:29:59.876] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.41 sec
[12:29:59.876] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[12:29:59.876] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0.01 sec
[12:29:59.876] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[12:29:59.876] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[12:29:59.876] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:29:59.876] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:29:59.876] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:29:59.876] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:29:59.876] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[12:29:59.876] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0 sec
[12:29:59.876] I [ timer  ] Timer: Total node execution : 21.459999579936266 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 92992 traces tagged "gauss"
FrameSaver: q=3.03083e+06 n=1468298 tag=gauss
wclsFrameSaver: saving 113985 traces tagged "wiener"
FrameSaver: q=4.31249e+06 n=1440194 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 35293 XUs and 27077 XVs -> 976 XUVs
C:0 T:3 22 XUs and 104 XVs -> 1 XUVs
C:0 T:4 5 XUs and 21 XVs -> 4 XUVs
C:0 T:5 23091 XUs and 17875 XVs -> 291 XUVs
C:0 T:6 40193 XUs and 48499 XVs -> 3175 XUVs
C:0 T:7 13 XUs and 8 XVs -> 0 XUVs
4447 XUVs total
1658 collection wire objects
4447 potential space points
Neighbour search...
325879 tests to find 189678 neighbours
Iterating with no regularization...
Begin: 1.92754e+07
0 1.77846e+07
1 1.76854e+07
2 1.76667e+07
3 1.766e+07
Now with regularization...
Begin: 1.65175e+07
0 1.65126e+07
18-Dec-2025 12:30:12 PST  Closed output file "pdhd_prod_beam__267378_58_1_20251212T160754Z_gen_g4_IonScintPDExt.root_268535_31_1_20251215T214751Z_PDInt.root_269119_37_1_20251218T195000Z_detsim_reco1.root"
18-Dec-2025 12:30:12 PST  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           26.7551       32.107        38.1301       31.6397       3.26542        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.000239978   0.00291359    0.00505589    0.00324217    0.00161373       10     
reco:wclsdatahd:WireCellToolkit                      20.8352       21.3337       22.0031       21.3747      0.348601        10     
reco:wirefilter:WireFilter                         0.00625707    0.00838869     0.0128587    0.00798744    0.00170495       10     
reco:gaushit:GausHitFinder                          0.157043      0.234529      0.334017      0.225894      0.0526124       10     
reco:nhitsfilter:NumberOfHitsFilter                 4.693e-05    9.60851e-05   0.000224598   7.7704e-05    4.75147e-05      10     
reco:reco3d:SpacePointSolver                        0.123324      0.186233      0.251572       0.17373      0.0464011       10     
reco:hitpdune:DisambigFromSpacePoints               0.0851847     0.143394      0.203066      0.135547      0.0380039       10     
reco:pandora:StandardPandora                         1.39896       3.39829       6.03674       3.40647       1.40346        10     
reco:pandoraWriter:StandardPandora                  0.0503894     0.0809142      0.11633      0.0755151     0.0168446       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.174111      0.376022      0.740033      0.342904      0.154008        10     
reco:pandoraShower:LArPandoraShowerCreation         0.141089      0.409261      0.708556      0.343356       0.17595        10     
reco:pandoracalo:Calorimetry                        0.0963897     0.198724      0.348178      0.195487      0.0644875       10     
reco:pandoracalonosce:Calorimetry                   0.0933401      0.18833      0.333875      0.176549      0.0620882       10     
reco:pandoraShowercalo:ShowerCalorimetry            0.114142      0.387505      0.633424       0.34337      0.158895        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.11294      0.384524       0.62844      0.347963      0.155584        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0120208     0.0176578     0.0235862     0.0169533    0.00284108       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0117057     0.0171081     0.0207193     0.0174721     0.002288        10     
reco:emtrkmichelid:EmTrackMichelId                   2.93551       4.20712       5.69173       3.96034      0.774564        10     
[art]:TriggerResults:TriggerResultInserter          1.111e-05    1.5693e-05     4.792e-05     1.204e-05    1.07683e-05      10     
end_path:out1:RootOutput                            3.049e-06    4.9138e-06     1.446e-05     3.415e-06    3.28658e-06      10     
end_path:out1:RootOutput(write)                     0.381081      0.513156      0.653858      0.492164      0.0779321       10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6380.81 MB
  Peak resident set size usage (VmHWM): 3993.68 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11294360
-rw-r--r-- 1 nilic nilic     231309 Dec 18 12:30 jobscript.log
-rw-r--r-- 1 nilic nilic        200 Dec 18 12:30 justin-processed-pfns.txt
-rw-r--r-- 1 nilic nilic        519 Dec 18 12:30 reco_protodunehd_hist.root
-rw-r--r-- 1 nilic nilic       3574 Dec 18 12:30 Pandora_Geometry.xml
-rw-r--r-- 1 nilic nilic     139264 Dec 18 12:30 mem.db
-rw-r--r-- 1 nilic nilic      28672 Dec 18 12:30 time.db
-rw-r--r-- 1 nilic nilic 5860073006 Dec 18 12:30 pdhd_prod_beam__267378_58_1_20251212T160754Z_gen_g4_IonScintPDExt.root_268535_31_1_20251215T214751Z_PDInt.root_269119_37_1_20251218T195000Z_detsim_reco1.root
-rw-r--r-- 1 nilic nilic   35844510 Dec 18 12:30 Pandora_Events.pndr
-rw-r--r-- 1 nilic nilic          0 Dec 18 12:24 debugprod.log
-rw-r--r-- 1 nilic nilic        519 Dec 18 12:24 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 nilic nilic 5669049599 Dec 18 12:24 temp_detsim.root
-rw-r--r-- 1 nilic nilic        121 Dec 18 11:50 all-input-dids.txt
justIN time: 2025-12-19 04:36:16 UTC       justIN version: 01.05.03