justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 263377.1@dunegpschedd02.fnal.gov

Jobsub ID263377.1@dunegpschedd02.fnal.gov
Workflow ID11264
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-19 05:20:34
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-19 06:37:27
From worker nodeHostnamedunegli-7955013-0-fnpc22002.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-19 05:21:47
Input filesusertests:pdhd_prod_beam__261612_109_1_20251212T160602Z_gen_g4_IonScintPDExt.root_262727_25_1_20251215T214329Z_PDInt.root
JobscriptExit code0
Real time1h (4434s)
CPU time1h (4285s = 96%)
Max RSS bytes12084281344 (11524 MiB)
Outputting started2025-12-19 06:36:33
Output filesusertests:pdhd_prod_beam__261612_109_1_20251212T160602Z_gen_g4_IonScintPDExt.root_262727_25_1_20251215T214329Z_PDInt.root_263377_1_1_20251219T052151Z_detsim_reco1.root
Finished2025-12-19 06:37:27
Saved logsjustin-logs:263377.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

[27500] ] cmm:[ bad:0 lf_noisy:27 ] 
[06:35:22.852] D [  glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=179 time=0 tick=512 with 51364 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":23864 [0] "wiener0":27500 [27500] ] cmm:[ bad:0 lf_noisy:27 ] input 1: frame: ident=179 time=0 tick=512 with 34849 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":15181 [0] "wiener1":19668 [19668] ] cmm:[ bad:0 ] input 2: frame: ident=179 time=0 tick=512 with 44321 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":18945 [0] "wiener2":25376 [25376] ] cmm:[ bad:0 ] input 3: frame: ident=179 time=0 tick=512 with 35247 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":14487 [0] "wiener3":20760 [20760] ] cmm:[ bad:0 ] output: frame: ident=179 time=0 tick=512 with 165781 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":23864 [0] "gauss1":15181 [0] "gauss2":18945 [0] "gauss3":14487 [0] "wiener0":27500 [27500] "wiener1":19668 [19668] "wiener2":25376 [25376] "wiener3":20760 [20760] ] cmm:[ bad:0 lf_noisy:27 ] 
Retagger: tagging trace set: wiener with 93304 traces, 93304 summary
Retagger: tagging trace set: gauss with 72477 traces, 0 summary
[06:35:22.857] D [  glue  ] sink frame: #179 @0 with 165781 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[06:35:22.857] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[06:35:22.857] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[06:35:22.857] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[06:35:22.857] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[06:35:22.857] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[06:35:22.857] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[06:35:22.857] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[06:35:22.857] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[06:35:22.857] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[06:35:22.857] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[06:35:22.857] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[06:35:22.857] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[06:35:22.857] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[06:35:22.857] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[06:35:22.857] D [  glue  ] frame sink sees EOS
[06:35:22.857] D [ pgraph ] <Pgrapher:> graph execution complete 
[06:35:22.857] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.74 sec
[06:35:22.857] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.3 sec
[06:35:22.857] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.15 sec
[06:35:22.857] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.15 sec
[06:35:22.857] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.92 sec
[06:35:22.857] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.9 sec
[06:35:22.857] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.9 sec
[06:35:22.857] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.9 sec
[06:35:22.857] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[06:35:22.857] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[06:35:22.857] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[06:35:22.857] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[06:35:22.857] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[06:35:22.857] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[06:35:22.857] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[06:35:22.857] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0 sec
[06:35:22.857] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[06:35:22.857] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[06:35:22.857] I [ timer  ] Timer: Total node execution : 30.970000037923455 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 72477 traces tagged "gauss"
FrameSaver: q=2.11548e+06 n=1082406 tag=gauss
wclsFrameSaver: saving 93304 traces tagged "wiener"
FrameSaver: q=2.69973e+06 n=1034779 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 13156 XUs and 30395 XVs -> 172 XUVs
C:0 T:5 62550 XUs and 50147 XVs -> 1305 XUVs
C:0 T:6 13643 XUs and 83224 XVs -> 656 XUVs
C:0 T:7 3 XUs and 6 XVs -> 0 XUVs
2133 XUVs total
1087 collection wire objects
2133 potential space points
Neighbour search...
51421 tests to find 30252 neighbours
Iterating with no regularization...
Begin: 2.61638e+07
0 2.53276e+07
1 2.5268e+07
2 2.52597e+07
Now with regularization...
Begin: 2.4003e+07
0 2.4e+07
19-Dec-2025 06:35:41 UTC  Closed output file "pdhd_prod_beam__261612_109_1_20251212T160602Z_gen_g4_IonScintPDExt.root_262727_25_1_20251215T214329Z_PDInt.root_263377_1_1_20251219T052151Z_detsim_reco1.root"
19-Dec-2025 06:35:41 UTC  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           33.8033       52.9437       62.3724       53.5127       7.72366        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.000744704   0.00216434    0.00323217    0.00230107    0.00087163       10     
reco:wclsdatahd:WireCellToolkit                      31.3368       31.8143       32.4972       31.7052      0.345698        10     
reco:wirefilter:WireFilter                         0.00815189    0.00994884     0.0118263    0.00991805    0.00117511       10     
reco:gaushit:GausHitFinder                          0.277568       0.39246      0.661014      0.364391      0.110992        10     
reco:nhitsfilter:NumberOfHitsFilter                0.00010032    0.000154598   0.000268299   0.000139861   5.2976e-05       10     
reco:reco3d:SpacePointSolver                        0.151421      0.316583      0.698738      0.292882      0.151583         9     
reco:hitpdune:DisambigFromSpacePoints               0.128879       0.21931      0.392043      0.210126      0.0784483        9     
reco:pandora:StandardPandora                         2.89205       4.9689        7.7044        4.58163       1.71446         9     
reco:pandoraWriter:StandardPandora                  0.0814314     0.0990801     0.118622      0.0949518     0.0135568        9     
reco:pandoraTrack:LArPandoraTrackCreation           0.253933      0.460843      0.665683      0.450511      0.152544         9     
reco:pandoraShower:LArPandoraShowerCreation         0.311938      0.462924      0.735316      0.407129      0.151592         9     
reco:pandoracalo:Calorimetry                         0.17854      0.286091      0.391783      0.274421      0.0764376        9     
reco:pandoracalonosce:Calorimetry                   0.176036      0.277631      0.383114      0.267531      0.0733829        9     
reco:pandoraShowercalo:ShowerCalorimetry            0.339613      0.494346      0.776674       0.44008      0.150169         9     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.335279      0.485314      0.772009      0.435292      0.151216         9     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0224867     0.0277584     0.0324126     0.0272741    0.00361372        9     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0215866     0.0265756     0.0325195     0.0270689    0.00342881        9     
reco:emtrkmichelid:EmTrackMichelId                   11.7496       14.1052       17.006        13.6139       1.89875         9     
[art]:TriggerResults:TriggerResultInserter         1.2243e-05    2.28102e-05   8.7666e-05    1.60155e-05   2.17351e-05      10     
end_path:out1:RootOutput                            2.585e-06    9.1504e-06    2.8554e-05    7.4595e-06    6.88337e-06      10     
end_path:out1:RootOutput(write)                      0.61648      0.696507      0.838072      0.672943      0.0748442       10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6161.24 MB
  Peak resident set size usage (VmHWM): 3989.1 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 13946668
-rw-r--r-- 1 dunegli fnalgrid     230766 Dec 19 06:35 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        201 Dec 19 06:35 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 19 06:35 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Dec 19 06:35 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     135168 Dec 19 06:35 mem.db
-rw-r--r-- 1 dunegli fnalgrid      28672 Dec 19 06:35 time.db
-rw-r--r-- 1 dunegli fnalgrid 7219872037 Dec 19 06:35 pdhd_prod_beam__261612_109_1_20251212T160602Z_gen_g4_IonScintPDExt.root_262727_25_1_20251215T214329Z_PDInt.root_263377_1_1_20251219T052151Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   32774155 Dec 19 06:35 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Dec 19 06:25 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 19 06:25 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 7028219508 Dec 19 06:25 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        122 Dec 19 05:21 all-input-dids.txt
justIN time: 2025-12-19 11:18:09 UTC       justIN version: 01.05.03