justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 267344.1@dunegpschedd01.fnal.gov

Jobsub ID267344.1@dunegpschedd01.fnal.gov
Workflow ID11129
Stage ID1
User nameavizcaya@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-12 13:21:27
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-12 15:04:09
From worker nodeHostnamedunegli-7890204-0-fnpc19119.fnal.gov
cpuinfoAMD EPYC 7502 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-12 13:22:59
Input filesusertests:pdhd_prod_beam__247395_1_1_20251112T002610Z_gen_g4_IonScintPDExt.root_258179_149_1_20251205T222957Z_PDInt.root
JobscriptExit code0
Real time1h (5948s)
CPU time1h (5483s = 92%)
Max RSS bytes6523133952 (6220 MiB)
Outputting started2025-12-12 15:02:07
Output filesusertests:pdhd_prod_beam__247395_1_1_20251112T002610Z_gen_g4_IonScintPDExt.root_258179_149_1_20251205T222957Z_PDInt.root_267344_1_1_20251212T132309Z_detsim_reco1.root
Finished2025-12-12 15:04:09
Saved logsjustin-logs:267344.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

bad:0 lf_noisy:5 ] 
[15:01:14.921] D [  glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=119 time=0 tick=512 with 55977 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":26867 [0] "wiener0":29110 [29110] ] cmm:[ bad:0 lf_noisy:5 ] input 1: frame: ident=119 time=0 tick=512 with 30551 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":13009 [0] "wiener1":17542 [17542] ] cmm:[ bad:0 ] input 2: frame: ident=119 time=0 tick=512 with 45461 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":19998 [0] "wiener2":25463 [25463] ] cmm:[ bad:0 ] input 3: frame: ident=119 time=0 tick=512 with 40786 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":18160 [0] "wiener3":22626 [22626] ] cmm:[ bad:0 ] output: frame: ident=119 time=0 tick=512 with 172775 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":26867 [0] "gauss1":13009 [0] "gauss2":19998 [0] "gauss3":18160 [0] "wiener0":29110 [29110] "wiener1":17542 [17542] "wiener2":25463 [25463] "wiener3":22626 [22626] ] cmm:[ bad:0 lf_noisy:5 ] 
Retagger: tagging trace set: wiener with 94741 traces, 94741 summary
Retagger: tagging trace set: gauss with 78034 traces, 0 summary
[15:01:14.954] D [  glue  ] sink frame: #119 @0 with 172775 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[15:01:14.954] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[15:01:14.954] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[15:01:14.954] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[15:01:14.954] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[15:01:14.954] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[15:01:14.954] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[15:01:14.954] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[15:01:14.955] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[15:01:14.955] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[15:01:14.955] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[15:01:14.955] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[15:01:14.955] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[15:01:14.955] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[15:01:14.955] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[15:01:14.955] D [  glue  ] frame sink sees EOS
[15:01:14.955] D [ pgraph ] <Pgrapher:> graph execution complete 
[15:01:14.955] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 18.96 sec
[15:01:14.955] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.88 sec
[15:01:14.955] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.77 sec
[15:01:14.955] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.51 sec
[15:01:14.955] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.77 sec
[15:01:14.955] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.72 sec
[15:01:14.955] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.68 sec
[15:01:14.955] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.62 sec
[15:01:14.955] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.06 sec
[15:01:14.955] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.02 sec
[15:01:14.955] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[15:01:14.955] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[15:01:14.955] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[15:01:14.955] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[15:01:14.955] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[15:01:14.955] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[15:01:14.955] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[15:01:14.955] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[15:01:14.955] I [ timer  ] Timer: Total node execution : 67.01999984495342 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 78034 traces tagged "gauss"
FrameSaver: q=2.34116e+06 n=1253524 tag=gauss
wclsFrameSaver: saving 94741 traces tagged "wiener"
FrameSaver: q=2.96868e+06 n=1217904 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 5391 XUs and 6934 XVs -> 594 XUVs
C:0 T:4 4 XUs and 4 XVs -> 0 XUVs
C:0 T:5 50588 XUs and 37486 XVs -> 780 XUVs
C:0 T:6 18732 XUs and 20853 XVs -> 619 XUVs
1993 XUVs total
1188 collection wire objects
1993 potential space points
Neighbour search...
25381 tests to find 14968 neighbours
Iterating with no regularization...
Begin: 1.3374e+07
0 1.29852e+07
1 1.29619e+07
2 1.29597e+07
Now with regularization...
Begin: 1.17659e+07
0 1.17653e+07
12-Dec-2025 15:02:05 UTC  Closed output file "pdhd_prod_beam__247395_1_1_20251112T002610Z_gen_g4_IonScintPDExt.root_258179_149_1_20251205T222957Z_PDInt.root_267344_1_1_20251212T132309Z_detsim_reco1.root"
12-Dec-2025 15:02:05 UTC  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           76.1191       109.62        132.823       112.055       16.4631        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.00176692    0.00545828    0.00966481    0.00534949    0.00242973       10     
reco:wclsdatahd:WireCellToolkit                      46.1424       65.9059       73.2511       69.8101       9.35282        10     
reco:wirefilter:WireFilter                          0.0214209     0.0617754     0.084047      0.0680114     0.0205371       10     
reco:gaushit:GausHitFinder                          0.342986       0.47661      0.567059      0.474286      0.0659779       10     
reco:nhitsfilter:NumberOfHitsFilter                0.000317943   0.00119038    0.00232165    0.00104901    0.000594054      10     
reco:reco3d:SpacePointSolver                        0.226941      0.511929       1.30951      0.513734      0.306172        10     
reco:hitpdune:DisambigFromSpacePoints               0.162503      0.267741      0.375875      0.253644      0.0670599       10     
reco:pandora:StandardPandora                         5.48442       8.91181       14.0911       8.80046       2.48948        10     
reco:pandoraWriter:StandardPandora                  0.167248      0.305265      0.409709      0.318541      0.0750376       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.483734       1.56232       2.82601       1.65168      0.765644        10     
reco:pandoraShower:LArPandoraShowerCreation         0.488192       1.64192       3.1152        1.47421       0.88509        10     
reco:pandoracalo:Calorimetry                        0.313017      0.857386       1.37507       0.95994       0.36149        10     
reco:pandoracalonosce:Calorimetry                   0.304426       0.78443       1.25339      0.855708      0.344479        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.502704       1.33666       2.61583       1.15829      0.730611        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.494481       1.28816       2.43341       1.14403      0.715588        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0389306     0.0523457     0.0725343     0.0502194     0.010829        10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0363615     0.0532211     0.0737648     0.0556485     0.0130417       10     
reco:emtrkmichelid:EmTrackMichelId                   17.531        24.4064       29.5211       26.6128       4.47956        10     
[art]:TriggerResults:TriggerResultInserter         2.4521e-05    8.74979e-05   0.000211032   6.82805e-05   6.10084e-05      10     
end_path:out1:RootOutput                            6.59e-06     1.47671e-05   3.7951e-05    1.2245e-05    8.42863e-06      10     
end_path:out1:RootOutput(write)                     0.735314       1.02929       1.22569       1.0393        0.15338        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5653.62 MB
  Peak resident set size usage (VmHWM): 3495.83 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 10872380
-rw-r--r-- 1 dunegli fnalgrid     230991 Dec 12 15:02 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        194 Dec 12 15:02 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 12 15:02 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Dec 12 15:02 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     139264 Dec 12 15:02 mem.db
-rw-r--r-- 1 dunegli fnalgrid      28672 Dec 12 15:02 time.db
-rw-r--r-- 1 dunegli fnalgrid 5637617022 Dec 12 15:02 pdhd_prod_beam__247395_1_1_20251112T002610Z_gen_g4_IonScintPDExt.root_258179_149_1_20251205T222957Z_PDInt.root_267344_1_1_20251212T132309Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   31681230 Dec 12 15:01 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Dec 12 14:41 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 12 14:41 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 5463489184 Dec 12 14:41 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        121 Dec 12 13:23 all-input-dids.txt
justIN time: 2026-02-07 14:58:50 UTC       justIN version: 01.06.00