justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 262739.0@dunegpschedd02.fnal.gov

Jobsub ID262739.0@dunegpschedd02.fnal.gov
Workflow ID11226
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-15 22:33:37
SiteUS_BNL
EntryDUNE_US_BNL_spoolce01
Last heartbeat2025-12-15 23:52:49
From worker nodeHostnamespool0128.sdcc.bnl.gov
cpuinfoIntel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-15 22:34:56
Input filesusertests:pdhd_prod_beam__259173_48_1_20251208T181241Z_gen_g4_IonScintPDExt.root_260261_105_1_20251210T190845Z_PDInt.root
JobscriptExit code0
Real time1h (4594s)
CPU time1h (4527s = 98%)
Max RSS bytes7353253888 (7012 MiB)
Outputting started2025-12-15 23:51:31
Output filesusertests:pdhd_prod_beam__259173_48_1_20251208T181241Z_gen_g4_IonScintPDExt.root_260261_105_1_20251210T190845Z_PDInt.root_262739_0_1_20251215T223500Z_detsim_reco1.root
Finished2025-12-15 23:52:49
Saved logsjustin-logs:262739.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

uss0":31938 [0] "wiener0":36256 [36256] ] cmm:[ bad:0 lf_noisy:184 ] input 1: frame: ident=149 time=0 tick=512 with 45959 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":20123 [0] "wiener1":25836 [25836] ] cmm:[ bad:0 lf_noisy:1 ] input 2: frame: ident=149 time=0 tick=512 with 50035 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":21195 [0] "wiener2":28840 [28840] ] cmm:[ bad:0 lf_noisy:1 ] input 3: frame: ident=149 time=0 tick=512 with 37712 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":16891 [0] "wiener3":20821 [20821] ] cmm:[ bad:0 ] output: frame: ident=149 time=0 tick=512 with 201900 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":31938 [0] "gauss1":20123 [0] "gauss2":21195 [0] "gauss3":16891 [0] "wiener0":36256 [36256] "wiener1":25836 [25836] "wiener2":28840 [28840] "wiener3":20821 [20821] ] cmm:[ bad:0 lf_noisy:186 ] 
Retagger: tagging trace set: wiener with 111753 traces, 111753 summary
Retagger: tagging trace set: gauss with 90147 traces, 0 summary
[18:50:54.011] D [  glue  ] sink frame: #149 @0 with 201900 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[18:50:54.011] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[18:50:54.011] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[18:50:54.011] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[18:50:54.011] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[18:50:54.011] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[18:50:54.011] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[18:50:54.011] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[18:50:54.011] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[18:50:54.011] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[18:50:54.011] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[18:50:54.011] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[18:50:54.011] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[18:50:54.011] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[18:50:54.011] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[18:50:54.012] D [  glue  ] frame sink sees EOS
[18:50:54.012] D [ pgraph ] <Pgrapher:> graph execution complete 
[18:50:54.012] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 14.35 sec
[18:50:54.012] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.97 sec
[18:50:54.012] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.85 sec
[18:50:54.012] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.84 sec
[18:50:54.012] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.77 sec
[18:50:54.012] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.68 sec
[18:50:54.012] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.62 sec
[18:50:54.012] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.59 sec
[18:50:54.012] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.03 sec
[18:50:54.012] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[18:50:54.012] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[18:50:54.012] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:50:54.012] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:50:54.012] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:50:54.012] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[18:50:54.012] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[18:50:54.012] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[18:50:54.012] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[18:50:54.012] I [ timer  ] Timer: Total node execution : 54.720001028850675 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 90147 traces tagged "gauss"
FrameSaver: q=3.00361e+06 n=1505091 tag=gauss
wclsFrameSaver: saving 111753 traces tagged "wiener"
FrameSaver: q=3.79895e+06 n=1457498 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 34828 XUs and 33058 XVs -> 894 XUVs
C:0 T:3 25 XUs and 17 XVs -> 0 XUVs
C:0 T:4 42 XUs and 33 XVs -> 0 XUVs
C:0 T:5 42400 XUs and 42455 XVs -> 1098 XUVs
C:0 T:6 16929 XUs and 21379 XVs -> 804 XUVs
C:0 T:7 9 XUs and 14 XVs -> 3 XUVs
2799 XUVs total
1880 collection wire objects
2799 potential space points
Neighbour search...
26229 tests to find 14148 neighbours
Iterating with no regularization...
Begin: 2.68349e+07
0 2.56766e+07
1 2.56158e+07
2 2.56119e+07
Now with regularization...
Begin: 2.38835e+07
0 2.38823e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
15-Dec-2025 18:51:30 EST  Closed output file "pdhd_prod_beam__259173_48_1_20251208T181241Z_gen_g4_IonScintPDExt.root_260261_105_1_20251210T190845Z_PDInt.root_262739_0_1_20251215T223500Z_detsim_reco1.root"
15-Dec-2025 18:51:30 EST  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           73.8041       84.9407       106.705       81.2622       9.14495        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.00056819    0.00500633    0.00847083    0.00536325    0.00254824       10     
reco:wclsdatahd:WireCellToolkit                      54.1011       56.2612       58.973        56.1103       1.65021        10     
reco:wirefilter:WireFilter                          0.0142781     0.0194427     0.0256665     0.0187695    0.00313462       10     
reco:gaushit:GausHitFinder                          0.490678      0.660985       1.02543       0.61084      0.143749        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000187861   0.000279334   0.000489199   0.000250912   8.45271e-05      10     
reco:reco3d:SpacePointSolver                        0.252146      0.548443       1.11094      0.493711       0.26923        10     
reco:hitpdune:DisambigFromSpacePoints               0.307675      0.490567      0.940463       0.44412      0.178694        10     
reco:pandora:StandardPandora                         5.71565       9.44339       19.3404       7.85058       4.21955        10     
reco:pandoraWriter:StandardPandora                  0.148931      0.182253      0.261403      0.168908      0.0344237       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.638768       1.03605       1.48758      0.947042      0.303671        10     
reco:pandoraShower:LArPandoraShowerCreation         0.511255       1.06809       2.07961      0.899001       0.46353        10     
reco:pandoracalo:Calorimetry                        0.400903      0.556347      0.834839      0.507314      0.155473        10     
reco:pandoracalonosce:Calorimetry                   0.390928      0.536759      0.789356       0.48218      0.147468        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.482712       1.05889       2.21754      0.883736      0.492599        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.479101       1.04868       2.16839      0.839055      0.492534        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0369557     0.0476225     0.0686361     0.0419959     0.0105283       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0372565     0.0475484     0.0651624     0.0431004    0.00955354       10     
reco:emtrkmichelid:EmTrackMichelId                   7.86182       10.727        16.6976       10.112        2.37828        10     
[art]:TriggerResults:TriggerResultInserter         1.9314e-05    3.30478e-05   9.1638e-05    2.4785e-05    2.04303e-05      10     
end_path:out1:RootOutput                            5.493e-06    8.3604e-06    2.4893e-05     6.212e-06    5.65119e-06      10     
end_path:out1:RootOutput(write)                     0.892316       1.16269       1.59078       1.11461      0.199113        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6608.21 MB
  Peak resident set size usage (VmHWM): 4187.56 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12991336
-rw-r--r-- 1 fermipro fermilab     231400 Dec 15 18:51 jobscript.log
-rw-r--r-- 1 fermipro fermilab        195 Dec 15 18:51 justin-processed-pfns.txt
-rw-r--r-- 1 fermipro fermilab        519 Dec 15 18:51 reco_protodunehd_hist.root
-rw-r--r-- 1 fermipro fermilab       3574 Dec 15 18:51 Pandora_Geometry.xml
-rw-r--r-- 1 fermipro fermilab     139264 Dec 15 18:51 mem.db
-rw-r--r-- 1 fermipro fermilab      28672 Dec 15 18:51 time.db
-rw-r--r-- 1 fermipro fermilab 6732975001 Dec 15 18:51 pdhd_prod_beam__259173_48_1_20251208T181241Z_gen_g4_IonScintPDExt.root_260261_105_1_20251210T190845Z_PDInt.root_262739_0_1_20251215T223500Z_detsim_reco1.root
-rw-r--r-- 1 fermipro fermilab   36962220 Dec 15 18:51 Pandora_Events.pndr
-rw-r--r-- 1 fermipro fermilab          0 Dec 15 18:35 debugprod.log
-rw-r--r-- 1 fermipro fermilab        519 Dec 15 18:35 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 fermipro fermilab 6532744726 Dec 15 18:35 temp_detsim.root
-rw-r--r-- 1 fermipro fermilab        122 Dec 15 17:35 all-input-dids.txt
justIN time: 2025-12-19 04:02:48 UTC       justIN version: 01.05.03