justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 263305.38@dunegpschedd02.fnal.gov

Jobsub ID263305.38@dunegpschedd02.fnal.gov
Workflow ID11264
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 19:31:07
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-18 22:00:27
From worker nodeHostnamedunegli-7951944-0-fnpc23110.fnal.gov
cpuinfoAMD EPYC 9354 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 21:01:45
Input filesusertests:pdhd_prod_beam__267378_24_1_20251212T160746Z_gen_g4_IonScintPDExt.root_262727_9_1_20251215T214250Z_PDInt.root
JobscriptExit code0
Real time57m (3475s)
CPU time56m (3397s = 97%)
Max RSS bytes7141978112 (6811 MiB)
Outputting started2025-12-18 21:59:40
Output filesusertests:pdhd_prod_beam__267378_24_1_20251212T160746Z_gen_g4_IonScintPDExt.root_262727_9_1_20251215T214250Z_PDInt.root_263305_38_1_20251218T210148Z_detsim_reco1.root
Finished2025-12-18 22:00:27
Saved logsjustin-logs:263305.38-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

gproc" ] 2 tagged trace sets:[ "gauss0":20909 [0] "wiener0":22151 [22151] ] cmm:[ bad:0 lf_noisy:26 ] input 1: frame: ident=179 time=0 tick=512 with 45226 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":18574 [0] "wiener1":26652 [26652] ] cmm:[ bad:0 ] input 2: frame: ident=179 time=0 tick=512 with 53414 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":22672 [0] "wiener2":30742 [30742] ] cmm:[ bad:0 ] input 3: frame: ident=179 time=0 tick=512 with 40718 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":18200 [0] "wiener3":22518 [22518] ] cmm:[ bad:0 ] output: frame: ident=179 time=0 tick=512 with 182418 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":20909 [0] "gauss1":18574 [0] "gauss2":22672 [0] "gauss3":18200 [0] "wiener0":22151 [22151] "wiener1":26652 [26652] "wiener2":30742 [30742] "wiener3":22518 [22518] ] cmm:[ bad:0 lf_noisy:26 ] 
Retagger: tagging trace set: wiener with 102063 traces, 102063 summary
Retagger: tagging trace set: gauss with 80355 traces, 0 summary
[21:59:19.473] D [  glue  ] sink frame: #179 @0 with 182418 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[21:59:19.473] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[21:59:19.473] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[21:59:19.473] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[21:59:19.473] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[21:59:19.473] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[21:59:19.473] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[21:59:19.473] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[21:59:19.473] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[21:59:19.473] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[21:59:19.473] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[21:59:19.473] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[21:59:19.473] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[21:59:19.473] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[21:59:19.473] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[21:59:19.473] D [  glue  ] frame sink sees EOS
[21:59:19.473] D [ pgraph ] <Pgrapher:> graph execution complete 
[21:59:19.473] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 6.74 sec
[21:59:19.473] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 4.77 sec
[21:59:19.473] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 4.68 sec
[21:59:19.473] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 4.57 sec
[21:59:19.473] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.79 sec
[21:59:19.473] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.79 sec
[21:59:19.473] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.79 sec
[21:59:19.473] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.79 sec
[21:59:19.473] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[21:59:19.473] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:59:19.473] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:59:19.473] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:59:19.473] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:59:19.473] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[21:59:19.473] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[21:59:19.473] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[21:59:19.473] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[21:59:19.473] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[21:59:19.473] I [ timer  ] Timer: Total node execution : 27.92999959923327 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 80355 traces tagged "gauss"
FrameSaver: q=2.61844e+06 n=1271969 tag=gauss
wclsFrameSaver: saving 102063 traces tagged "wiener"
FrameSaver: q=3.41398e+06 n=1246672 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 70588 XUs and 45573 XVs -> 1825 XUVs
C:0 T:3 8 XUs and 6 XVs -> 0 XUVs
C:0 T:4 27 XUs and 20 XVs -> 0 XUVs
C:0 T:5 34992 XUs and 35350 XVs -> 1257 XUVs
C:0 T:6 26878 XUs and 22813 XVs -> 515 XUVs
C:0 T:7 3 XUs and 11 XVs -> 1 XUVs
3598 XUVs total
1802 collection wire objects
3598 potential space points
Neighbour search...
85110 tests to find 49166 neighbours
Iterating with no regularization...
Begin: 6.48874e+07
0 5.08843e+07
1 5.03205e+07
2 5.01917e+07
3 5.01493e+07
Now with regularization...
Begin: 4.79473e+07
0 4.7924e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
18-Dec-2025 21:59:40 UTC  Closed output file "pdhd_prod_beam__267378_24_1_20251212T160746Z_gen_g4_IonScintPDExt.root_262727_9_1_20251215T214250Z_PDInt.root_263305_38_1_20251218T210148Z_detsim_reco1.root"
18-Dec-2025 21:59:40 UTC  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           39.071        46.4171       55.9145       46.9868       4.55512        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.000353168   0.00329974    0.00537643    0.00353226    0.00168798       10     
reco:wclsdatahd:WireCellToolkit                      27.6561       28.3549       28.7542       28.3865      0.316714        10     
reco:wirefilter:WireFilter                         0.00729599    0.00929308     0.0116187    0.00970779    0.00121721       10     
reco:gaushit:GausHitFinder                          0.218619      0.323662      0.443772      0.337655      0.0631243       10     
reco:nhitsfilter:NumberOfHitsFilter                7.6877e-05    0.000111589   0.000255009   9.70925e-05   4.88777e-05      10     
reco:reco3d:SpacePointSolver                        0.159756       0.26734      0.477953      0.229512       0.10576        10     
reco:hitpdune:DisambigFromSpacePoints               0.0983004     0.229161      0.450757      0.214147      0.0953313       10     
reco:pandora:StandardPandora                         2.09605       4.45594       7.59582       4.49132       1.48493        10     
reco:pandoraWriter:StandardPandora                  0.0652496     0.0951814     0.151053      0.0942324     0.0235156       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.268862      0.452554       0.73808      0.419592      0.131788        10     
reco:pandoraShower:LArPandoraShowerCreation         0.217052      0.463303      0.780361      0.504598      0.161156        10     
reco:pandoracalo:Calorimetry                        0.164103      0.276339      0.434141      0.260263      0.0723745       10     
reco:pandoracalonosce:Calorimetry                    0.15922      0.268329      0.418653      0.253426      0.0691883       10     
reco:pandoraShowercalo:ShowerCalorimetry            0.197197      0.470017      0.769612      0.522071      0.159573        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.195067      0.463809      0.762429      0.514222      0.157704        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0169147     0.024888      0.0351326     0.0253578    0.00528936       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0163123     0.0237043     0.0319975     0.0242915    0.00473747       10     
reco:emtrkmichelid:EmTrackMichelId                   6.48831       9.51085       13.1169       9.74165       1.8218         10     
[art]:TriggerResults:TriggerResultInserter         1.2459e-05    2.03479e-05   6.6622e-05    1.57135e-05   1.55022e-05      10     
end_path:out1:RootOutput                            3.135e-06    5.5164e-06    1.9419e-05     3.385e-06    4.77455e-06      10     
end_path:out1:RootOutput(write)                     0.516716      0.703011      0.979624      0.710627       0.12639        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6257.85 MB
  Peak resident set size usage (VmHWM): 3894.77 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12126224
-rw-r--r-- 1 dunegli fnalgrid     231267 Dec 18 21:59 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        199 Dec 18 21:59 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 18 21:59 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Dec 18 21:59 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     139264 Dec 18 21:59 mem.db
-rw-r--r-- 1 dunegli fnalgrid      28672 Dec 18 21:59 time.db
-rw-r--r-- 1 dunegli fnalgrid 6290607916 Dec 18 21:59 pdhd_prod_beam__267378_24_1_20251212T160746Z_gen_g4_IonScintPDExt.root_262727_9_1_20251215T214250Z_PDInt.root_263305_38_1_20251218T210148Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   38782830 Dec 18 21:59 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Dec 18 21:51 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 18 21:50 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 6087406724 Dec 18 21:50 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        120 Dec 18 21:01 all-input-dids.txt
justIN time: 2025-12-18 23:06:49 UTC       justIN version: 01.05.03