justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 263305.26@dunegpschedd02.fnal.gov

Jobsub ID263305.26@dunegpschedd02.fnal.gov
Workflow ID11264
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 19:31:07
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-18 22:02:17
From worker nodeHostnamedunegli-7951903-0-fnpc23032.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 20:52:05
Input filesusertests:pdhd_prod_beam__261612_22_1_20251212T155525Z_gen_g4_IonScintPDExt.root_268535_35_1_20251215T215104Z_PDInt.root
JobscriptExit code0
Real time1h (4159s)
CPU time1h (3951s = 94%)
Max RSS bytes8205930496 (7825 MiB)
Outputting started2025-12-18 22:01:25
Output filesusertests:pdhd_prod_beam__261612_22_1_20251212T155525Z_gen_g4_IonScintPDExt.root_268535_35_1_20251215T215104Z_PDInt.root_263305_26_1_20251218T205208Z_detsim_reco1.root
Finished2025-12-18 22:02:17
Saved logsjustin-logs:263305.26-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

tick=512 with 52021 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":24789 [0] "wiener0":27232 [27232] ] cmm:[ bad:0 lf_noisy:119 ] input 1: frame: ident=179 time=0 tick=512 with 38591 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":16201 [0] "wiener1":22390 [22390] ] cmm:[ bad:0 lf_noisy:1 ] input 2: frame: ident=179 time=0 tick=512 with 43768 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":19135 [0] "wiener2":24633 [24633] ] cmm:[ bad:0 lf_noisy:1 ] input 3: frame: ident=179 time=0 tick=512 with 59181 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":25088 [0] "wiener3":34093 [34093] ] cmm:[ bad:0 ] output: frame: ident=179 time=0 tick=512 with 193561 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":24789 [0] "gauss1":16201 [0] "gauss2":19135 [0] "gauss3":25088 [0] "wiener0":27232 [27232] "wiener1":22390 [22390] "wiener2":24633 [24633] "wiener3":34093 [34093] ] cmm:[ bad:0 lf_noisy:121 ] 
Retagger: tagging trace set: wiener with 108348 traces, 108348 summary
Retagger: tagging trace set: gauss with 85213 traces, 0 summary
[22:00:53.180] D [  glue  ] sink frame: #179 @0 with 193561 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[22:00:53.180] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[22:00:53.180] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[22:00:53.180] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[22:00:53.180] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[22:00:53.180] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[22:00:53.180] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[22:00:53.180] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[22:00:53.180] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[22:00:53.180] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[22:00:53.180] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[22:00:53.180] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[22:00:53.180] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[22:00:53.180] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[22:00:53.180] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[22:00:53.180] D [  glue  ] frame sink sees EOS
[22:00:53.180] D [ pgraph ] <Pgrapher:> graph execution complete 
[22:00:53.180] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.67 sec
[22:00:53.180] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.48 sec
[22:00:53.180] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.33 sec
[22:00:53.180] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.33 sec
[22:00:53.180] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.94 sec
[22:00:53.180] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.93 sec
[22:00:53.180] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.93 sec
[22:00:53.180] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.9 sec
[22:00:53.180] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[22:00:53.180] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[22:00:53.180] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[22:00:53.180] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[22:00:53.180] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[22:00:53.180] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[22:00:53.180] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[22:00:53.180] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[22:00:53.180] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[22:00:53.180] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[22:00:53.180] I [ timer  ] Timer: Total node execution : 31.529999870806932 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 85213 traces tagged "gauss"
FrameSaver: q=3.00561e+06 n=1430335 tag=gauss
wclsFrameSaver: saving 108348 traces tagged "wiener"
FrameSaver: q=3.77567e+06 n=1386409 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 18594 XUs and 26099 XVs -> 3139 XUVs
C:0 T:3 81 XUs and 91 XVs -> 3 XUVs
C:0 T:4 5 XUs and 8 XVs -> 0 XUVs
C:0 T:5 23149 XUs and 25673 XVs -> 374 XUVs
C:0 T:6 96698 XUs and 68688 XVs -> 3949 XUVs
C:0 T:7 51 XUs and 76 XVs -> 26 XUVs
7491 XUVs total
2155 collection wire objects
7491 potential space points
Neighbour search...
573171 tests to find 308880 neighbours
Iterating with no regularization...
Begin: 5.43842e+07
0 4.88694e+07
1 4.84427e+07
2 4.83797e+07
3 4.83595e+07
Now with regularization...
Begin: 4.50468e+07
0 4.50265e+07
18-Dec-2025 22:01:24 UTC  Closed output file "pdhd_prod_beam__261612_22_1_20251212T155525Z_gen_g4_IonScintPDExt.root_268535_35_1_20251215T215104Z_PDInt.root_263305_26_1_20251218T205208Z_detsim_reco1.root"
18-Dec-2025 22:01:24 UTC  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           46.1361       61.0781       75.5619       62.3266       10.9606        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.000793842    0.0028248    0.00491078    0.00287993    0.00125106       10     
reco:wclsdatahd:WireCellToolkit                      31.5565       32.6782       33.8734       32.4701       0.63752        10     
reco:wirefilter:WireFilter                         0.00842229     0.0106237     0.0129962     0.0107079    0.00154072       10     
reco:gaushit:GausHitFinder                          0.231128      0.416298       0.62032       0.46509      0.132356        10     
reco:nhitsfilter:NumberOfHitsFilter                8.9712e-05    0.000200691   0.000345853   0.00021089    6.72481e-05      10     
reco:reco3d:SpacePointSolver                         0.11032      0.410735      0.792589      0.425184      0.225195        10     
reco:hitpdune:DisambigFromSpacePoints               0.0763056     0.358594      0.585479      0.439851      0.200475        10     
reco:pandora:StandardPandora                         2.22093       6.26223       13.265        6.30465       3.48101        10     
reco:pandoraWriter:StandardPandora                  0.0695244      0.12244      0.168946      0.130628       0.03689        10     
reco:pandoraTrack:LArPandoraTrackCreation           0.212873      0.609334       1.13561      0.609858      0.307686        10     
reco:pandoraShower:LArPandoraShowerCreation         0.151044      0.690398       1.19747      0.677726      0.370654        10     
reco:pandoracalo:Calorimetry                        0.128021      0.374722       0.69127      0.375901      0.182789        10     
reco:pandoracalonosce:Calorimetry                   0.121616      0.360547      0.665125      0.356295      0.177467        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.147884      0.748905       1.41327      0.707207       0.43012        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.143638      0.736567       1.4038       0.693573       0.42214        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0215929     0.034192      0.0538191     0.0340136     0.0100065       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0184183     0.0330963     0.0530245     0.033083      0.010275        10     
reco:emtrkmichelid:EmTrackMichelId                   9.65204       16.3604       21.9338       17.4649       4.6896         10     
[art]:TriggerResults:TriggerResultInserter         1.9127e-05    2.88981e-05   8.9381e-05    2.32245e-05   2.02661e-05      10     
end_path:out1:RootOutput                            6.372e-06    1.08499e-05   3.1541e-05    8.5915e-06    6.97281e-06      10     
end_path:out1:RootOutput(write)                      0.54664       0.83487       1.12311      0.866217      0.204982        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6460.49 MB
  Peak resident set size usage (VmHWM): 4291.29 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 13627708
-rw-r--r-- 1 dunegli fnalgrid     231472 Dec 18 22:01 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        200 Dec 18 22:01 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 18 22:01 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Dec 18 22:01 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     139264 Dec 18 22:01 mem.db
-rw-r--r-- 1 dunegli fnalgrid      28672 Dec 18 22:01 time.db
-rw-r--r-- 1 dunegli fnalgrid 7066423845 Dec 18 22:01 pdhd_prod_beam__261612_22_1_20251212T155525Z_gen_g4_IonScintPDExt.root_268535_35_1_20251215T215104Z_PDInt.root_263305_26_1_20251218T205208Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   41515670 Dec 18 22:01 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Dec 18 21:49 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 18 21:49 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 6846370539 Dec 18 21:49 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        121 Dec 18 20:52 all-input-dids.txt
justIN time: 2025-12-18 23:05:59 UTC       justIN version: 01.05.03