justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 268896.43@dunegpschedd01.fnal.gov

Jobsub ID268896.43@dunegpschedd01.fnal.gov
Workflow ID11239
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-17 17:53:51
SiteUS_FNAL-T1
EntryCMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat2025-12-17 20:48:06
From worker nodeHostnamedunegli-49199-0-cmswn2413.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-17 18:32:25
Input filesusertests:pdhd_prod_beam__267379_22_1_20251212T160839Z_gen_g4_IonScintPDExt.root_262726_96_1_20251215T212416Z_PDInt.root
JobscriptExit code0
Real time2h (7974s)
CPU time1h (6002s = 75%)
Max RSS bytes6857756672 (6540 MiB)
Outputting started2025-12-17 20:45:20
Output files
Finished2025-12-17 20:48:06
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

[20:44:31.955] D [  glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=189 time=0 tick=512 with 40174 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":19369 [0] "wiener0":20805 [20805] ] cmm:[ bad:0 lf_noisy:130 ] input 1: frame: ident=189 time=0 tick=512 with 43772 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":18706 [0] "wiener1":25066 [25066] ] cmm:[ bad:0 ] input 2: frame: ident=189 time=0 tick=512 with 44554 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":19265 [0] "wiener2":25289 [25289] ] cmm:[ bad:0 lf_noisy:1 ] input 3: frame: ident=189 time=0 tick=512 with 32637 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":13820 [0] "wiener3":18817 [18817] ] cmm:[ bad:0 ] output: frame: ident=189 time=0 tick=512 with 161137 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":19369 [0] "gauss1":18706 [0] "gauss2":19265 [0] "gauss3":13820 [0] "wiener0":20805 [20805] "wiener1":25066 [25066] "wiener2":25289 [25289] "wiener3":18817 [18817] ] cmm:[ bad:0 lf_noisy:131 ] 
Retagger: tagging trace set: wiener with 89977 traces, 89977 summary
Retagger: tagging trace set: gauss with 71160 traces, 0 summary
[20:44:31.966] D [  glue  ] sink frame: #189 @0 with 161137 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[20:44:31.966] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[20:44:31.966] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[20:44:31.966] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[20:44:31.966] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[20:44:31.966] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[20:44:31.966] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[20:44:31.966] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[20:44:31.966] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[20:44:31.966] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[20:44:31.966] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[20:44:31.966] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[20:44:31.966] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[20:44:31.966] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[20:44:31.966] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[20:44:31.966] D [  glue  ] frame sink sees EOS
[20:44:31.966] D [ pgraph ] <Pgrapher:> graph execution complete 
[20:44:31.966] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.56 sec
[20:44:31.966] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.91 sec
[20:44:31.966] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.57 sec
[20:44:31.966] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.48 sec
[20:44:31.966] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.82 sec
[20:44:31.966] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.81 sec
[20:44:31.966] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.78 sec
[20:44:31.966] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.76 sec
[20:44:31.966] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.02 sec
[20:44:31.966] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:44:31.966] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:44:31.967] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:44:31.967] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:44:31.967] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[20:44:31.967] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[20:44:31.967] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[20:44:31.967] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[20:44:31.967] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[20:44:31.967] I [ timer  ] Timer: Total node execution : 44.710000295192 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 71160 traces tagged "gauss"
FrameSaver: q=2.10062e+06 n=1088578 tag=gauss
wclsFrameSaver: saving 89977 traces tagged "wiener"
FrameSaver: q=2.62811e+06 n=1061563 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 55968 XUs and 46872 XVs -> 1786 XUVs
C:0 T:5 28155 XUs and 28811 XVs -> 878 XUVs
C:0 T:6 23325 XUs and 47928 XVs -> 4523 XUVs
C:0 T:7 3 XUs and 2 XVs -> 0 XUVs
7187 XUVs total
1810 collection wire objects
7187 potential space points
Neighbour search...
807607 tests to find 428196 neighbours
Iterating with no regularization...
Begin: 2.32299e+07
0 2.01029e+07
1 1.98883e+07
2 1.9847e+07
3 1.98318e+07
Now with regularization...
Begin: 1.83075e+07
0 1.82969e+07
17-Dec-2025 20:45:18 UTC  Closed output file "pdhd_prod_beam__267379_22_1_20251212T160839Z_gen_g4_IonScintPDExt.root_262726_96_1_20251215T212416Z_PDInt.root_268896_43_1_20251217T183230Z_detsim_reco1.root"
17-Dec-2025 20:45:18 UTC  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           79.9026       89.5686       99.4367       90.2193       6.4657         10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.00150712     0.0144798      0.11164     0.00417271     0.0324083       10     
reco:wclsdatahd:WireCellToolkit                      44.9177       48.2166       53.1775       47.1814       2.68177        10     
reco:wirefilter:WireFilter                           0.01411      0.0166856     0.0200132     0.0158548    0.00191394       10     
reco:gaushit:GausHitFinder                            0.479       0.552855      0.722233      0.525398      0.0706612       10     
reco:nhitsfilter:NumberOfHitsFilter                0.000229042   0.000306409   0.000612595   0.000265041   0.000108561      10     
reco:reco3d:SpacePointSolver                        0.214459      0.550173       1.08438       0.50501      0.252799        10     
reco:hitpdune:DisambigFromSpacePoints               0.227136      0.335781      0.520189      0.303973      0.104019        10     
reco:pandora:StandardPandora                         4.52607       6.50521       12.1194       5.69357       2.22753        10     
reco:pandoraWriter:StandardPandora                  0.147489      0.265543      0.899828      0.179809      0.219061        10     
reco:pandoraTrack:LArPandoraTrackCreation           0.440096      0.714315       1.02711      0.676596      0.167958        10     
reco:pandoraShower:LArPandoraShowerCreation         0.521876      0.718074       1.11825      0.626127      0.195962        10     
reco:pandoracalo:Calorimetry                        0.269886      0.374129      0.497422      0.358474      0.0625232       10     
reco:pandoracalonosce:Calorimetry                   0.261929      0.362819       0.48622      0.351758      0.0624939       10     
reco:pandoraShowercalo:ShowerCalorimetry            0.471442      0.688371       1.03681      0.595754      0.189706        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.464631      0.705229       1.02688      0.623015      0.200334        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0351656     0.0456637     0.055989      0.0442582    0.00607002       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0342004     0.0445702     0.0531222     0.0437666    0.00537264       10     
reco:emtrkmichelid:EmTrackMichelId                   22.8502       28.0349       34.475        26.1784       4.03504        10     
[art]:TriggerResults:TriggerResultInserter          2.611e-05    3.87693e-05   9.0121e-05    3.0795e-05    1.96428e-05      10     
end_path:out1:RootOutput                            5.19e-06      8.685e-06     2.671e-05     5.765e-06    6.31605e-06      10     
end_path:out1:RootOutput(write)                      1.00108       1.37709       2.81035       1.13356      0.514874        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6040.27 MB
  Peak resident set size usage (VmHWM): 3810.15 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11648360
-rw-r--r-- 1 dunegli fnalgrid     231196 Dec 17 20:45 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        200 Dec 17 20:45 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 17 20:45 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Dec 17 20:45 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     139264 Dec 17 20:45 mem.db
-rw-r--r-- 1 dunegli fnalgrid      28672 Dec 17 20:45 time.db
-rw-r--r-- 1 dunegli fnalgrid 6040762385 Dec 17 20:45 pdhd_prod_beam__267379_22_1_20251212T160839Z_gen_g4_IonScintPDExt.root_262726_96_1_20251215T212416Z_PDInt.root_268896_43_1_20251217T183230Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   34622190 Dec 17 20:44 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Dec 17 20:09 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 17 20:09 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 5852081352 Dec 17 20:09 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        121 Dec 17 18:32 all-input-dids.txt
justIN time: 2025-12-19 23:27:03 UTC       justIN version: 01.05.03