justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 268896.117@dunegpschedd01.fnal.gov

Jobsub ID268896.117@dunegpschedd01.fnal.gov
Workflow ID11239
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-17 17:53:51
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-17 21:09:46
From worker nodeHostnamedunegli-7943783-0-fnpc8209.fnal.gov
cpuinfoIntel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-17 19:11:30
Input filesusertests:pdhd_prod_beam__267379_71_1_20251212T161710Z_gen_g4_IonScintPDExt.root_262726_103_1_20251215T212449Z_PDInt.root
JobscriptExit code0
Real time1h (7078s)
CPU time1h (6606s = 93%)
Max RSS bytes7629479936 (7276 MiB)
Outputting started2025-12-17 21:09:29
Output files
Finished2025-12-17 21:09:46
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

s0":29590 [0] "wiener0":32396 [32396] ] cmm:[ bad:0 lf_noisy:196 ] input 1: frame: ident=189 time=0 tick=512 with 46299 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":19189 [0] "wiener1":27110 [27110] ] cmm:[ bad:0 ] input 2: frame: ident=189 time=0 tick=512 with 37809 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":16484 [0] "wiener2":21325 [21325] ] cmm:[ bad:0 lf_noisy:1 ] input 3: frame: ident=189 time=0 tick=512 with 66737 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":27891 [0] "wiener3":38846 [38846] ] cmm:[ bad:0 ] output: frame: ident=189 time=0 tick=512 with 212831 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":29590 [0] "gauss1":19189 [0] "gauss2":16484 [0] "gauss3":27891 [0] "wiener0":32396 [32396] "wiener1":27110 [27110] "wiener2":21325 [21325] "wiener3":38846 [38846] ] cmm:[ bad:0 lf_noisy:197 ] 
Retagger: tagging trace set: wiener with 119677 traces, 119677 summary
Retagger: tagging trace set: gauss with 93154 traces, 0 summary
[21:07:45.660] D [  glue  ] sink frame: #189 @0 with 212831 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[21:07:45.660] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[21:07:45.661] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[21:07:45.661] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[21:07:45.661] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[21:07:45.661] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[21:07:45.661] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[21:07:45.661] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[21:07:45.661] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[21:07:45.661] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[21:07:45.661] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[21:07:45.661] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[21:07:45.661] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[21:07:45.661] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[21:07:45.661] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[21:07:45.661] D [  glue  ] frame sink sees EOS
[21:07:45.661] D [ pgraph ] <Pgrapher:> graph execution complete 
[21:07:45.661] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 17.69 sec
[21:07:45.661] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.73 sec
[21:07:45.661] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.24 sec
[21:07:45.661] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.87 sec
[21:07:45.661] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.83 sec
[21:07:45.661] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.72 sec
[21:07:45.661] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.26 sec
[21:07:45.661] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.13 sec
[21:07:45.661] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.05 sec
[21:07:45.661] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.02 sec
[21:07:45.661] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[21:07:45.661] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[21:07:45.661] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[21:07:45.661] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:07:45.661] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[21:07:45.661] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[21:07:45.661] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[21:07:45.661] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[21:07:45.661] I [ timer  ] Timer: Total node execution : 68.55999979004264 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 93154 traces tagged "gauss"
FrameSaver: q=3.0413e+06 n=1539533 tag=gauss
wclsFrameSaver: saving 119677 traces tagged "wiener"
FrameSaver: q=3.7992e+06 n=1501053 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 50509 XUs and 68496 XVs -> 1330 XUVs
C:0 T:3 43 XUs and 359 XVs -> 3 XUVs
C:0 T:4 3 XUs and 8 XVs -> 0 XUVs
C:0 T:5 14939 XUs and 17551 XVs -> 916 XUVs
C:0 T:6 91958 XUs and 123891 XVs -> 4078 XUVs
C:0 T:7 55 XUs and 49 XVs -> 7 XUVs
6334 XUVs total
2515 collection wire objects
6334 potential space points
Neighbour search...
183284 tests to find 106864 neighbours
Iterating with no regularization...
Begin: 3.40357e+07
0 3.08908e+07
1 3.05693e+07
2 3.05234e+07
3 3.05081e+07
Now with regularization...
Begin: 2.82733e+07
0 2.82643e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
17-Dec-2025 21:09:24 UTC  Closed output file "pdhd_prod_beam__267379_71_1_20251212T161710Z_gen_g4_IonScintPDExt.root_262726_103_1_20251215T212449Z_PDInt.root_268896_117_1_20251217T191137Z_detsim_reco1.root"
17-Dec-2025 21:09:24 UTC  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           103.174       131.455       166.584       125.962       20.2312        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.00131288    0.00611328     0.0238883    0.00487779    0.00612068       10     
reco:wclsdatahd:WireCellToolkit                      61.6144       72.2644       89.1567       70.1961       7.96762        10     
reco:wirefilter:WireFilter                          0.0266824      0.03539      0.0480938     0.035187     0.00510715       10     
reco:gaushit:GausHitFinder                           0.51033       0.79204       1.15267      0.819555      0.169639        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000202267   0.000312875   0.000533962   0.000287406   9.16185e-05      10     
reco:reco3d:SpacePointSolver                        0.267444       0.69488       1.89203      0.476749      0.496193        10     
reco:hitpdune:DisambigFromSpacePoints               0.254044      0.679069       1.38531      0.585047      0.337009        10     
reco:pandora:StandardPandora                         6.12452       11.7075       20.7358       12.2792       4.69524        10     
reco:pandoraWriter:StandardPandora                  0.177015      0.318969      0.871018      0.237712      0.192794        10     
reco:pandoraTrack:LArPandoraTrackCreation           0.892358       1.52012       2.38305       1.48408      0.455039        10     
reco:pandoraShower:LArPandoraShowerCreation          0.72269       1.48121       2.74626       1.45193       0.66584        10     
reco:pandoracalo:Calorimetry                        0.469468      0.783346       1.19732      0.803732      0.213137        10     
reco:pandoracalonosce:Calorimetry                   0.418701      0.757212       1.18444      0.793609       0.22969        10     
reco:pandoraShowercalo:ShowerCalorimetry             0.6589         1.476        2.77077       1.47654      0.681201        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.624178       1.45677       2.9287        1.43711      0.692045        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0436787     0.0665236     0.110374      0.0621312     0.0202726       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0436982     0.062117      0.108662      0.057633      0.0187735       10     
reco:emtrkmichelid:EmTrackMichelId                   24.5395       35.014        55.286        31.0613       9.48382        10     
[art]:TriggerResults:TriggerResultInserter         2.3295e-05    4.1486e-05    8.0875e-05    3.75705e-05   1.57425e-05      10     
end_path:out1:RootOutput                            8.079e-06    1.10742e-05   2.1067e-05    9.8375e-06    3.67455e-06      10     
end_path:out1:RootOutput(write)                      1.07421       2.28003       3.79676       2.1723       0.812499        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5909.98 MB
  Peak resident set size usage (VmHWM): 3749.34 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12457076
-rw-r--r-- 1 dunegli fnalgrid     231412 Dec 17 21:09 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        201 Dec 17 21:09 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 17 21:09 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Dec 17 21:09 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     139264 Dec 17 21:09 mem.db
-rw-r--r-- 1 dunegli fnalgrid      28672 Dec 17 21:09 time.db
-rw-r--r-- 1 dunegli fnalgrid 6457418860 Dec 17 21:09 pdhd_prod_beam__267379_71_1_20251212T161710Z_gen_g4_IonScintPDExt.root_262726_103_1_20251215T212449Z_PDInt.root_268896_117_1_20251217T191137Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   36188040 Dec 17 21:08 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Dec 17 20:43 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 17 20:42 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 6261986003 Dec 17 20:42 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        122 Dec 17 19:11 all-input-dids.txt
justIN time: 2025-12-19 04:23:37 UTC       justIN version: 01.05.03