justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 262926.1@dunegpschedd02.fnal.gov

Jobsub ID262926.1@dunegpschedd02.fnal.gov
Workflow ID11227
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-16 17:44:37
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-16 18:45:09
From worker nodeHostnamedunegli-8081979-0-fnpc23045.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-16 17:45:34
Input filesusertests:pdhd_prod_beam__265029_13_1_20251208T213855Z_gen_g4_IonScintPDExt.root_260262_24_1_20251210T192009Z_PDInt.root
JobscriptExit code0
Real time56m (3418s)
CPU time55m (3345s = 97%)
Max RSS bytes7245119488 (6909 MiB)
Outputting started2025-12-16 18:42:32
Output files
Finished2025-12-16 18:45:09
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=159 time=0 tick=512 with 57302 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":27358 [0] "wiener0":29944 [29944] ] cmm:[ bad:0 lf_noisy:178 ] input 1: frame: ident=159 time=0 tick=512 with 37473 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":16574 [0] "wiener1":20899 [20899] ] cmm:[ bad:0 lf_noisy:1 ] input 2: frame: ident=159 time=0 tick=512 with 47414 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":19814 [0] "wiener2":27600 [27600] ] cmm:[ bad:0 ] input 3: frame: ident=159 time=0 tick=512 with 38136 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":16572 [0] "wiener3":21564 [21564] ] cmm:[ bad:0 ] output: frame: ident=159 time=0 tick=512 with 180325 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":27358 [0] "gauss1":16574 [0] "gauss2":19814 [0] "gauss3":16572 [0] "wiener0":29944 [29944] "wiener1":20899 [20899] "wiener2":27600 [27600] "wiener3":21564 [21564] ] cmm:[ bad:0 lf_noisy:179 ] 
Retagger: tagging trace set: wiener with 100007 traces, 100007 summary
Retagger: tagging trace set: gauss with 80318 traces, 0 summary
[18:42:08.767] D [  glue  ] sink frame: #159 @0 with 180325 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[18:42:08.767] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[18:42:08.767] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[18:42:08.767] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[18:42:08.767] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[18:42:08.767] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[18:42:08.767] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[18:42:08.767] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[18:42:08.767] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[18:42:08.767] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[18:42:08.767] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[18:42:08.767] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[18:42:08.767] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[18:42:08.767] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[18:42:08.767] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[18:42:08.767] D [  glue  ] frame sink sees EOS
[18:42:08.767] D [ pgraph ] <Pgrapher:> graph execution complete 
[18:42:08.767] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.08 sec
[18:42:08.767] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.53 sec
[18:42:08.767] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.43 sec
[18:42:08.767] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.37 sec
[18:42:08.767] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.98 sec
[18:42:08.767] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.96 sec
[18:42:08.767] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.96 sec
[18:42:08.767] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.96 sec
[18:42:08.767] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[18:42:08.767] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:42:08.767] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[18:42:08.767] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:42:08.767] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:42:08.767] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:42:08.767] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[18:42:08.767] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[18:42:08.767] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[18:42:08.767] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[18:42:08.767] I [ timer  ] Timer: Total node execution : 32.279999980702996 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 80318 traces tagged "gauss"
FrameSaver: q=2.33456e+06 n=1210039 tag=gauss
wclsFrameSaver: saving 100007 traces tagged "wiener"
FrameSaver: q=3.25015e+06 n=1166896 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 20805 XUs and 23191 XVs -> 364 XUVs
C:0 T:4 45 XUs and 52 XVs -> 19 XUVs
C:0 T:5 35832 XUs and 51299 XVs -> 2331 XUVs
C:0 T:6 36718 XUs and 21289 XVs -> 577 XUVs
C:0 T:7 3 XUs and 7 XVs -> 0 XUVs
3291 XUVs total
1303 collection wire objects
3291 potential space points
Neighbour search...
119881 tests to find 74058 neighbours
Iterating with no regularization...
Begin: 1.50478e+07
0 1.36831e+07
1 1.3594e+07
2 1.35828e+07
Now with regularization...
Begin: 1.25972e+07
0 1.25928e+07
16-Dec-2025 18:42:32 UTC  Closed output file "pdhd_prod_beam__265029_13_1_20251208T213855Z_gen_g4_IonScintPDExt.root_260262_24_1_20251210T192009Z_PDInt.root_262926_1_1_20251216T174537Z_detsim_reco1.root"
16-Dec-2025 18:42:32 UTC  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           50.1151       55.8252       64.8662       54.8752       3.94519        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.000406211   0.00226797    0.00356065    0.00245074    0.00101464       10     
reco:wclsdatahd:WireCellToolkit                      32.1567       32.7769       33.3205       32.8551      0.381761        10     
reco:wirefilter:WireFilter                         0.00861486    0.00981936     0.0115508     0.0096254    0.00093463       10     
reco:gaushit:GausHitFinder                          0.278438      0.358705      0.464708      0.339412      0.053954        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000104291   0.000156262   0.000398577   0.000131393   8.15528e-05      10     
reco:reco3d:SpacePointSolver                         0.13921      0.304717      0.956136      0.222528      0.228385        10     
reco:hitpdune:DisambigFromSpacePoints               0.136462      0.212307      0.376825      0.193421      0.0663901       10     
reco:pandora:StandardPandora                         2.48746       4.85952       8.59111       4.21712       1.66423        10     
reco:pandoraWriter:StandardPandora                  0.0833831     0.0991599     0.126993      0.0942145     0.0144208       10     
reco:pandoraTrack:LArPandoraTrackCreation            0.2863       0.480403      0.643646      0.456422      0.104359        10     
reco:pandoraShower:LArPandoraShowerCreation         0.313345      0.454006      0.724956      0.445843      0.122073        10     
reco:pandoracalo:Calorimetry                        0.202925      0.299399      0.380891       0.29892      0.0520933       10     
reco:pandoracalonosce:Calorimetry                   0.196113      0.287521      0.369326      0.285922      0.050476        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.330617      0.490599      0.778223      0.471262      0.130584        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.327242      0.482159      0.761874      0.466158      0.129117        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0214006     0.0262987     0.0315741     0.026329     0.00309775       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0205044     0.0250405     0.0301717     0.0254647    0.00243933       10     
reco:emtrkmichelid:EmTrackMichelId                   11.3885       13.9144       17.733        13.8063       1.6512         10     
[art]:TriggerResults:TriggerResultInserter         1.6121e-05    2.58849e-05   9.4432e-05    1.84305e-05   2.28865e-05      10     
end_path:out1:RootOutput                            3.587e-06    8.2468e-06    3.0449e-05     6.567e-06    7.60059e-06      10     
end_path:out1:RootOutput(write)                     0.589817      0.724261      0.907595      0.720357      0.0802434       10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6091.67 MB
  Peak resident set size usage (VmHWM): 3930.77 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11933260
-rw-r--r-- 1 dunegli fnalgrid     231333 Dec 16 18:42 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        200 Dec 16 18:42 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 16 18:42 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Dec 16 18:42 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     139264 Dec 16 18:42 mem.db
-rw-r--r-- 1 dunegli fnalgrid      28672 Dec 16 18:42 time.db
-rw-r--r-- 1 dunegli fnalgrid 6178548698 Dec 16 18:42 pdhd_prod_beam__265029_13_1_20251208T213855Z_gen_g4_IonScintPDExt.root_260262_24_1_20251210T192009Z_PDInt.root_262926_1_1_20251216T174537Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   34892460 Dec 16 18:42 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Dec 16 18:32 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 16 18:32 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 5993822125 Dec 16 18:32 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        121 Dec 16 17:45 all-input-dids.txt
justIN time: 2025-12-19 04:23:24 UTC       justIN version: 01.05.03