justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 262973.2@dunegpschedd02.fnal.gov

Jobsub ID262973.2@dunegpschedd02.fnal.gov
Workflow ID11235
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-16 23:46:54
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-17 00:59:02
From worker nodeHostnamedunegli-7935992-0-fnpc23051.fnal.gov
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-16 23:47:44
Input filesusertests:pdhd_prod_beam__261613_113_1_20251212T161722Z_gen_g4_IonScintPDExt.root_262728_44_1_20251215T215841Z_PDInt.root
JobscriptExit code0
Real time1h (3875s)
CPU time1h (3691s = 95%)
Max RSS bytes7339454464 (6999 MiB)
Outputting started2025-12-17 00:52:19
Output files
Finished2025-12-17 00:59:02
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

call=18 input 0: frame: ident=199 time=0 tick=512 with 52237 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":24431 [0] "wiener0":27806 [27806] ] cmm:[ bad:0 lf_noisy:279 ] input 1: frame: ident=199 time=0 tick=512 with 57647 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":23497 [0] "wiener1":34150 [34150] ] cmm:[ bad:0 ] input 2: frame: ident=199 time=0 tick=512 with 34399 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":15078 [0] "wiener2":19321 [19321] ] cmm:[ bad:0 ] input 3: frame: ident=199 time=0 tick=512 with 49926 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":20912 [0] "wiener3":29014 [29014] ] cmm:[ bad:0 ] output: frame: ident=199 time=0 tick=512 with 194209 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":24431 [0] "gauss1":23497 [0] "gauss2":15078 [0] "gauss3":20912 [0] "wiener0":27806 [27806] "wiener1":34150 [34150] "wiener2":19321 [19321] "wiener3":29014 [29014] ] cmm:[ bad:0 lf_noisy:279 ] 
Retagger: tagging trace set: wiener with 110291 traces, 110291 summary
Retagger: tagging trace set: gauss with 83918 traces, 0 summary
[00:51:49.423] D [  glue  ] sink frame: #199 @0 with 194209 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[00:51:49.423] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[00:51:49.423] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[00:51:49.423] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[00:51:49.423] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[00:51:49.423] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[00:51:49.423] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[00:51:49.423] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[00:51:49.423] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[00:51:49.423] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[00:51:49.423] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[00:51:49.423] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[00:51:49.423] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[00:51:49.423] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[00:51:49.423] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[00:51:49.423] D [  glue  ] frame sink sees EOS
[00:51:49.423] D [ pgraph ] <Pgrapher:> graph execution complete 
[00:51:49.423] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 7.8 sec
[00:51:49.423] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.39 sec
[00:51:49.423] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.31 sec
[00:51:49.423] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 5.14 sec
[00:51:49.423] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.97 sec
[00:51:49.423] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.94 sec
[00:51:49.423] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.94 sec
[00:51:49.423] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 1.93 sec
[00:51:49.423] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[00:51:49.423] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:51:49.423] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:51:49.423] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:51:49.423] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[00:51:49.423] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[00:51:49.423] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[00:51:49.423] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[00:51:49.423] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[00:51:49.423] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[00:51:49.423] I [ timer  ] Timer: Total node execution : 31.42999995686114 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 83918 traces tagged "gauss"
FrameSaver: q=2.92013e+06 n=1382430 tag=gauss
wclsFrameSaver: saving 110291 traces tagged "wiener"
FrameSaver: q=3.54882e+06 n=1317630 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 57498 XUs and 54615 XVs -> 1668 XUVs
C:0 T:3 29 XUs and 27 XVs -> 3 XUVs
C:0 T:4 7 XUs and 16 XVs -> 2 XUVs
C:0 T:5 18191 XUs and 22202 XVs -> 655 XUVs
C:0 T:6 35630 XUs and 53064 XVs -> 1858 XUVs
C:0 T:7 27 XUs and 85 XVs -> 4 XUVs
4190 XUVs total
2385 collection wire objects
4190 potential space points
Neighbour search...
95416 tests to find 56006 neighbours
Iterating with no regularization...
Begin: 4.00324e+07
0 3.79454e+07
1 3.77395e+07
2 3.77235e+07
Now with regularization...
Begin: 3.54398e+07
0 3.54324e+07
17-Dec-2025 00:52:18 UTC  Closed output file "pdhd_prod_beam__261613_113_1_20251212T161722Z_gen_g4_IonScintPDExt.root_262728_44_1_20251215T215841Z_PDInt.root_262973_2_1_20251216T234747Z_detsim_reco1.root"
17-Dec-2025 00:52:18 UTC  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           50.0361       56.0286       61.9203       56.6426       4.17415        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.000618788   0.00238869    0.00354883     0.0027261    0.000971155      10     
reco:wclsdatahd:WireCellToolkit                      31.692        32.2482       33.9412       32.068       0.617799        10     
reco:wirefilter:WireFilter                         0.00871953     0.0101197     0.0117361     0.0101502      0.00112        10     
reco:gaushit:GausHitFinder                          0.301532      0.370466      0.432308      0.372302      0.0404792       10     
reco:nhitsfilter:NumberOfHitsFilter                0.000119259   0.000185547   0.000494059   0.000153961   0.000105811      10     
reco:reco3d:SpacePointSolver                        0.145861      0.278556      0.652886      0.244042      0.141616        10     
reco:hitpdune:DisambigFromSpacePoints               0.146388      0.252339      0.431905      0.228751      0.0922935       10     
reco:pandora:StandardPandora                         2.92665       4.59725       6.80692       4.41368       1.13992        10     
reco:pandoraWriter:StandardPandora                  0.0819441     0.107449      0.133712      0.105377      0.0176127       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.325914      0.468637      0.848319      0.453325      0.142801        10     
reco:pandoraShower:LArPandoraShowerCreation         0.315361      0.493842      0.703366      0.441185      0.143449        10     
reco:pandoracalo:Calorimetry                        0.208768      0.287498      0.455859      0.284402      0.066267        10     
reco:pandoracalonosce:Calorimetry                   0.202275      0.276902      0.434161      0.274591      0.0625516       10     
reco:pandoraShowercalo:ShowerCalorimetry            0.349069      0.524812      0.759236      0.470177      0.145064        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.344738       0.51765      0.748832      0.461577      0.144403        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0215902     0.0278279     0.0366935     0.0275439    0.00423666       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.020707      0.026782      0.035442      0.0268321    0.00398099       10     
reco:emtrkmichelid:EmTrackMichelId                   11.907        14.7706       17.5592       15.1469       1.96931        10     
[art]:TriggerResults:TriggerResultInserter         1.4318e-05     2.653e-05    9.5574e-05    1.90465e-05   2.30813e-05      10     
end_path:out1:RootOutput                            6.833e-06    1.01977e-05   3.4106e-05    7.5445e-06    7.97849e-06      10     
end_path:out1:RootOutput(write)                     0.613747      0.745876      0.873938      0.750408      0.0935864       10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5931.41 MB
  Peak resident set size usage (VmHWM): 3770.26 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12691540
-rw-r--r-- 1 dunegli fnalgrid     231394 Dec 17 00:52 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        201 Dec 17 00:52 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 17 00:52 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Dec 17 00:52 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     139264 Dec 17 00:52 mem.db
-rw-r--r-- 1 dunegli fnalgrid      28672 Dec 17 00:52 time.db
-rw-r--r-- 1 dunegli fnalgrid 6578670759 Dec 17 00:52 pdhd_prod_beam__261613_113_1_20251212T161722Z_gen_g4_IonScintPDExt.root_262728_44_1_20251215T215841Z_PDInt.root_262973_2_1_20251216T234747Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   36826920 Dec 17 00:51 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Dec 17 00:41 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 17 00:41 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 6380182318 Dec 17 00:41 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        122 Dec 16 23:47 all-input-dids.txt
justIN time: 2025-12-19 04:23:38 UTC       justIN version: 01.05.03