justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 268896.141@dunegpschedd01.fnal.gov

Jobsub ID268896.141@dunegpschedd01.fnal.gov
Workflow ID11239
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-17 17:53:51
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-17 20:48:10
From worker nodeHostnamedunegli-8092434-0-fnpc9041.fnal.gov
cpuinfoIntel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-17 19:26:03
Input filesusertests:pdhd_prod_beam__267379_78_1_20251212T161846Z_gen_g4_IonScintPDExt.root_262726_21_1_20251215T212452Z_PDInt.root
JobscriptExit code0
Real time1h (4893s)
CPU time1h (4772s = 97%)
Max RSS bytes6848630784 (6531 MiB)
Outputting started2025-12-17 20:47:37
Output files
Finished2025-12-17 20:48:10
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

g_nf> call=18 input 0: frame: ident=189 time=0 tick=512 with 59218 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":27901 [0] "wiener0":31317 [31317] ] cmm:[ bad:0 lf_noisy:214 ] input 1: frame: ident=189 time=0 tick=512 with 59624 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":24631 [0] "wiener1":34993 [34993] ] cmm:[ bad:0 lf_noisy:4 ] input 2: frame: ident=189 time=0 tick=512 with 40189 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":17523 [0] "wiener2":22666 [22666] ] cmm:[ bad:0 ] input 3: frame: ident=189 time=0 tick=512 with 35484 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":15367 [0] "wiener3":20117 [20117] ] cmm:[ bad:0 lf_noisy:1 ] output: frame: ident=189 time=0 tick=512 with 194515 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":27901 [0] "gauss1":24631 [0] "gauss2":17523 [0] "gauss3":15367 [0] "wiener0":31317 [31317] "wiener1":34993 [34993] "wiener2":22666 [22666] "wiener3":20117 [20117] ] cmm:[ bad:0 lf_noisy:219 ] 
Retagger: tagging trace set: wiener with 109093 traces, 109093 summary
Retagger: tagging trace set: gauss with 85422 traces, 0 summary
[20:46:33.399] D [  glue  ] sink frame: #189 @0 with 194515 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[20:46:33.399] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[20:46:33.399] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[20:46:33.399] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[20:46:33.399] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[20:46:33.399] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[20:46:33.399] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[20:46:33.399] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[20:46:33.399] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[20:46:33.399] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[20:46:33.399] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[20:46:33.399] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[20:46:33.399] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[20:46:33.399] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[20:46:33.399] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[20:46:33.399] D [  glue  ] frame sink sees EOS
[20:46:33.399] D [ pgraph ] <Pgrapher:> graph execution complete 
[20:46:33.399] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 13.43 sec
[20:46:33.399] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.65 sec
[20:46:33.399] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.2 sec
[20:46:33.399] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.01 sec
[20:46:33.399] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.76 sec
[20:46:33.399] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.71 sec
[20:46:33.399] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.68 sec
[20:46:33.399] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.66 sec
[20:46:33.399] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.04 sec
[20:46:33.399] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.02 sec
[20:46:33.399] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:46:33.399] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:46:33.399] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:46:33.399] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:46:33.399] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[20:46:33.399] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[20:46:33.399] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[20:46:33.399] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[20:46:33.399] I [ timer  ] Timer: Total node execution : 52.16000014171004 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 85422 traces tagged "gauss"
FrameSaver: q=2.75351e+06 n=1342978 tag=gauss
wclsFrameSaver: saving 109093 traces tagged "wiener"
FrameSaver: q=3.70348e+06 n=1287759 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 68922 XUs and 74153 XVs -> 3309 XUVs
C:0 T:3 6 XUs and 13 XVs -> 0 XUVs
C:0 T:4 37 XUs and 65 XVs -> 15 XUVs
C:0 T:5 20630 XUs and 23528 XVs -> 1176 XUVs
C:0 T:6 20479 XUs and 20251 XVs -> 413 XUVs
4913 XUVs total
1970 collection wire objects
4913 potential space points
Neighbour search...
263111 tests to find 137682 neighbours
Iterating with no regularization...
Begin: 3.38199e+07
0 3.14193e+07
1 3.12568e+07
2 3.12344e+07
Now with regularization...
Begin: 2.93903e+07
0 2.93826e+07
17-Dec-2025 20:47:34 UTC  Closed output file "pdhd_prod_beam__267379_78_1_20251212T161846Z_gen_g4_IonScintPDExt.root_262726_21_1_20251215T212452Z_PDInt.root_268896_141_1_20251217T192607Z_detsim_reco1.root"
17-Dec-2025 20:47:34 UTC  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           85.528        96.9879       112.968       96.191        7.67359        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.00116662    0.00369468     0.0054126     0.0044391    0.00154859       10     
reco:wclsdatahd:WireCellToolkit                      52.9623       55.9632       60.6292       54.9486       2.66323        10     
reco:wirefilter:WireFilter                          0.0211688     0.0282617     0.0358984     0.0267796    0.00478491       10     
reco:gaushit:GausHitFinder                          0.320406      0.545411       0.68838      0.537484      0.102242        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000102244   0.000218848   0.000348853   0.000221204   6.76907e-05      10     
reco:reco3d:SpacePointSolver                        0.201537      0.530915       1.17562      0.377317      0.329449        10     
reco:hitpdune:DisambigFromSpacePoints                0.2498       0.491699      0.838794      0.473805      0.197674        10     
reco:pandora:StandardPandora                         3.52717       7.96452       12.2059       7.7766        2.77435        10     
reco:pandoraWriter:StandardPandora                  0.115149      0.178103      0.219415      0.182278      0.032494        10     
reco:pandoraTrack:LArPandoraTrackCreation           0.477569       1.04021       1.59256       1.03037        0.294         10     
reco:pandoraShower:LArPandoraShowerCreation         0.241142      0.847873       1.4344       0.895607      0.288555        10     
reco:pandoracalo:Calorimetry                         0.24332      0.535237      0.794818      0.539733       0.13922        10     
reco:pandoracalonosce:Calorimetry                   0.237133       0.51727       0.7692       0.519294      0.133309        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.213763      0.833399       1.34731      0.857885      0.287026        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.210368      0.841399       1.34144      0.828536       0.29694        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0290312     0.0436828     0.0553703     0.0429256    0.00737295       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0294765     0.0439693     0.0561691     0.0432865    0.00748878       10     
reco:emtrkmichelid:EmTrackMichelId                   16.8301       25.2096       37.2788       23.3439       5.61902        10     
[art]:TriggerResults:TriggerResultInserter         2.3417e-05    4.28995e-05   0.000104335   3.88025e-05   2.19025e-05      10     
end_path:out1:RootOutput                            5.503e-06    9.6727e-06    2.5187e-05    8.1205e-06     5.376e-06       10     
end_path:out1:RootOutput(write)                     0.906484       1.3284        1.90992       1.18774      0.352697        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5852.88 MB
  Peak resident set size usage (VmHWM): 3689.62 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11725940
-rw-r--r-- 1 dunegli fnalgrid     231198 Dec 17 20:47 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        200 Dec 17 20:47 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 17 20:47 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Dec 17 20:47 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     139264 Dec 17 20:47 mem.db
-rw-r--r-- 1 dunegli fnalgrid      28672 Dec 17 20:47 time.db
-rw-r--r-- 1 dunegli fnalgrid 6071369376 Dec 17 20:47 pdhd_prod_beam__267379_78_1_20251212T161846Z_gen_g4_IonScintPDExt.root_262726_21_1_20251215T212452Z_PDInt.root_268896_141_1_20251217T192607Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   31951500 Dec 17 20:46 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Dec 17 20:28 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 17 20:28 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 5891862428 Dec 17 20:28 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        121 Dec 17 19:26 all-input-dids.txt
justIN time: 2025-12-19 04:23:31 UTC       justIN version: 01.05.03