justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 268896.160@dunegpschedd01.fnal.gov

Jobsub ID268896.160@dunegpschedd01.fnal.gov
Workflow ID11239
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-17 17:53:51
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-12-17 20:58:18
From worker nodeHostnamedunegli-7943973-0-fnpc9025.fnal.gov
cpuinfoIntel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Started2025-12-17 19:38:31
Input filesusertests:pdhd_prod_beam__267379_21_1_20251212T160808Z_gen_g4_IonScintPDExt.root_262726_49_1_20251215T212408Z_PDInt.root
JobscriptExit code0
Real time1h (4773s)
CPU time1h (4626s = 96%)
Max RSS bytes6863183872 (6545 MiB)
Outputting started2025-12-17 20:58:05
Output files
Finished2025-12-17 20:58:18
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

sigproc" ] 2 tagged trace sets:[ "gauss0":25465 [0] "wiener0":29059 [29059] ] cmm:[ bad:0 lf_noisy:173 ] input 1: frame: ident=189 time=0 tick=512 with 41890 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":17955 [0] "wiener1":23935 [23935] ] cmm:[ bad:0 ] input 2: frame: ident=189 time=0 tick=512 with 34270 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":14325 [0] "wiener2":19945 [19945] ] cmm:[ bad:0 ] input 3: frame: ident=189 time=0 tick=512 with 50352 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":21245 [0] "wiener3":29107 [29107] ] cmm:[ bad:0 lf_noisy:1 ] output: frame: ident=189 time=0 tick=512 with 181036 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":25465 [0] "gauss1":17955 [0] "gauss2":14325 [0] "gauss3":21245 [0] "wiener0":29059 [29059] "wiener1":23935 [23935] "wiener2":19945 [19945] "wiener3":29107 [29107] ] cmm:[ bad:0 lf_noisy:174 ] 
Retagger: tagging trace set: wiener with 102046 traces, 102046 summary
Retagger: tagging trace set: gauss with 78990 traces, 0 summary
[20:57:12.722] D [  glue  ] sink frame: #189 @0 with 181036 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[20:57:12.722] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[20:57:12.722] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[20:57:12.722] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[20:57:12.722] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[20:57:12.722] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[20:57:12.722] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[20:57:12.722] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[20:57:12.722] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[20:57:12.722] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[20:57:12.722] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[20:57:12.722] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[20:57:12.722] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[20:57:12.722] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[20:57:12.722] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[20:57:12.722] D [  glue  ] frame sink sees EOS
[20:57:12.722] D [ pgraph ] <Pgrapher:> graph execution complete 
[20:57:12.722] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 15.76 sec
[20:57:12.722] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.57 sec
[20:57:12.722] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.44 sec
[20:57:12.722] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.28 sec
[20:57:12.722] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.75 sec
[20:57:12.722] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.73 sec
[20:57:12.722] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.69 sec
[20:57:12.722] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.67 sec
[20:57:12.722] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.04 sec
[20:57:12.722] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.02 sec
[20:57:12.722] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[20:57:12.722] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:57:12.722] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:57:12.722] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[20:57:12.722] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[20:57:12.723] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[20:57:12.723] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[20:57:12.723] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[20:57:12.723] I [ timer  ] Timer: Total node execution : 57.959999388083816 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 78990 traces tagged "gauss"
FrameSaver: q=2.61642e+06 n=1227230 tag=gauss
wclsFrameSaver: saving 102046 traces tagged "wiener"
FrameSaver: q=3.39483e+06 n=1187520 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 26378 XUs and 30207 XVs -> 524 XUVs
C:0 T:3 13 XUs and 14 XVs -> 0 XUVs
C:0 T:4 23 XUs and 38 XVs -> 1 XUVs
C:0 T:5 18880 XUs and 24103 XVs -> 3692 XUVs
C:0 T:6 38355 XUs and 33686 XVs -> 1044 XUVs
5261 XUVs total
1465 collection wire objects
5261 potential space points
Neighbour search...
337299 tests to find 209690 neighbours
Iterating with no regularization...
Begin: 1.94623e+07
0 1.82485e+07
1 1.81429e+07
2 1.81204e+07
3 1.81119e+07
Now with regularization...
Begin: 1.67965e+07
0 1.67894e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
17-Dec-2025 20:58:00 UTC  Closed output file "pdhd_prod_beam__267379_21_1_20251212T160808Z_gen_g4_IonScintPDExt.root_262726_49_1_20251215T212408Z_PDInt.root_268896_160_1_20251217T193834Z_detsim_reco1.root"
17-Dec-2025 20:58:00 UTC  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           82.9283       97.1252       117.273       94.0923       10.6639        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.000809286   0.00688729     0.0248364    0.00412082    0.00751017       10     
reco:wclsdatahd:WireCellToolkit                      52.4699       55.827        60.0495       54.6275       2.72458        10     
reco:wirefilter:WireFilter                          0.0197767     0.0311127     0.0458391     0.029864     0.00748009       10     
reco:gaushit:GausHitFinder                          0.359181      0.590151      0.869695      0.593565      0.129327        10     
reco:nhitsfilter:NumberOfHitsFilter                0.00014207    0.000256328   0.000433105   0.000241161   9.15968e-05      10     
reco:reco3d:SpacePointSolver                        0.158212      0.574603       1.15978      0.550342      0.251883        10     
reco:hitpdune:DisambigFromSpacePoints               0.169891      0.529773      0.877455      0.550743      0.196228        10     
reco:pandora:StandardPandora                         4.36432       8.29708       12.1338       9.49385       2.4863         10     
reco:pandoraWriter:StandardPandora                  0.139879      0.262228      0.521166      0.221571      0.106077        10     
reco:pandoraTrack:LArPandoraTrackCreation           0.514266       1.05482       1.59339       1.09655      0.312497        10     
reco:pandoraShower:LArPandoraShowerCreation         0.312412      0.932706       1.95547      0.881519       0.41355        10     
reco:pandoracalo:Calorimetry                        0.300917       0.54145      0.748962      0.576346      0.139406        10     
reco:pandoracalonosce:Calorimetry                    0.28253      0.528236      0.721594      0.551912      0.142277        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.310679       0.86413       1.60141      0.807175      0.345765        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.299303      0.849455       1.56956      0.807123      0.336906        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0317994     0.0446991     0.0520825     0.0459417    0.00628028       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0325814     0.0446734     0.0535607     0.0459383    0.00631308       10     
reco:emtrkmichelid:EmTrackMichelId                   16.7705       24.8308       40.8266       23.9228       6.21899        10     
[art]:TriggerResults:TriggerResultInserter         2.2949e-05    4.08763e-05   0.000110691   3.5843e-05    2.41605e-05      10     
end_path:out1:RootOutput                            8.132e-06    1.22099e-05   3.4404e-05    1.05585e-05   7.48694e-06      10     
end_path:out1:RootOutput(write)                     0.867194       1.26513       1.88964       1.20689      0.283996        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5892.42 MB
  Peak resident set size usage (VmHWM): 3731.12 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11566836
-rw-r--r-- 1 dunegli fnalgrid     231483 Dec 17 20:58 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        200 Dec 17 20:58 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 17 20:58 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Dec 17 20:58 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     139264 Dec 17 20:58 mem.db
-rw-r--r-- 1 dunegli fnalgrid      28672 Dec 17 20:58 time.db
-rw-r--r-- 1 dunegli fnalgrid 5989524597 Dec 17 20:58 pdhd_prod_beam__267379_21_1_20251212T160808Z_gen_g4_IonScintPDExt.root_262726_49_1_20251215T212408Z_PDInt.root_268896_160_1_20251217T193834Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   33115190 Dec 17 20:57 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Dec 17 20:39 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 17 20:39 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 5809772280 Dec 17 20:39 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        121 Dec 17 19:38 all-input-dids.txt
justIN time: 2025-12-19 04:23:36 UTC       justIN version: 01.05.03