justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 243743.1@dunegpschedd01.fnal.gov

Jobsub ID243743.1@dunegpschedd01.fnal.gov
Workflow ID9567
Stage ID1
User namemwrobel@fnal.gov
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-11-05 22:05:52
SiteUS_UChicago
EntryEngage_US_MWT2_iut2_gk02_condce_mcore
Last heartbeat2025-11-06 00:00:13
From worker nodeHostnameiut2-c326.iu.edu
cpuinfoAMD EPYC 7402 24-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit86400 (24 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-11-05 22:13:14
Input filesusertests:pdhd_prod_beam__226470_100_1_20251008T212517Z_gen_g4_IonScintPDExt.root_227040_54_1_20251010T170046Z_PDInt.root
JobscriptExit code0
Real time1h (6308s)
CPU time1h (6118s = 96%)
Max RSS bytes8342532096 (7956 MiB)
Outputting started2025-11-05 23:58:22
Output filesusertests:pdhd_prod_beam__226470_100_1_20251008T212517Z_gen_g4_IonScintPDExt.root_227040_54_1_20251010T170046Z_PDInt.root_243743_1_1_20251105T221320Z_detsim_reco1.root
Finished2025-11-06 00:00:13
Saved logsjustin-logs:243743.1-dunegpschedd01.fnal.gov.logs.tgz
List job events     (HTCondor job logs unavailable)

Jobscript log (last 10,000 characters)

68] ] cmm:[ bad:0 lf_noisy:85 ] 
[18:57:46.570] D [  glue  ] <FrameFanin:sn_mag_nf> call=18 input 0: frame: ident=99 time=0 tick=512 with 60397 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":28329 [0] "wiener0":32068 [32068] ] cmm:[ bad:0 lf_noisy:85 ] input 1: frame: ident=99 time=0 tick=512 with 29248 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":13383 [0] "wiener1":15865 [15865] ] cmm:[ bad:0 lf_noisy:1 ] input 2: frame: ident=99 time=0 tick=512 with 41763 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":18330 [0] "wiener2":23433 [23433] ] cmm:[ bad:0 ] input 3: frame: ident=99 time=0 tick=512 with 32697 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":14200 [0] "wiener3":18497 [18497] ] cmm:[ bad:0 ] output: frame: ident=99 time=0 tick=512 with 164105 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":28329 [0] "gauss1":13383 [0] "gauss2":18330 [0] "gauss3":14200 [0] "wiener0":32068 [32068] "wiener1":15865 [15865] "wiener2":23433 [23433] "wiener3":18497 [18497] ] cmm:[ bad:0 lf_noisy:86 ] 
Retagger: tagging trace set: wiener with 89863 traces, 89863 summary
Retagger: tagging trace set: gauss with 74242 traces, 0 summary
[18:57:46.585] D [  glue  ] sink frame: #99 @0 with 164105 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[18:57:46.585] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[18:57:46.585] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[18:57:46.585] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[18:57:46.585] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[18:57:46.585] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[18:57:46.585] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[18:57:46.585] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[18:57:46.585] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[18:57:46.585] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[18:57:46.585] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[18:57:46.585] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[18:57:46.585] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[18:57:46.585] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[18:57:46.585] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[18:57:46.585] D [  glue  ] frame sink sees EOS
[18:57:46.585] D [ pgraph ] <Pgrapher:> graph execution complete 
[18:57:46.585] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 14.65 sec
[18:57:46.585] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.9 sec
[18:57:46.585] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.87 sec
[18:57:46.585] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.47 sec
[18:57:46.585] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.23 sec
[18:57:46.585] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.21 sec
[18:57:46.585] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.06 sec
[18:57:46.585] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.89 sec
[18:57:46.585] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.02 sec
[18:57:46.585] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[18:57:46.586] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:57:46.586] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:57:46.586] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:57:46.586] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[18:57:46.586] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[18:57:46.586] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[18:57:46.586] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[18:57:46.586] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[18:57:46.586] I [ timer  ] Timer: Total node execution : 55.29999949410558 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 74242 traces tagged "gauss"
FrameSaver: q=2.28193e+06 n=1188531 tag=gauss
wclsFrameSaver: saving 89863 traces tagged "wiener"
FrameSaver: q=2.86703e+06 n=1161062 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 8692 XUs and 6353 XVs -> 262 XUVs
C:0 T:3 9 XUs and 10 XVs -> 0 XUVs
C:0 T:5 23032 XUs and 24004 XVs -> 1215 XUVs
C:0 T:6 21732 XUs and 15656 XVs -> 1011 XUVs
C:0 T:7 90 XUs and 34 XVs -> 0 XUVs
2488 XUVs total
1033 collection wire objects
2488 potential space points
Neighbour search...
82580 tests to find 51580 neighbours
Iterating with no regularization...
Begin: 1.50016e+07
0 1.39086e+07
1 1.37854e+07
2 1.37633e+07
3 1.37585e+07
Now with regularization...
Begin: 1.27253e+07
0 1.27234e+07
05-Nov-2025 18:58:20 EST  Closed output file "pdhd_prod_beam__226470_100_1_20251008T212517Z_gen_g4_IonScintPDExt.root_227040_54_1_20251010T170046Z_PDInt.root_243743_1_1_20251105T221320Z_detsim_reco1.root"
05-Nov-2025 18:58:20 EST  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           72.9195       90.5286       129.628       88.183        14.9455        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.000908122   0.00370493    0.00554397    0.00435293    0.00146962       10     
reco:wclsdatahd:WireCellToolkit                      44.8097       50.6175       57.8261       51.5466       4.25261        10     
reco:wirefilter:WireFilter                          0.0125768     0.0184241     0.0277021     0.0178097    0.00393624       10     
reco:gaushit:GausHitFinder                          0.321179      0.637302       1.06752      0.618497      0.193146        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000145324   0.000295465   0.000472601   0.000253848   9.98229e-05      10     
reco:reco3d:SpacePointSolver                        0.194771      0.451129       1.16621      0.360246      0.264841        10     
reco:hitpdune:DisambigFromSpacePoints               0.169173      0.470553       1.44485      0.335861      0.346318        10     
reco:pandora:StandardPandora                         3.28138       8.9366        27.0243       6.41779       6.61021        10     
reco:pandoraWriter:StandardPandora                  0.139343      0.228233      0.503948       0.20879      0.0977982       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.379148      0.985669       3.08057      0.766229      0.746735        10     
reco:pandoraShower:LArPandoraShowerCreation         0.266632      0.947552       3.16042      0.708611      0.777378        10     
reco:pandoracalo:Calorimetry                        0.208186      0.460732       1.05024      0.423463      0.229998        10     
reco:pandoracalonosce:Calorimetry                   0.190115      0.449443       1.04188       0.40657      0.223306        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.240559      0.923445       3.13829      0.696037      0.767075        10     
reco:pandoraShowercalonosce:ShowerCalorimetry        0.24209      0.918904       2.83135      0.686506       0.69061        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0329002     0.0534073     0.0715766     0.0524589     0.0124542       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0327022     0.0528728     0.074679      0.0525326     0.0128593       10     
reco:emtrkmichelid:EmTrackMichelId                   14.9595       22.9494       36.1603       21.2447       5.83121        10     
[art]:TriggerResults:TriggerResultInserter         2.0138e-05    3.29332e-05   8.0452e-05    2.8974e-05    1.62332e-05      10     
end_path:out1:RootOutput                            7.033e-06    9.8624e-06    2.5328e-05    8.2105e-06    5.20868e-06      10     
end_path:out1:RootOutput(write)                     0.853499       1.36635       1.89967       1.25448      0.319842        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6249.13 MB
  Peak resident set size usage (VmHWM): 4064.14 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12452180
-rw-r--r--. 1 dune osgvo     230911 Nov  5 18:58 jobscript.log
-rw-r--r--. 1 dune osgvo        201 Nov  5 18:58 justin-processed-pfns.txt
-rw-r--r--. 1 dune osgvo        519 Nov  5 18:58 reco_protodunehd_hist.root
-rw-r--r--. 1 dune osgvo       3574 Nov  5 18:58 Pandora_Geometry.xml
-rw-r--r--. 1 dune osgvo     139264 Nov  5 18:58 mem.db
-rw-r--r--. 1 dune osgvo      28672 Nov  5 18:58 time.db
-rw-r--r--. 1 dune osgvo 6455324769 Nov  5 18:58 pdhd_prod_beam__226470_100_1_20251008T212517Z_gen_g4_IonScintPDExt.root_227040_54_1_20251010T170046Z_PDInt.root_243743_1_1_20251105T221320Z_detsim_reco1.root
-rw-r--r--. 1 dune osgvo   36693600 Nov  5 18:57 Pandora_Events.pndr
-rw-r--r--. 1 dune osgvo          0 Nov  5 18:41 debugprod.log
-rw-r--r--. 1 dune osgvo        519 Nov  5 18:41 detsim_single_protodunehd_hist.root
-rw-r--r--. 1 dune osgvo 6258563025 Nov  5 18:41 temp_detsim.root
-rw-r--r--. 1 dune osgvo        122 Nov  5 17:13 all-input-dids.txt
justIN time: 2026-02-11 12:16:28 UTC       justIN version: 01.06.00