justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 263074.1@dunegpschedd02.fnal.gov

Jobsub ID263074.1@dunegpschedd02.fnal.gov
Workflow ID11235
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-17 11:25:28
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce04_mcore_op_duneonly
Last heartbeat2025-12-17 12:53:51
From worker nodeHostnamedunegli-8092434-0-fnpc9041.fnal.gov
cpuinfoIntel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-17 11:26:28
Input filesusertests:pdhd_prod_beam__261613_28_1_20251212T161323Z_gen_g4_IonScintPDExt.root_262728_148_1_20251215T220738Z_PDInt.root
JobscriptExit code0
Real time1h (5171s)
CPU time1h (4990s = 96%)
Max RSS bytes6743957504 (6431 MiB)
Outputting started2025-12-17 12:52:40
Output filesusertests:pdhd_prod_beam__261613_28_1_20251212T161323Z_gen_g4_IonScintPDExt.root_262728_148_1_20251215T220738Z_PDInt.root_263074_1_1_20251217T112633Z_detsim_reco1.root
Finished2025-12-17 12:53:51
Saved logsjustin-logs:263074.1-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

ameFanin:sn_mag_nf> call=18 input 0: frame: ident=199 time=0 tick=512 with 53354 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":25506 [0] "wiener0":27848 [27848] ] cmm:[ bad:0 lf_noisy:78 ] input 1: frame: ident=199 time=0 tick=512 with 39947 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":17765 [0] "wiener1":22182 [22182] ] cmm:[ bad:0 ] input 2: frame: ident=199 time=0 tick=512 with 45966 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":19869 [0] "wiener2":26097 [26097] ] cmm:[ bad:0 ] input 3: frame: ident=199 time=0 tick=512 with 43064 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":19292 [0] "wiener3":23772 [23772] ] cmm:[ bad:0 ] output: frame: ident=199 time=0 tick=512 with 182331 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":25506 [0] "gauss1":17765 [0] "gauss2":19869 [0] "gauss3":19292 [0] "wiener0":27848 [27848] "wiener1":22182 [22182] "wiener2":26097 [26097] "wiener3":23772 [23772] ] cmm:[ bad:0 lf_noisy:78 ] 
Retagger: tagging trace set: wiener with 99899 traces, 99899 summary
Retagger: tagging trace set: gauss with 82432 traces, 0 summary
[12:51:37.597] D [  glue  ] sink frame: #199 @0 with 182331 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[12:51:37.597] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[12:51:37.597] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[12:51:37.597] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[12:51:37.597] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[12:51:37.597] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[12:51:37.597] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[12:51:37.597] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[12:51:37.597] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[12:51:37.597] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[12:51:37.597] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[12:51:37.597] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[12:51:37.597] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[12:51:37.597] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[12:51:37.597] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[12:51:37.597] D [  glue  ] frame sink sees EOS
[12:51:37.597] D [ pgraph ] <Pgrapher:> graph execution complete 
[12:51:37.597] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 14.14 sec
[12:51:37.597] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.15 sec
[12:51:37.597] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.14 sec
[12:51:37.597] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 9.69 sec
[12:51:37.597] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.74 sec
[12:51:37.597] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.67 sec
[12:51:37.597] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.66 sec
[12:51:37.597] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.66 sec
[12:51:37.597] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.03 sec
[12:51:37.597] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.02 sec
[12:51:37.597] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.02 sec
[12:51:37.597] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:51:37.597] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[12:51:37.597] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:51:37.597] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[12:51:37.597] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[12:51:37.597] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[12:51:37.597] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[12:51:37.597] I [ timer  ] Timer: Total node execution : 54.920000141486526 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 82432 traces tagged "gauss"
FrameSaver: q=2.60856e+06 n=1381717 tag=gauss
wclsFrameSaver: saving 99899 traces tagged "wiener"
FrameSaver: q=3.4966e+06 n=1347339 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 28166 XUs and 16025 XVs -> 500 XUVs
C:0 T:3 8 XUs and 8 XVs -> 0 XUVs
C:0 T:4 14 XUs and 28 XVs -> 7 XUVs
C:0 T:5 24143 XUs and 37191 XVs -> 2736 XUVs
C:0 T:6 23615 XUs and 21281 XVs -> 541 XUVs
3784 XUVs total
1579 collection wire objects
3784 potential space points
Neighbour search...
146634 tests to find 90272 neighbours
Iterating with no regularization...
Begin: 1.75105e+07
0 1.64948e+07
1 1.63749e+07
2 1.63502e+07
3 1.63413e+07
Now with regularization...
Begin: 1.49071e+07
0 1.49009e+07
17-Dec-2025 12:52:31 UTC  Closed output file "pdhd_prod_beam__261613_28_1_20251212T161323Z_gen_g4_IonScintPDExt.root_262728_148_1_20251215T220738Z_PDInt.root_263074_1_1_20251217T112633Z_detsim_reco1.root"
17-Dec-2025 12:52:31 UTC  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           80.2815       100.082       121.233       100.245       15.509         10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.00127741    0.00365805    0.00605735    0.00411661     0.0015149       10     
reco:wclsdatahd:WireCellToolkit                      53.0908       55.5319       58.7703       54.5359       2.14416        10     
reco:wirefilter:WireFilter                          0.0204501     0.029708      0.0458621     0.0282333    0.00759298       10     
reco:gaushit:GausHitFinder                          0.361366      0.668014       1.31919      0.577783      0.265399        10     
reco:nhitsfilter:NumberOfHitsFilter                0.00013206     0.0002626    0.000533455   0.000244448   0.000124824      10     
reco:reco3d:SpacePointSolver                        0.0951712     0.540764       1.14343      0.434964       0.34108        10     
reco:hitpdune:DisambigFromSpacePoints               0.165717      0.543058       1.00652      0.554133      0.304607        10     
reco:pandora:StandardPandora                         2.98873       9.32258       16.7983       9.88284       4.40556        10     
reco:pandoraWriter:StandardPandora                  0.115737      0.200857      0.306363      0.219806      0.0575776       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.479772       1.28577       2.11218       1.41848      0.562114        10     
reco:pandoraShower:LArPandoraShowerCreation         0.362991       1.13572        1.946        1.24823      0.553341        10     
reco:pandoracalo:Calorimetry                        0.276234      0.681186       1.13867       0.7565       0.291161        10     
reco:pandoracalonosce:Calorimetry                   0.265999      0.651051       1.09345       0.74807       0.27522        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.331618       1.13117       2.16465       1.20479      0.602848        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.318686       1.09046       1.96633       1.11915       0.57027        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0283057     0.0552619     0.0953301     0.0546993     0.0218002       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0289719     0.0550068     0.0969005     0.054518      0.021015        10     
reco:emtrkmichelid:EmTrackMichelId                   17.196        25.5532       33.9562       24.9852       6.2823         10     
[art]:TriggerResults:TriggerResultInserter         2.4381e-05    4.21283e-05    9.904e-05    3.81255e-05   1.99336e-05      10     
end_path:out1:RootOutput                            8.694e-06    1.27943e-05    3.343e-05    1.11025e-05   6.94983e-06      10     
end_path:out1:RootOutput(write)                     0.839897       1.55501       2.52218       1.5673       0.600971        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5876.47 MB
  Peak resident set size usage (VmHWM): 3714.4 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 11589336
-rw-r--r-- 1 dunegli fnalgrid     230895 Dec 17 12:52 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        201 Dec 17 12:52 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 17 12:52 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Dec 17 12:52 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     139264 Dec 17 12:52 mem.db
-rw-r--r-- 1 dunegli fnalgrid      28672 Dec 17 12:52 time.db
-rw-r--r-- 1 dunegli fnalgrid 6001593629 Dec 17 12:52 pdhd_prod_beam__261613_28_1_20251212T161323Z_gen_g4_IonScintPDExt.root_262728_148_1_20251215T220738Z_PDInt.root_263074_1_1_20251217T112633Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   33455310 Dec 17 12:51 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Dec 17 12:33 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 17 12:33 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 5820388753 Dec 17 12:33 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        122 Dec 17 11:26 all-input-dids.txt
justIN time: 2025-12-18 20:22:06 UTC       justIN version: 01.05.03