justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 268970.0@dunegpschedd01.fnal.gov

Jobsub ID268970.0@dunegpschedd01.fnal.gov
Workflow ID11239
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 01:20:12
SiteUS_FNAL-T1
EntryCMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat2025-12-18 03:22:59
From worker nodeHostnamedunegli-49199-0-cmswn2413.fnal.gov
cpuinfoAMD EPYC 7551 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 01:21:21
Input filesusertests:pdhd_prod_beam__267379_39_1_20251212T160905Z_gen_g4_IonScintPDExt.root_262726_73_1_20251215T212426Z_PDInt.root
JobscriptExit code0
Real time2h (7224s)
CPU time1h (5769s = 79%)
Max RSS bytes7461302272 (7115 MiB)
Outputting started2025-12-18 03:21:46
Output filesusertests:pdhd_prod_beam__267379_39_1_20251212T160905Z_gen_g4_IonScintPDExt.root_262726_73_1_20251215T212426Z_PDInt.root_268970_0_1_20251218T012126Z_detsim_reco1.root
Finished2025-12-18 03:22:59
Saved logsjustin-logs:268970.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

mag_nf> call=18 input 0: frame: ident=189 time=0 tick=512 with 43815 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":21014 [0] "wiener0":22801 [22801] ] cmm:[ bad:0 lf_noisy:273 ] input 1: frame: ident=189 time=0 tick=512 with 36496 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":16186 [0] "wiener1":20310 [20310] ] cmm:[ bad:0 ] input 2: frame: ident=189 time=0 tick=512 with 44000 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":18960 [0] "wiener2":25040 [25040] ] cmm:[ bad:0 ] input 3: frame: ident=189 time=0 tick=512 with 30879 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":13841 [0] "wiener3":17038 [17038] ] cmm:[ bad:0 lf_noisy:2 ] output: frame: ident=189 time=0 tick=512 with 155190 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":21014 [0] "gauss1":16186 [0] "gauss2":18960 [0] "gauss3":13841 [0] "wiener0":22801 [22801] "wiener1":20310 [20310] "wiener2":25040 [25040] "wiener3":17038 [17038] ] cmm:[ bad:0 lf_noisy:275 ] 
Retagger: tagging trace set: wiener with 85189 traces, 85189 summary
Retagger: tagging trace set: gauss with 70001 traces, 0 summary
[03:21:09.368] D [  glue  ] sink frame: #189 @0 with 155190 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[03:21:09.368] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[03:21:09.368] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[03:21:09.368] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[03:21:09.368] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[03:21:09.368] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[03:21:09.368] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[03:21:09.368] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[03:21:09.368] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[03:21:09.368] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[03:21:09.368] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[03:21:09.368] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[03:21:09.368] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[03:21:09.368] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[03:21:09.368] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[03:21:09.368] D [  glue  ] frame sink sees EOS
[03:21:09.368] D [ pgraph ] <Pgrapher:> graph execution complete 
[03:21:09.368] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 11.82 sec
[03:21:09.368] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.42 sec
[03:21:09.368] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.32 sec
[03:21:09.368] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 8.22 sec
[03:21:09.368] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.83 sec
[03:21:09.368] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.81 sec
[03:21:09.368] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.76 sec
[03:21:09.368] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.76 sec
[03:21:09.368] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0.01 sec
[03:21:09.368] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[03:21:09.368] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[03:21:09.368] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.01 sec
[03:21:09.368] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[03:21:09.368] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[03:21:09.368] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[03:21:09.368] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[03:21:09.368] I [ timer  ] Timer: WireCell::Gen::Retagger : 0 sec
[03:21:09.368] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[03:21:09.368] I [ timer  ] Timer: Total node execution : 47.97999957948923 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 70001 traces tagged "gauss"
FrameSaver: q=1.95654e+06 n=1080415 tag=gauss
wclsFrameSaver: saving 85189 traces tagged "wiener"
FrameSaver: q=2.39672e+06 n=1038796 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 18424 XUs and 17504 XVs -> 639 XUVs
C:0 T:4 6 XUs and 4 XVs -> 0 XUVs
C:0 T:5 23484 XUs and 21024 XVs -> 1709 XUVs
C:0 T:6 10436 XUs and 11348 XVs -> 342 XUVs
C:0 T:7 11 XUs and 10 XVs -> 2 XUVs
2692 XUVs total
1314 collection wire objects
2692 potential space points
Neighbour search...
94886 tests to find 58736 neighbours
Iterating with no regularization...
Begin: 1.63451e+07
0 1.52745e+07
1 1.51504e+07
2 1.51337e+07
3 1.51286e+07
Now with regularization...
Begin: 1.39868e+07
0 1.39829e+07
18-Dec-2025 03:21:44 UTC  Closed output file "pdhd_prod_beam__267379_39_1_20251212T160905Z_gen_g4_IonScintPDExt.root_262726_73_1_20251215T212426Z_PDInt.root_268970_0_1_20251218T012126Z_detsim_reco1.root"
18-Dec-2025 03:21:44 UTC  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           76.7596       93.2788       122.434       90.582        12.9554        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.000756363    0.0033188    0.00487123    0.00329408    0.00137319       10     
reco:wclsdatahd:WireCellToolkit                      44.1142       49.3242       51.5469       50.5092       2.57328        10     
reco:wirefilter:WireFilter                          0.0135634     0.0173249     0.0251384     0.0169636    0.00319668       10     
reco:gaushit:GausHitFinder                          0.389835       0.56327      0.815247      0.521725      0.140579        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000155831   0.000291676   0.000581102   0.000261522   0.000115391      10     
reco:reco3d:SpacePointSolver                        0.187812      0.425022      0.804445      0.421544      0.200109        10     
reco:hitpdune:DisambigFromSpacePoints               0.134846       0.35745      0.719362      0.315839      0.170465        10     
reco:pandora:StandardPandora                         4.58589       9.07238       19.028        8.83257       4.31314        10     
reco:pandoraWriter:StandardPandora                   0.11073      0.176794      0.252012      0.174374      0.0446342       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.531366       0.98415       1.47588      0.896121      0.347869        10     
reco:pandoraShower:LArPandoraShowerCreation         0.471243      0.929479       2.2113       0.734967      0.493434        10     
reco:pandoracalo:Calorimetry                        0.256897      0.477326      0.734878      0.448733       0.16181        10     
reco:pandoracalonosce:Calorimetry                   0.276463      0.464503       0.71003      0.433958      0.151672        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.433818      0.842336       1.86005       0.68386      0.398367        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.426548      0.905987       2.07185      0.774422      0.459783        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0289113     0.0446845     0.0631165     0.0432026     0.0111666       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0279759     0.0437385     0.0602884     0.0422246     0.0106396       10     
reco:emtrkmichelid:EmTrackMichelId                   17.3128       27.4385       38.7077       26.2025       6.8016         10     
[art]:TriggerResults:TriggerResultInserter          2.217e-05    2.8989e-05     7.124e-05     2.399e-05    1.41883e-05      10     
end_path:out1:RootOutput                            9.27e-06     1.1645e-05     2.658e-05    1.0105e-05    5.00226e-06      10     
end_path:out1:RootOutput(write)                     0.829241       1.16534       1.58139       1.10927       0.24388        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6434.29 MB
  Peak resident set size usage (VmHWM): 4210.79 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12722956
-rw-r--r-- 1 dunegli fnalgrid     231398 Dec 18 03:21 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid        194 Dec 18 03:21 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 18 03:21 reco_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid       3574 Dec 18 03:21 Pandora_Geometry.xml
-rw-r--r-- 1 dunegli fnalgrid     139264 Dec 18 03:21 mem.db
-rw-r--r-- 1 dunegli fnalgrid      28672 Dec 18 03:21 time.db
-rw-r--r-- 1 dunegli fnalgrid 6592875156 Dec 18 03:21 pdhd_prod_beam__267379_39_1_20251212T160905Z_gen_g4_IonScintPDExt.root_262726_73_1_20251215T212426Z_PDInt.root_268970_0_1_20251218T012126Z_detsim_reco1.root
-rw-r--r-- 1 dunegli fnalgrid   35764320 Dec 18 03:21 Pandora_Events.pndr
-rw-r--r-- 1 dunegli fnalgrid          0 Dec 18 03:04 debugprod.log
-rw-r--r-- 1 dunegli fnalgrid        519 Dec 18 03:04 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 dunegli fnalgrid 6399221738 Dec 18 03:04 temp_detsim.root
-rw-r--r-- 1 dunegli fnalgrid        121 Dec 18 01:21 all-input-dids.txt
justIN time: 2025-12-18 20:22:37 UTC       justIN version: 01.05.03