justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 269138.0@dunegpschedd01.fnal.gov

Jobsub ID269138.0@dunegpschedd01.fnal.gov
Workflow ID11264
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 22:38:16
SiteUS_BNL
EntryDUNE_US_BNL_spoolce01
Last heartbeat2025-12-19 00:43:34
From worker nodeHostnamespool0118.sdcc.bnl.gov
cpuinfoIntel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-12-18 22:38:55
Input filesusertests:pdhd_prod_beam__261612_46_1_20251212T155710Z_gen_g4_IonScintPDExt.root_262727_49_1_20251215T214514Z_PDInt.root
JobscriptExit code0
Real time2h (7385s)
CPU time1h (7157s = 96%)
Max RSS bytes7118839808 (6789 MiB)
Outputting started2025-12-19 00:42:01
Output filesusertests:pdhd_prod_beam__261612_46_1_20251212T155710Z_gen_g4_IonScintPDExt.root_262727_49_1_20251215T214514Z_PDInt.root_269138_0_1_20251218T223859Z_detsim_reco1.root
Finished2025-12-19 00:43:34
Saved logsjustin-logs:269138.0-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

races.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":30535 [0] "wiener0":34042 [34042] ] cmm:[ bad:0 lf_noisy:69 ] input 1: frame: ident=179 time=0 tick=512 with 62077 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":26107 [0] "wiener1":35970 [35970] ] cmm:[ bad:0 lf_noisy:3 ] input 2: frame: ident=179 time=0 tick=512 with 47069 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":19856 [0] "wiener2":27213 [27213] ] cmm:[ bad:0 ] input 3: frame: ident=179 time=0 tick=512 with 40761 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":17782 [0] "wiener3":22979 [22979] ] cmm:[ bad:0 ] output: frame: ident=179 time=0 tick=512 with 214484 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":30535 [0] "gauss1":26107 [0] "gauss2":19856 [0] "gauss3":17782 [0] "wiener0":34042 [34042] "wiener1":35970 [35970] "wiener2":27213 [27213] "wiener3":22979 [22979] ] cmm:[ bad:0 lf_noisy:72 ] 
Retagger: tagging trace set: wiener with 120204 traces, 120204 summary
Retagger: tagging trace set: gauss with 94280 traces, 0 summary
[19:40:59.818] D [  glue  ] sink frame: #179 @0 with 214484 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[19:40:59.818] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[19:40:59.818] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[19:40:59.818] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[19:40:59.818] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[19:40:59.818] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[19:40:59.818] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[19:40:59.818] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[19:40:59.818] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[19:40:59.818] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[19:40:59.818] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[19:40:59.818] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[19:40:59.818] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[19:40:59.818] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[19:40:59.818] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[19:40:59.818] D [  glue  ] frame sink sees EOS
[19:40:59.818] D [ pgraph ] <Pgrapher:> graph execution complete 
[19:40:59.818] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 23.17 sec
[19:40:59.818] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 14.7 sec
[19:40:59.818] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.99 sec
[19:40:59.818] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 12.5 sec
[19:40:59.818] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 5.29 sec
[19:40:59.818] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 5.13 sec
[19:40:59.818] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 4.1 sec
[19:40:59.818] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 3.87 sec
[19:40:59.818] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.05 sec
[19:40:59.818] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.03 sec
[19:40:59.818] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[19:40:59.818] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[19:40:59.818] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[19:40:59.818] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[19:40:59.818] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[19:40:59.818] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[19:40:59.818] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[19:40:59.818] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[19:40:59.818] I [ timer  ] Timer: Total node execution : 81.82999952323735 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 94280 traces tagged "gauss"
FrameSaver: q=3.00896e+06 n=1520622 tag=gauss
wclsFrameSaver: saving 120204 traces tagged "wiener"
FrameSaver: q=3.95966e+06 n=1463740 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 45508 XUs and 61699 XVs -> 1267 XUVs
C:0 T:4 5 XUs and 9 XVs -> 0 XUVs
C:0 T:5 47222 XUs and 46689 XVs -> 1916 XUVs
C:0 T:6 22211 XUs and 19846 XVs -> 472 XUVs
C:0 T:7 8 XUs and 8 XVs -> 0 XUVs
3655 XUVs total
1741 collection wire objects
3655 potential space points
Neighbour search...
71275 tests to find 43674 neighbours
Iterating with no regularization...
Begin: 3.04776e+07
0 2.86632e+07
1 2.8577e+07
2 2.85636e+07
Now with regularization...
Begin: 2.68717e+07
0 2.68672e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
18-Dec-2025 19:41:59 EST  Closed output file "pdhd_prod_beam__261612_46_1_20251212T155710Z_gen_g4_IonScintPDExt.root_262727_49_1_20251215T214514Z_PDInt.root_269138_0_1_20251218T223859Z_detsim_reco1.root"
18-Dec-2025 19:41:59 EST  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           116.862       130.414       146.886       127.361       10.1723        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.00142011    0.00602275     0.0131573    0.00574232    0.00322775       10     
reco:wclsdatahd:WireCellToolkit                      76.8758       84.412        100.372       81.9247       6.43297        10     
reco:wirefilter:WireFilter                          0.0279254     0.0358887     0.0458733     0.0354643    0.00584355       10     
reco:gaushit:GausHitFinder                           0.74465       1.17955       1.83592       1.09452      0.368533        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000244211   0.000414802   0.000593676   0.000404098   9.90484e-05      10     
reco:reco3d:SpacePointSolver                        0.426323      0.916243       1.35983      0.967298      0.307784        10     
reco:hitpdune:DisambigFromSpacePoints               0.362647      0.959186       1.79354      0.998047      0.441907        10     
reco:pandora:StandardPandora                         7.77529       14.7885       19.9342       16.3904       4.42819        10     
reco:pandoraWriter:StandardPandora                  0.226258      0.322954      0.444983      0.314082      0.0659949       10     
reco:pandoraTrack:LArPandoraTrackCreation            1.14614       1.73938       2.52764       1.58585      0.447061        10     
reco:pandoraShower:LArPandoraShowerCreation         0.613713       1.78822       3.3499        1.62374      0.718036        10     
reco:pandoracalo:Calorimetry                         0.54894      0.892912       1.1629       0.855992      0.202977        10     
reco:pandoracalonosce:Calorimetry                   0.525951      0.864576       1.16418      0.837998       0.19472        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.517509       1.69508       3.20025       1.57766      0.702848        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.513125       1.66533       3.1491        1.60862      0.700364        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.0652555     0.0963504     0.132581      0.0966364     0.0218206       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0678019     0.0967994     0.131126      0.0915896     0.0198359       10     
reco:emtrkmichelid:EmTrackMichelId                   10.0888       16.9046       21.6944       17.2519       3.02011        10     
[art]:TriggerResults:TriggerResultInserter         3.3384e-05    5.03612e-05   9.4358e-05    4.44795e-05   1.60589e-05      10     
end_path:out1:RootOutput                            5.787e-06    1.01384e-05   2.4572e-05     8.65e-06     4.97149e-06      10     
end_path:out1:RootOutput(write)                      1.33378       1.99278       2.52699       1.9486       0.332452        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6336.54 MB
  Peak resident set size usage (VmHWM): 3958.35 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12775984
-rw-r--r-- 1 fermipro fermilab     231561 Dec 18 19:42 jobscript.log
-rw-r--r-- 1 fermipro fermilab        200 Dec 18 19:42 justin-processed-pfns.txt
-rw-r--r-- 1 fermipro fermilab        519 Dec 18 19:41 reco_protodunehd_hist.root
-rw-r--r-- 1 fermipro fermilab       3574 Dec 18 19:41 Pandora_Geometry.xml
-rw-r--r-- 1 fermipro fermilab     139264 Dec 18 19:41 mem.db
-rw-r--r-- 1 fermipro fermilab      28672 Dec 18 19:41 time.db
-rw-r--r-- 1 fermipro fermilab 6621525284 Dec 18 19:41 pdhd_prod_beam__261612_46_1_20251212T155710Z_gen_g4_IonScintPDExt.root_262727_49_1_20251215T214514Z_PDInt.root_269138_0_1_20251218T223859Z_detsim_reco1.root
-rw-r--r-- 1 fermipro fermilab   36703830 Dec 18 19:41 Pandora_Events.pndr
-rw-r--r-- 1 fermipro fermilab          0 Dec 18 19:17 debugprod.log
-rw-r--r-- 1 fermipro fermilab        519 Dec 18 19:17 detsim_single_protodunehd_hist.root
-rw-r--r-- 1 fermipro fermilab 6423924744 Dec 18 19:17 temp_detsim.root
-rw-r--r-- 1 fermipro fermilab        121 Dec 18 17:38 all-input-dids.txt
justIN time: 2025-12-19 01:23:19 UTC       justIN version: 01.05.03