justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 269119.10@dunegpschedd01.fnal.gov

Jobsub ID269119.10@dunegpschedd01.fnal.gov
Workflow ID11264
Stage ID1
User nameavizcaya@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes9437184000 (9000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-12-18 19:29:50
SiteUS_UCSD
EntryCMSHTPC_T2_US_UCSD_gw7
Last heartbeat2025-12-18 21:05:14
From worker nodeHostnamesdsc-63.t2.ucsd.edu
cpuinfoIntel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job stateoutputting
Started2025-12-18 19:30:47
Input filesusertests:pdhd_prod_beam__261612_94_1_20251212T160259Z_gen_g4_IonScintPDExt.root_262727_19_1_20251215T214314Z_PDInt.root
JobscriptExit code0
Real time1h (5666s)
CPU time1h (5234s = 92%)
Max RSS bytes7441399808 (7096 MiB)
Outputting started2025-12-18 21:05:14
Output files
Finished 
List job events     Live HTCondor job logs

Jobscript log (last 10,000 characters)

rame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss0":28781 [0] "wiener0":33036 [33036] ] cmm:[ bad:0 lf_noisy:286 ] input 1: frame: ident=179 time=0 tick=512 with 47010 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss1":19205 [0] "wiener1":27805 [27805] ] cmm:[ bad:0 ] input 2: frame: ident=179 time=0 tick=512 with 31491 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss2":14167 [0] "wiener2":17324 [17324] ] cmm:[ bad:0 ] input 3: frame: ident=179 time=0 tick=512 with 56009 traces.  frame tags:[ "sigproc" ] 2 tagged trace sets:[ "gauss3":24518 [0] "wiener3":31491 [31491] ] cmm:[ bad:0 ] output: frame: ident=179 time=0 tick=512 with 196327 traces.  frame tags:[ "framefanin" ] 8 tagged trace sets:[ "gauss0":28781 [0] "gauss1":19205 [0] "gauss2":14167 [0] "gauss3":24518 [0] "wiener0":33036 [33036] "wiener1":27805 [27805] "wiener2":17324 [17324] "wiener3":31491 [31491] ] cmm:[ bad:0 lf_noisy:286 ] 
Retagger: tagging trace set: wiener with 109656 traces, 109656 summary
Retagger: tagging trace set: gauss with 86671 traces, 0 summary
[13:04:18.795] D [  glue  ] sink frame: #179 @0 with 196327 traces, frame tags:[retagger], trace tags:[gauss, wiener]
[13:04:18.795] D [  glue  ] <FrameFanout:sn_mag_nf> call=19: see EOS 
[13:04:18.795] D [  glue  ] <ChannelSelector:chsel3> see EOS at call=9 
[13:04:18.795] D [sigproc ] <OmnibusNoiseFilter:nf3> EOS at call=19 
[13:04:18.795] D [sigproc ] <OmnibusSigProc:apa3sigproc3> EOS at call=19 anode=3 
[13:04:18.795] D [  glue  ] <ChannelSelector:chsel2> see EOS at call=9 
[13:04:18.795] D [sigproc ] <OmnibusNoiseFilter:nf2> EOS at call=19 
[13:04:18.795] D [sigproc ] <OmnibusSigProc:apa2sigproc2> EOS at call=19 anode=2 
[13:04:18.795] D [  glue  ] <ChannelSelector:chsel1> see EOS at call=9 
[13:04:18.795] D [sigproc ] <OmnibusNoiseFilter:nf1> EOS at call=19 
[13:04:18.795] D [sigproc ] <OmnibusSigProc:apa1sigproc1> EOS at call=19 anode=1 
[13:04:18.795] D [  glue  ] <ChannelSelector:chsel0> see EOS at call=9 
[13:04:18.796] D [sigproc ] <OmnibusNoiseFilter:nf0> EOS at call=19 
[13:04:18.796] D [sigproc ] <OmnibusSigProc:apa0sigproc0> EOS at call=19 anode=0 
[13:04:18.796] D [  glue  ] <FrameFanin:sn_mag_nf> EOS at call=19 with 4 
[13:04:18.796] D [  glue  ] frame sink sees EOS
[13:04:18.796] D [ pgraph ] <Pgrapher:> graph execution complete 
[13:04:18.796] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 14.75 sec
[13:04:18.796] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.81 sec
[13:04:18.796] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.56 sec
[13:04:18.796] I [ timer  ] Timer: WireCell::SigProc::OmnibusSigProc : 10.23 sec
[13:04:18.796] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.82 sec
[13:04:18.796] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.81 sec
[13:04:18.796] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.74 sec
[13:04:18.796] I [ timer  ] Timer: WireCell::SigProc::OmnibusNoiseFilter : 2.7 sec
[13:04:18.796] I [ timer  ] Timer: WireCell::Gen::FrameFanin : 0.05 sec
[13:04:18.796] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0.01 sec
[13:04:18.796] I [ timer  ] Timer: WireCell::Gen::Retagger : 0.01 sec
[13:04:18.796] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:04:18.796] I [ timer  ] Timer: WireCell::Gen::FrameFanout : 0 sec
[13:04:18.796] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:04:18.796] I [ timer  ] Timer: WireCell::SigProc::ChannelSelector : 0 sec
[13:04:18.796] I [ timer  ] Timer: wcls::RawFrameSource : 0 sec
[13:04:18.796] I [ timer  ] Timer: wcls::FrameSaver : 0 sec
[13:04:18.796] I [ timer  ] Timer: WireCell::Gen::DumpFrames : 0 sec
[13:04:18.796] I [ timer  ] Timer: Total node execution : 57.49000031501055 sec
wclsFrameSaver saving cooked to 6000 ticks
wclsFrameSaver: saving 86671 traces tagged "gauss"
FrameSaver: q=3.02629e+06 n=1458299 tag=gauss
wclsFrameSaver: saving 109656 traces tagged "wiener"
FrameSaver: q=3.81708e+06 n=1415302 tag=wiener
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:2 46868 XUs and 73568 XVs -> 7315 XUVs
C:0 T:3 24 XUs and 18 XVs -> 2 XUVs
C:0 T:4 3 XUs and 85 XVs -> 1 XUVs
C:0 T:5 13171 XUs and 14332 XVs -> 407 XUVs
C:0 T:6 53539 XUs and 48761 XVs -> 1090 XUVs
C:0 T:7 13 XUs and 10 XVs -> 0 XUVs
8815 XUVs total
1437 collection wire objects
8815 potential space points
Neighbour search...
1274393 tests to find 703794 neighbours
Iterating with no regularization...
Begin: 2.14502e+07
0 1.86726e+07
1 1.84265e+07
2 1.83719e+07
3 1.83529e+07
4 1.83434e+07
Now with regularization...
Begin: 1.70403e+07
0 1.70289e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
18-Dec-2025 13:05:13 PST  Closed output file "pdhd_prod_beam__261612_94_1_20251212T160259Z_gen_g4_IonScintPDExt.root_262727_19_1_20251215T214314Z_PDInt.root_269119_10_1_20251218T193053Z_detsim_reco1.root"
18-Dec-2025 13:05:13 PST  Closed input file "temp_detsim.root"

=====================================================================================================================================
TimeTracker printout (sec)                             Min           Avg           Max         Median          RMS         nEvts   
=====================================================================================================================================
Full event                                           85.6572       105.669       133.382       107.517       14.1249        10     
-------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                             0.00130764    0.00444814     0.0129776    0.00391178    0.00317873       10     
reco:wclsdatahd:WireCellToolkit                      56.0593       60.3724       64.223        61.0723       2.3871         10     
reco:wirefilter:WireFilter                          0.0232024     0.0387922     0.0668173     0.0399735     0.0109789       10     
reco:gaushit:GausHitFinder                          0.397363      0.686058       1.00665      0.676192      0.183201        10     
reco:nhitsfilter:NumberOfHitsFilter                0.000156741   0.000324015   0.000604318   0.000299567   0.000133092      10     
reco:reco3d:SpacePointSolver                        0.297944      0.531164       1.12468      0.480733      0.224813        10     
reco:hitpdune:DisambigFromSpacePoints               0.225106      0.539678      0.952862      0.467497      0.225153        10     
reco:pandora:StandardPandora                         4.49563       10.1534       17.0903       9.93777       3.9809         10     
reco:pandoraWriter:StandardPandora                  0.135211      0.220795      0.265472      0.230114      0.0387908       10     
reco:pandoraTrack:LArPandoraTrackCreation           0.685443       1.21416       1.95921       1.13994      0.391883        10     
reco:pandoraShower:LArPandoraShowerCreation         0.405857       1.41672       2.62355       1.38548      0.707519        10     
reco:pandoracalo:Calorimetry                        0.355218      0.704839       1.16771      0.656394      0.224662        10     
reco:pandoracalonosce:Calorimetry                   0.333719      0.663204      0.978945      0.635159      0.197753        10     
reco:pandoraShowercalo:ShowerCalorimetry            0.300197       1.29443       2.23756       1.39272      0.605862        10     
reco:pandoraShowercalonosce:ShowerCalorimetry       0.300612       1.29211       2.49027       1.32491      0.641688        10     
reco:pandoraGnocchiCalo:GnocchiCalorimetry          0.029197      0.0552547     0.0770277     0.0571278     0.0141619       10     
reco:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0303481     0.0538655     0.0668685     0.0572597     0.0116756       10     
reco:emtrkmichelid:EmTrackMichelId                   16.208        25.0391       35.9895       25.7302       6.39777        10     
[art]:TriggerResults:TriggerResultInserter         2.4601e-05    4.17459e-05   0.000124653   3.4306e-05    2.79143e-05      10     
end_path:out1:RootOutput                            4.345e-06    8.9315e-06    3.1378e-05    6.2715e-06    7.70729e-06      10     
end_path:out1:RootOutput(write)                     0.778907       1.33238       2.15076       1.2793       0.443162        10     
=====================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6082.42 MB
  Peak resident set size usage (VmHWM): 3893.12 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Reco returns 0
.:
total 12187736
-rw-r--r--. 1 cuser cuser     231185 Dec 18 13:05 jobscript.log
-rw-r--r--. 1 cuser cuser        194 Dec 18 13:05 justin-processed-pfns.txt
-rw-r--r--. 1 cuser cuser        519 Dec 18 13:05 reco_protodunehd_hist.root
-rw-r--r--. 1 cuser cuser       3574 Dec 18 13:05 Pandora_Geometry.xml
-rw-r--r--. 1 cuser cuser     139264 Dec 18 13:05 mem.db
-rw-r--r--. 1 cuser cuser      28672 Dec 18 13:05 time.db
-rw-r--r--. 1 cuser cuser 6319250373 Dec 18 13:05 pdhd_prod_beam__261612_94_1_20251212T160259Z_gen_g4_IonScintPDExt.root_262727_19_1_20251215T214314Z_PDInt.root_269119_10_1_20251218T193053Z_detsim_reco1.root
-rw-r--r--. 1 cuser cuser   36483390 Dec 18 13:04 Pandora_Events.pndr
-rw-r--r--. 1 cuser cuser          0 Dec 18 12:45 debugprod.log
-rw-r--r--. 1 cuser cuser        519 Dec 18 12:44 detsim_single_protodunehd_hist.root
-rw-r--r--. 1 cuser cuser 6124049927 Dec 18 12:44 temp_detsim.root
-rw-r--r--. 1 cuser cuser        121 Dec 18 11:30 all-input-dids.txt
justIN time: 2025-12-18 21:07:56 UTC       justIN version: 01.05.03