justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 14363.0@dunegpschedd02.fnal.gov

Jobsub ID14363.0@dunegpschedd02.fnal.gov
Workflow ID330
Stage ID1
User namemsiden@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8387559424 (7999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-08-04 17:26:06
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc03
Last heartbeat2025-08-04 22:08:35
From worker nodeHostnamewn-la-21.gina.surf.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-08-04 17:27:35
Input fileshd-protodune:np04hd_raw_run027087_0020_dataflow2_datawriter_0_20240614T145558.hdf5
JobscriptExit code0
Real time4h (16755s)
CPU time4h (16679s = 99%)
Max RSS bytes6067453952 (5786 MiB)
Outputting started2025-08-04 22:06:52
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/msiden/channel_map_test/fnal/00330/1/001/pdhd_processing_1_14363_0_1754328471_stage1_stage2.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/msiden/channel_map_test/fnal/00330/1/001/pdhd_processing_1_14363_0_1754328471_stage1_stage2_tfile.root
Finished2025-08-04 22:08:35
Saved logsjustin-logs:14363.0-dunegpschedd02.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

"wiener"
FrameSaver: q=2.12972e+06 n=994041 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 65049 traces, 65049 summary
Retagger: tagging trace set: gauss with 58255 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 58255 traces tagged "gauss"
FrameSaver: q=1.60496e+06 n=807084 tag=gauss
wclsFrameSaver: saving 65049 traces tagged "wiener"
FrameSaver: q=1.50462e+06 n=814945 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 70602 traces, 70602 summary
Retagger: tagging trace set: gauss with 63193 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 63193 traces tagged "gauss"
FrameSaver: q=1.56727e+06 n=807956 tag=gauss
wclsFrameSaver: saving 70602 traces tagged "wiener"
FrameSaver: q=1.79622e+06 n=865010 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 76654 traces, 76654 summary
Retagger: tagging trace set: gauss with 66926 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 66926 traces tagged "gauss"
FrameSaver: q=1.6759e+06 n=834574 tag=gauss
wclsFrameSaver: saving 76654 traces tagged "wiener"
FrameSaver: q=2.00557e+06 n=919667 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 64060 traces, 64060 summary
Retagger: tagging trace set: gauss with 55717 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 55717 traces tagged "gauss"
FrameSaver: q=1.16991e+06 n=690022 tag=gauss
wclsFrameSaver: saving 64060 traces tagged "wiener"
FrameSaver: q=1.24614e+06 n=696577 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 71387 traces, 71387 summary
Retagger: tagging trace set: gauss with 60040 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 60040 traces tagged "gauss"
FrameSaver: q=1.58799e+06 n=844863 tag=gauss
wclsFrameSaver: saving 71387 traces tagged "wiener"
FrameSaver: q=1.72693e+06 n=838202 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 80325 traces, 80325 summary
Retagger: tagging trace set: gauss with 66979 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 66979 traces tagged "gauss"
FrameSaver: q=1.74408e+06 n=840088 tag=gauss
wclsFrameSaver: saving 80325 traces tagged "wiener"
FrameSaver: q=1.93054e+06 n=875524 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 80631 traces, 80631 summary
Retagger: tagging trace set: gauss with 70837 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 70837 traces tagged "gauss"
FrameSaver: q=1.93763e+06 n=983212 tag=gauss
wclsFrameSaver: saving 80631 traces tagged "wiener"
FrameSaver: q=2.01997e+06 n=1009163 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 76714 traces, 76714 summary
Retagger: tagging trace set: gauss with 67911 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 67911 traces tagged "gauss"
FrameSaver: q=1.77158e+06 n=916695 tag=gauss
wclsFrameSaver: saving 76714 traces tagged "wiener"
FrameSaver: q=1.91293e+06 n=967027 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 69079 traces, 69079 summary
Retagger: tagging trace set: gauss with 60355 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 60355 traces tagged "gauss"
FrameSaver: q=1.69027e+06 n=792290 tag=gauss
wclsFrameSaver: saving 69079 traces tagged "wiener"
FrameSaver: q=1.74591e+06 n=827816 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 99340 traces, 99340 summary
Retagger: tagging trace set: gauss with 85440 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 85440 traces tagged "gauss"
FrameSaver: q=2.69591e+06 n=1226225 tag=gauss
wclsFrameSaver: saving 99340 traces tagged "wiener"
FrameSaver: q=2.92024e+06 n=1272214 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 52630 traces, 52630 summary
Retagger: tagging trace set: gauss with 48945 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 48945 traces tagged "gauss"
FrameSaver: q=895676 n=584107 tag=gauss
wclsFrameSaver: saving 52630 traces tagged "wiener"
FrameSaver: q=967595 n=606011 tag=wiener
05-Aug-2025 00:06:49 CEST  Closed output file "pdhd_processing_1_14363_0_1754328471_stage1_stage2.root"
05-Aug-2025 00:06:49 CEST  Closed input file "pdhd_processing_1_14363_0_1754328471_stage1.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              157.53        392.788       657.512       396.964       125.032        40     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000367019   0.00658002     0.0857537    0.00378667     0.0130943       40     
produce:fembfilter:PDHDFEMBFilter                      0.640319      0.713285      0.863491      0.703493      0.0638358       40     
produce:wclsdatahd:WireCellToolkit                      41.3545       53.6536       67.2309       52.7102       4.99455        40     
produce:gaushit:GausHitFinder                          0.377498      0.646196       1.01018      0.626719      0.149559        40     
produce:nhitsfilter:NumberOfHitsFilter                0.00022135    0.00104895    0.00278645    0.000484023    0.0010038       40     
produce:reco3d:SpacePointSolver                         0.20271       0.95425       5.43301      0.653613       1.03528        40     
produce:hitpdune:DisambigFromSpacePoints               0.168973      0.506872       1.52282      0.463228      0.270084        40     
produce:pandora:StandardPandora                         5.45097       15.4022       66.879        12.3966       10.5989        40     
produce:pandoraWriter:StandardPandora                   0.09501      0.179244      0.314432      0.176225      0.043465        40     
produce:pandoraTrack:LArPandoraTrackCreation           0.641676       6.59067       19.6111       5.89998       4.3369         40     
produce:pandoraShower:LArPandoraShowerCreation         0.818239       8.98531       38.182        7.67773       7.25341        40     
produce:pandoracalo:Calorimetry                        0.356952       2.29832       6.69894       2.01968       1.5114         40     
produce:pandoracalonosce:Calorimetry                   0.319488       2.24578       6.72108       1.89761       1.50121        40     
produce:pandorapid:Chi2ParticleID                     0.00132618    0.00277003    0.00692925    0.00212984    0.00146041       40     
produce:pandoraShowercalo:ShowerCalorimetry            0.855635       6.13737       29.2948       4.76226       5.33536        40     
produce:pandoraShowercalonosce:ShowerCalorimetry       0.794372       6.07521       29.0277       4.79646       5.3877         40     
produce:pandoraGnocchiCalo:GnocchiCalorimetry          0.0504403     0.130477      0.259206      0.130122      0.0395255       40     
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0266042     0.0563185     0.122662      0.0531845     0.0177105       40     
produce:emtrkmichelid:EmTrackMichelId                   9.92858       17.4615       26.2376       16.797        3.7033         40     
[art]:TriggerResults:TriggerResultInserter            2.8412e-05    6.00289e-05   9.4811e-05    6.1572e-05    1.45878e-05      40     
end_path:out1:RootOutput                               5.948e-06    1.19593e-05   2.3225e-05    1.12675e-05   3.82997e-06      40     
end_path:michelremoving:michelremoving                 0.0199734     0.144949      0.410725      0.122511      0.0971517       40     
end_path:michelremovinggnocchi:michelremoving          0.015578      0.139096      0.396362      0.120173      0.0967567       40     
end_path:calibanagnocchi:CalibAnaTree                   76.401        268.678       497.056       267.732       105.343        40     
end_path:out1:RootOutput(write)                        0.396441      0.757374       1.13124       0.72826      0.167787        40     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9157.72 MB
  Peak resident set size usage (VmHWM): 6067.45 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'pdhd_processing_1_14363_0_1754328471_stage1_stage2.root', 'tfile_out': 'pdhd_processing_1_14363_0_1754328471_stage1_stage2_tfile.root', 'start_time': 1754329453.8789554, 'end_time': 1754345211.6397676}
justIN time: 2025-08-05 01:19:03 UTC       justIN version: 01.04.00