justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 20914.186@dunegpschedd01.fnal.gov

Jobsub ID20914.186@dunegpschedd01.fnal.gov
Workflow ID404
Stage ID1
User namemsiden@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8387559424 (7999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-08-05 02:58:41
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2025-08-05 08:46:48
From worker nodeHostnamewn2209251.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes10485760000 (10000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statefinished
Started2025-08-05 03:22:49
Input fileshd-protodune:np04hd_raw_run027087_0022_dataflow1_datawriter_0_20240614T145657.hdf5
JobscriptExit code0
Real time5h (19387s)
CPU time5h (18614s = 96%)
Max RSS bytes3693477888 (3522 MiB)
Outputting started2025-08-05 08:45:57
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/msiden/channel_map_test/fnal/00404/1/001/pdhd_processing_1_20914_186_1754364185_stage1_stage2_tfile.root
https://fndcadoor.fnal.gov:2880/dune/scratch/users/msiden/channel_map_test/fnal/00404/1/001/pdhd_processing_1_20914_186_1754364185_stage1_stage2.root
Finished2025-08-05 08:46:48
Saved logsjustin-logs:20914.186-dunegpschedd01.fnal.gov.logs.tgz
List job events     Cached HTCondor job logs

Jobscript log (last 10,000 characters)

.43879e+06 n=855006 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 84218 traces, 84218 summary
Retagger: tagging trace set: gauss with 71827 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 71827 traces tagged "gauss"
FrameSaver: q=2.17338e+06 n=1002905 tag=gauss
wclsFrameSaver: saving 84218 traces tagged "wiener"
FrameSaver: q=2.40139e+06 n=1010428 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 81111 traces, 81111 summary
Retagger: tagging trace set: gauss with 69492 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 69492 traces tagged "gauss"
FrameSaver: q=1.87705e+06 n=931591 tag=gauss
wclsFrameSaver: saving 81111 traces tagged "wiener"
FrameSaver: q=2.01302e+06 n=959739 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 68858 traces, 68858 summary
Retagger: tagging trace set: gauss with 58147 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 58147 traces tagged "gauss"
FrameSaver: q=1.43541e+06 n=758468 tag=gauss
wclsFrameSaver: saving 68858 traces tagged "wiener"
FrameSaver: q=1.63153e+06 n=793640 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 67811 traces, 67811 summary
Retagger: tagging trace set: gauss with 60128 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 60128 traces tagged "gauss"
FrameSaver: q=1.40483e+06 n=791227 tag=gauss
wclsFrameSaver: saving 67811 traces tagged "wiener"
FrameSaver: q=1.55654e+06 n=823983 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 80273 traces, 80273 summary
Retagger: tagging trace set: gauss with 70572 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 70572 traces tagged "gauss"
FrameSaver: q=1.88327e+06 n=1011714 tag=gauss
wclsFrameSaver: saving 80273 traces tagged "wiener"
FrameSaver: q=1.93854e+06 n=1034843 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 87265 traces, 87265 summary
Retagger: tagging trace set: gauss with 76623 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 76623 traces tagged "gauss"
FrameSaver: q=2.34323e+06 n=1075174 tag=gauss
wclsFrameSaver: saving 87265 traces tagged "wiener"
FrameSaver: q=2.37647e+06 n=1130704 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 82300 traces, 82300 summary
Retagger: tagging trace set: gauss with 70238 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 70238 traces tagged "gauss"
FrameSaver: q=1.95322e+06 n=988375 tag=gauss
wclsFrameSaver: saving 82300 traces tagged "wiener"
FrameSaver: q=2.07696e+06 n=1030111 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 79930 traces, 79930 summary
Retagger: tagging trace set: gauss with 69543 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 69543 traces tagged "gauss"
FrameSaver: q=2.18629e+06 n=1081987 tag=gauss
wclsFrameSaver: saving 79930 traces tagged "wiener"
FrameSaver: q=2.27825e+06 n=1080256 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 92292 traces, 92292 summary
Retagger: tagging trace set: gauss with 79347 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 79347 traces tagged "gauss"
FrameSaver: q=2.32849e+06 n=1044381 tag=gauss
wclsFrameSaver: saving 92292 traces tagged "wiener"
FrameSaver: q=2.6442e+06 n=1112236 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 85309 traces, 85309 summary
Retagger: tagging trace set: gauss with 74723 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 74723 traces tagged "gauss"
FrameSaver: q=2.06304e+06 n=978525 tag=gauss
wclsFrameSaver: saving 85309 traces tagged "wiener"
FrameSaver: q=2.18514e+06 n=1053968 tag=wiener
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 79601 traces, 79601 summary
Retagger: tagging trace set: gauss with 69463 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 69463 traces tagged "gauss"
FrameSaver: q=1.98351e+06 n=964121 tag=gauss
wclsFrameSaver: saving 79601 traces tagged "wiener"
FrameSaver: q=1.99053e+06 n=979452 tag=wiener
05-Aug-2025 09:45:54 BST  Closed output file "pdhd_processing_1_20914_186_1754364185_stage1_stage2.root"
05-Aug-2025 09:45:54 BST  Closed input file "pdhd_processing_1_20914_186_1754364185_stage1.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              219.863       445.632       898.392       405.213       145.483        40     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000636027   0.00509959     0.041683     0.00385115     0.006215        40     
produce:fembfilter:PDHDFEMBFilter                      0.632037      0.906966       1.13786      0.933423       0.11393        40     
produce:wclsdatahd:WireCellToolkit                      37.9276       49.7764       62.0303       50.0891       5.37697        40     
produce:gaushit:GausHitFinder                          0.575541       1.02812       1.69941      0.948408      0.266866        40     
produce:nhitsfilter:NumberOfHitsFilter                0.000266271   0.000536655   0.00115029    0.000439192   0.000250476      40     
produce:reco3d:SpacePointSolver                        0.222689      0.946922       2.05997      0.806102      0.497459        40     
produce:hitpdune:DisambigFromSpacePoints               0.324345      0.859916       1.95457      0.732442      0.388778        40     
produce:pandora:StandardPandora                         6.62642       23.2517       58.2895       18.8408       11.4573        40     
produce:pandoraWriter:StandardPandora                  0.119498      0.241472       0.42119      0.242115      0.0634217       40     
produce:pandoraTrack:LArPandoraTrackCreation            1.12252       3.87166       8.4955        3.62298       1.6216         40     
produce:pandoraShower:LArPandoraShowerCreation          1.11704       4.61319       10.4945       4.27923       2.17412        40     
produce:pandoracalo:Calorimetry                        0.481594       1.41675       3.19202       1.31961      0.555235        40     
produce:pandoracalonosce:Calorimetry                   0.402427       1.30168       3.04401       1.10118      0.539032        40     
produce:pandorapid:Chi2ParticleID                     0.00128051    0.00334067     0.0100277     0.0027154    0.00183814       40     
produce:pandoraShowercalo:ShowerCalorimetry            0.979607       3.6803        9.51736       2.96727       1.99528        40     
produce:pandoraShowercalonosce:ShowerCalorimetry        1.0369        3.58761       8.68696       2.9215        1.8684         40     
produce:pandoraGnocchiCalo:GnocchiCalorimetry          0.0815278     0.194692      0.318574      0.192459      0.0630108       40     
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry     0.0348873     0.0706835     0.110595      0.0747132     0.0204443       40     
produce:emtrkmichelid:EmTrackMichelId                   23.7595       36.6527       74.2546       35.436        9.51808        40     
[art]:TriggerResults:TriggerResultInserter             3.725e-05    9.43221e-05   0.000238629   8.76895e-05   3.2154e-05       40     
end_path:out1:RootOutput                              1.2844e-05    2.79329e-05   9.3746e-05    2.38395e-05   1.5583e-05       40     
end_path:michelremoving:michelremoving                 0.0384074     0.224492       0.85918      0.185168      0.182101        40     
end_path:michelremovinggnocchi:michelremoving          0.0348503     0.213642      0.890772       0.18248      0.178215        40     
end_path:calibanagnocchi:CalibAnaTree                   121.793       310.923       658.486       286.756       124.418        40     
end_path:out1:RootOutput(write)                        0.603199       1.15499       1.93618       1.15353      0.301194        40     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6091.02 MB
  Peak resident set size usage (VmHWM): 3693.48 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'pdhd_processing_1_20914_186_1754364185_stage1_stage2.root', 'tfile_out': 'pdhd_processing_1_20914_186_1754364185_stage1_stage2_tfile.root', 'start_time': 1754365656.4656997, 'end_time': 1754383556.6894166}
justIN time: 2025-08-05 23:17:46 UTC       justIN version: 01.04.00